Coverage providers are prone to out-of-memory errors

[AriPerkkio]

Describe the bug

Vitest runner sends coverage reports to main thread which stores them in memory during test run:

vitest/packages/coverage-istanbul/src/provider.ts

Lines 46 to 52 in cc19117

	/**
	* Coverage objects collected from workers.
	* Some istanbul utilizers write these into file system instead of storing in memory.
	* If storing in memory causes issues, we can simply write these into fs in `onAfterSuiteRun`
	* and read them back when merging coverage objects in `onAfterAllFilesRun`.
	*/
	coverages = new Map<ProjectName, CoverageByTransformMode>()

vitest/packages/coverage-v8/src/provider.ts

Line 59 in cc19117

coverages = new Map<ProjectName, CoverageByTransformMode>()

This is fine for most projects but larger projects can run into Node's out-of-memory issues. This is common problem for coverage tools. As commented in the Istanbul's provider.ts, this was only a matter of time before Vitest started to run into these. Similar reports form others tools:

• Potential OOM error bcoe/c8#338
• JavaScript heap out of memory in v15.0.0. istanbuljs/nyc#1263
• [Bug]: Jest main process memory usage grows unboundedly with v8 coverage jestjs/jest#14287

Instead of storing reports in memory, we should write them on file system. This should be done in main thread here without blocking the runner.

vitest/packages/coverage-istanbul/src/provider.ts

Lines 118 to 130 in cc19117

	onAfterSuiteRun({ coverage, transformMode, projectName }: AfterSuiteRunMeta) {
	if (transformMode !== 'web' && transformMode !== 'ssr')
	throw new Error(`Invalid transform mode: ${transformMode}`)
	let entry = this.coverages.get(projectName || DEFAULT_PROJECT)
	if (!entry) {
	entry = { web: [], ssr: [] }
	this.coverages.set(projectName || DEFAULT_PROJECT, entry)
	}
	entry[transformMode].push(coverage as CoverageMapData)
	}

Example below demonstrates the idea. Note that this is just a simplified example. Generating reports is much more complicated:

import * as fs from "node:fs/promises";

export class IstanbulCoverageProvider {
  pendingPromises = []

  onAfterSuiteRun({ coverage, transformMode, projectName }) {
    // Write reports to file system. Do not block the runner by `await`'in this call.
    // Instead store the promise for later awaiting
    const promise = fs.writeFile('some-filename.json', JSON.stringify({ coverage, transformMode, projectName }))

    this.pendingPromises.push(promise)
  }

  async reportCoverage() {
    // All tests have run, time to generate report
    await Promise.all(this.pendingPromises);

    const coverageMap = { ... };

    // Read results from file system one by one to avoid causing OOMs
    for (const filename of fs.readdir(...)) {
      const coverage = JSON.stringify(fs.readFileSync(filename));
      coverageMap.merge(coverage);
    }
  }
}

This should be possible to implement without causing breaking changes. Only the coverage providers need changes.

Reproduction

https://github.com/vitest-tests/coverage-large

Increase these to trigger the error: https://github.com/vitest-tests/coverage-large/blob/4bb1e194fa51c0ae03b9a7c76952a59b43540e49/generate-files.mjs#L4-L6

System Info

Vitest 0.34.x and anything below 1.0.0-beta.4

Used Package Manager

pnpm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[AriPerkkio]

Also need to check if vitenode.transformRequest is caching uncovered files and if there's a way to disable it:

vitest/packages/coverage-istanbul/src/provider.ts

Line 220 in abe6c72

const transformResult = await this.ctx.vitenode.transformRequest(filename)