feat!: --merge-reports to support coverage

[AriPerkkio]

Description

• Fixes Improve support for parallel runs and coverage report merging when --shard option is used #5125
• Closes [Docs] Explain how to use sharding #4269

Coverage providers can now have an optional generateCoverage function. The blob reporter will save output of this function in its reports. When --merge-reports is called, the saved blobs will be passed to new mergeReports method.

class MyCoverageProvider {
  generateCoverage(): MyCustomCoverage {
    return { some: "coverage result" } // This will be saved by blob reporter
  }
  // Function called during '--merge-reports'
  mergeReports(coverages: MyCustomCoverage[]) {
            // ^^ List of coverage results
  }
  reportCoverage(coverage: MyCustomCoverage, reportContext: ReportContext) {
              // ^^ Also available in reportContext now
  }
}

BREAKING CHANGES:

• Affects only custom coverage providers
• In reportCoverage the previous first argument reportContext is now second argument:

- reportCoverage: (reportContext?: ReportContext) => void | Promise<void>
+ reportCoverage: (coverage: CoverageResults, reportContext: ReportContext) => void | Promise<void>

Example

$ vitest --run --reporter=blob --shard=1/3

 ✓ test/first.test.ts  (1 test)
blob report written to /<root>/vitest-example/.vitest-reports/blob-1-3.json

$ vitest --run --reporter=blob --shard=2/3

 ✓ test/second.test.ts  (1 test) 1ms
blob report written to /<root>/vitest-example/.vitest-reports/blob-2-3.json

$ vitest --run --reporter=blob --shard=3/3

 ✓ test/third.test.ts  (1 test) 1ms
blob report written to /<root>/vitest-example/.vitest-reports/blob-3-3.json

$ vitest --run --merge-reports

 RUN  v2.0.0-beta.3 /<root>/vitest-example

 ✓ test/first.test.ts (1)
 ✓ test/third.test.ts (1)
 ✓ test/second.test.ts (1)

 Test Files  3 passed (3)
      Tests  3 passed (3)
   Start at  15:33:58
   Duration  1ms (transform 0ms, setup 0ms, collect 29ms, tests 3ms, environment 0ms, prepare 126ms)

 % Coverage report from v8
-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------|---------|----------|---------|---------|-------------------
All files  |     100 |      100 |     100 |     100 |                   
 first.ts  |     100 |      100 |     100 |     100 |                   
 second.ts |     100 |      100 |     100 |     100 |                   
 third.ts  |     100 |      100 |     100 |     100 |                   
-----------|---------|----------|---------|---------|-------------------

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
• Ideally, include a test that fails without this PR but passes with it.
• Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

• Run the tests with pnpm test:ci.

Documentation

• If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

• Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

[AriPerkkio]

We might need to extend vitest --merge-reports to support specifying separate directory for coverage, e.g. --merge-reports.coverage=./some/directory. By default results are read from .vites-reports/coverage, but if user defines custom directory for --coverage.reporter=blob, it difficult to use path of --merge-reports=<path>. I would like to avoid adding completely separate command --merge-coverage-reports.

[sheremet-va]

We might need to extend vitest --merge-reports to support specifying separate directory for coverage, e.g. --merge-reports.coverage=./some/directory. By default results are read from .vites-reports/coverage, but if user defines custom directory for --coverage.reporter=blob, it difficult to use path of --merge-reports=<path>. I would like to avoid adding completely separate command --merge-coverage-reports.

I would like the blob report to just be a single blob file with anything needed for merging. Can't we record everything to a directory (even tmp directory is fine by me), then archive it and unarchive during mering?

[AriPerkkio]

That would be great. Should we only allow user to specify directory and nothing else? Then store both test and coverage reports there. When merging the reports, we would respect reporters' outputFile, reportsDirectory etc. flags as usual.

[sheremet-va]

That would be great. Should we only allow user to specify directory and nothing else? Then store both test and coverage reports there. When merging the reports, we would respect reporters' outputFile, reportsDirectory etc. flags as usual.

I mean we can still allow specifying the blob name, why not? In the end it should output a single file

[AriPerkkio]

In the end it should output a single file

Does this mean that coverage reports should be included in the same file as test results?

[sheremet-va]

In the end it should output a single file

Does this mean that coverage reports should be included in the same file as test results?

Yes, I would expect everything to be inside that blob