Document some of the tradeoffs of V8 coverage (vs Babel/Istanbul coverage)

[Zalathar]

🐛 Bug Report

Jest supports two code coverage providers: the default babel provider based on Istanbul, and a newer experimental-but-encouraged v8 provider (#8596) based on V8's built-in coverage features.

When I tried switching to V8 coverage, I noticed a few subtle-but-important differences that don't seem to be explicitly documented. And when I investigated further, I found that some of those differences are not a result of V8 coverage being experimental, but are caused by deeper limitations and deliberate tradeoffs that might not change in the foreseeable future.

(V8 coverage still has plenty of merit, especially in terms of performance; I'd just like to help people understand the decision they're making when they use it.)

The main differences I want to point out are:

• V8 coverage is not a 1:1 replacement for Babel/Istanbul coverage
  • Switching between the two will usually change the reported coverage statistics, which can change whether coverage thresholds are reached or not
  • Switching between the two can cause regions of uncovered code to be discovered or ignored
  • (This is mostly just an overall summary of the other points)
• V8 coverage works by taking a coverage report for output/transpiled code, and then using v8-to-istanbul and source maps to convert that report into an Istanbul-compatible format
  • This is an inherently imprecise and heuristic-driven process, though in many cases it works well enough for practical purposes
  • In some cases this can give confusing or misleading results, or fail to distinguish between user code and generated code (e.g. uncovered branches introduced by __esModule detection shims)
  • Babel/Istanbul is able to be more precise because it usually operates directly on the user's original source code
• V8 coverage tracks “blocks”, not individual statements
  • In particular, if you have a sequence of statements, and the middle statements always throw an exception, V8 coverage will mark the later statements as “covered” even though they never ran
  • This is a deliberate tradeoff made by the V8 developers who implemented coverage
  • Babel/Istanbul is able to be more precise because it explicitly instruments every source statement
• V8 coverage does not track the else branch of an if-statement without an explicit else
  • So if a one-sided if-statement's condition is always true, V8 will not warn about an uncovered branch
  • Babel/Istanbul is able to track these by artificially inserting an else with a branch counter
• V8 and Babel/Istanbul use different comment conventions for ignoring uncovered code
  • V8 coverage uses c8 ignore comments that are processed on a line-by-line basis without parsing
  • Babel/Istanbul uses istanbul ignore comments that are handled during AST processing

I'd like to find a way to update the coverageProvider documentation to mention some or all of these tradeoffs, hopefully in a way that isn't too intimidating or discouraging.

[thymikee]

Having such a comparison would be great! Looking forward to a PR, you got most of the work done :)

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.