[Feature] Track Code Coverage

[udit0802]

It would be great if playwright can provide OOTB support for calculating the code coverage of e2e tests.
This feature is provided by every test framework, as we are migrating to playwright from wdio, this would be a great help if this feature is available.

[aslushnikov]

There's a guide on how to setup test coverage: https://github.com/mxschmitt/playwright-test-coverage

Does it help?

[udit0802]

https://github.com/mxschmitt/playwright-test-coverage

I have tried that out, .nycoutput folder is generated but no reports are generated.
It did not help out much as this is react based, whereas my application is not.
I would highly appreciate, something OOTB can be provided by playwright team.
Looking forward for a positive response

[pavelfeldman]

How do you use this coverage data? (Coverage is typically used in the unit test context only, rarely used for e2e).

[udit0802]

How do you use this coverage data? (Coverage is typically used in the unit test context only, rarely used for e2e).

Very similar use case as mentioned here.
#7030 (comment)

[udit0802]

Hi any plans/ETA for this feature request?

[udit0802]

@aslushnikov
Gentle Reminder!!
Any update on this feature request?

[udit0802]

Gentle Reminder!!
Any update for the same.
Any help would be highly appreciated.

[p01]

Code coverage of the final bundles would be quite useful to know how much ground the E2E tests cover and to assess whether a new test add sufficient value compared to the cost of running on CI in a big a repo.

[Uninen]

The guide uses babel-plugin-istanbul which is not helpful for modern projects that don't use babel (like Vite / esbuild). It would be nice to have a way to use for example c8 instead.

[jfgreffier]

Playwright actually offers API to get v8's coverage. There is an example using v8-to-istanbul, but it is rather incomplete since it doesn't allow generating coverage report.
https://playwright.dev/docs/api/class-coverage

You can indeed get coverage from v8 and read it from c8. Here is what I did :

const { test, expect } = require("@playwright/test");
const fs = require("fs");
const url = require("url");
const path = require("path");

test("test", async ({ page }) => {
  // start coverage
  await page.coverage.startJSCoverage();

  // TODO Your test
  ...

  // Get and save V8 coverage
  const coverage = await page.coverage.stopJSCoverage();
  const rootPath = path.normalize(`${__dirname}/..`);
  const coverageWithPath = coverage.map((entry) => {
    const fileName = new url.URL(entry.url).pathname;
    return { ...entry, url: `file:///${rootPath}${fileName}` };
  });

  fs.writeFileSync(
    "coverage/tmp/coverage.json",
    JSON.stringify({ result: coverageWithPath }, null, 2)
  );
});

I had to fix the url, by getting the full path of the files. You might need to adjust rootPath.

After that, just run npx c8 report

[merishabhgupta]

Hi
I don't have much experience with testing. So my query is how to get code coverage of e2e tests written with playwright.
Scenario:
We have a different repo for automation test cases which contains only the automation test cases and they point to some instance where the production application is deployed. So we are running automation test cases on local over a deployed application on some server.
We have to get the code coverage from these e2e tests when we will be running these tests on the same server after deploying this test cases repo on the same server as our application.
How should the config look in this scenario.
Any kind of help is appreciated.
@aslushnikov
@jfgreffier
@udit0802

[testgitdl]

@jfgreffier I tried using your code but I do not get any coverage information. I am starting my app via "ng serve -c dev" and afterwards want to get coverage information.
I have an Angular 14 project with ESM and the test framework is a cucumberjs + Playwright + TypeScript.
I start and stop jscoverage in the beforeall/afterall hooks but do not receive any info although the coverage.json is created.
Do you know what I am doing wrong?

hOPE YOU CAN HELP
``
BeforeAll(async function () {
browser = await chromium.launch(configChrome);
context = await browser.newContext();
page = await context.newPage();
await page.coverage.startJSCoverage();
});

AfterAll(async function () {
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const coverage = await page.coverage.stopJSCoverage();
const rootPath = path.normalize(${__dirname}/..);
const coverageWithPath = coverage.map((entry) => {
const fileName = new url1.URL(entry.url).pathname;
return { ...entry, url: file:///${rootPath}${fileName} };
});
fs.writeFileSync(
"coverage/tmp/coverage.json",
JSON.stringify({ result: coverageWithPath }, null, 2)
);
``

Output of npx c8 report
----------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------|---------|----------|---------|---------|-------------------
All files | 0 | 0 | 0 | 0 |
----------|---------|----------|---------|---------|-------------------

[kachkaev]

For those who are looking for a temp drop-in solution, I can recommend anishkny/playwright-test-coverage, which is available on NPM. It uses Istanbul instead of C8, which allows to collect coverage from all browsers. Before running tests, the app needs to be built with coverage instrumented (via babel-plugin-istanbul).

We use playwright-test-coverage package in blockprotocol/blockprotocol inside integration tests (five browsers). The stats are uploaded to Codecov. Search for TEST_COVERAGE and playwright-test-coverage for hints.

Hope to see built-in coverage reporting at some point!

[testgitdl]

@kachkaev thank you very much for checking. I already took this as an example and the problem is that the nyc-output is empty - probably it has to do with the nyc ng serve. Please let me know if you have more ideas - would help big time
Below my npm scripts:
"serve": "nyc ng serve", "test:only": "playwright test", "test": "start-server-and-test serve http://localhost:4200 test:only"

And the output of npm run test

new-project@0.0.0 test
start-server-and-test serve http://localhost:4200 test:only

1: starting server using command "npm run serve"
and when url "[ 'http://localhost:4200' ]" is responding with HTTP status code 200
running tests using command "npm run test:only"

new-project@0.0.0 serve
nyc ng serve

✔ Browser application bundle generation complete.

Initial Chunk Files | Names | Raw Size
vendor.js | vendor | 1.77 MB |
polyfills.js | polyfills | 318.06 kB |
styles.css, styles.js | styles | 210.09 kB |
main.js | main | 48.04 kB |
runtime.js | runtime | 6.52 kB |

                  | Initial Total |   2.34 MB

Build at: 2022-11-18T14:33:43.337Z - Hash: 0380aa086066a4e4 - Time: 5119ms

** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **

√ Compiled successfully.

new-project@0.0.0 test:only
playwright test

Running 3 tests using 3 workers

3 passed (7s)

To open last HTML report run:

npx playwright show-report

[testgitdl]

@kachkaev I have created a sample project here: https://github.com/testgitdl/playwright-coverage

[jfgreffier]

@testgitdl for c8 coverage, yoiu should open the coverage.json and check it. You might have coverage info, or not. Maybe you have coverage info but the wrong path... You should come over to the Slack, a GitHub issue isn't very convenient

[jennydaman]

@jfgreffier provided a great example in #9208 (comment)

I've improved on his code snippet by putting it into a fixture.

// fixtures.ts
import { test as base } from "@playwright/test";
import * as fsPromises from "node:fs/promises";
import * as path from "node:path";

type MyFixtures = {
  _withCoverage: undefined;
};

export const test = base.extend<MyFixtures>({
  _withCoverage: async ({ page }, use, testInfo) => {
    await page.coverage.startJSCoverage();
    await use(undefined);

    const coverage = await page.coverage.stopJSCoverage();
    const srcPath = "@fs" + path.normalize(`${__dirname}/../../src`);
    const srcCoverage = coverage
      .filter((entry) => entry.url.includes(srcPath))
      .map((entry) => {
        return { ...entry, url: entry.url.replace(/^.+@fs/, "file://") };
      });
    await fsPromises.mkdir("coverage/tmp", { recursive: true });
    const testTitle = testInfo.title.replaceAll("/", "_");
    await fsPromises.writeFile(
      `coverage/tmp/${testTitle}.json`,
      JSON.stringify({ result: srcCoverage }, null, 2),
    );
  },
});

export { expect } from "@playwright/test";

Then in your test files:

import { test, expect } from "./fixtures.ts";

test("test1", async ({ page, _withCoverage }) => {
  await page.goto("http://localhost:5173/");
});

test("test2", async ({ page, _withCoverage }) => {
  await page.goto("http://localhost:5173/");
});

After running playwright test, the fixture creates coverage/tmp/test1.json and coverage/tmp/test2.json

[jfgreffier]

@jennydaman It is actually improving on my example ! Do you think you can make it into a Github repository for people to try it out and contribute ?

[jennydaman]

I couldn't figure it out how to do so. I wanted to try extending defineConfig and test in a separate package following examples from https://github.com/microsoft/playwright/tree/main/packages/playwright-ct-core and https://github.com/anishkny/playwright-test-coverage however I kept getting errors saying

Error: Requiring @playwright/test second time,

EDIT: I found the workaround here #15819, will create the package!

[jennydaman]

I have developed a minimal package which configures Playwright to collect test coverage and write the data to a directory.

https://www.npmjs.com/package/playwright-test-coverage-native

[BagchiMB]

@merishabhgupta I have the exact use case that you had, were you able to find any solution?

[cenfun]

If you are looking for a JavaScript code coverage tool, You could try monocart-coverage-reports

• Native V8 coverage reports
• Integration Istanbul reports

Especially for Playwright, it has been seamlessly integrated into the custom reporter monocart-reporter, see Code Coverage Report
Any suggestions are welcome.

[pgangwani]

I am at hour of deciding that whether should I select cypress or playwright. This feature is very important for me to tackle like cypress is doing here.

[spuxx1701]

Playwright really needs to provide better support for coverage reporting. The current documentation is really lackluster and folk need to scavenge the internet for anything they can find on how to properly implement this.

Since Microsoft appears to have been unsure on why people want this in the first place, let me explain. Coverage is (albeit not perfect) an important metric to teams to check whether workflows have been tested thorougly, and UI tests are no exception to this. When it comes to frontends in particular, many teams rely heavily on integration and UI tests, which are hard to get done properly with existing (unit) test frameworks like vitest. In our team, we use vitest for unit-focused testing and Playwright for integrated and UI testing.

If coverage reports only cover e.g. vitest tests, reviewers have to guess whether any uncovered code might be covered by UI tests. If we get coverage reports from both playwright and vitest, we can even merge the reports with tools like Istanbul to get the full picture. This is extremely beneficial to automated quality checks.

So please, Microsoft, don't neglect proper coverage reporting. Playwright really needs to provide a better API and documentation on coverage reporting, just like e.g. vitest does.

[sszemeresky]

ok, i wanted to add code coverage report for my project to work with sonar-qube, but i see that is not that easy...
+1 on the feature request