[SPIKE] Testing with Playwright

[romaricpascal]

Explores the use of Playwright as a potential replacement for Puppeteer to run our in-browser tests.

This should offer us a wider range of browsers to test in, increasing our confidence that things run OK. It should offer a better debugging experience locally (viewing browser and pausing when querying its content), as well as remotely

Changes

The PR adds Playwrigh to the project and ports two tests to it:

1. the CharacterCount JavaScript component tests, as a way to see how different things would be from our current use of Puppeteer
2. the Checkboxes template test (except snapshot matching assertions for now) as a way to explore the tradeoffs of using Playwright's browser-oriented ways of locating elements and making expectations

The PR also sets up Github workflows to compare the tests running:

a. using Jest
b. using Playwright, in a single run targeting all 3 available browsers
c. using Playwright, in parallel runs for each of the 3 available browsers

The workflows are set up to run either each file individually or both files in a single run. This is in the hope to highlight the cost of moving from Jest+Cheerio template tests to Playwright ones.

Findings

Running tests

Puppeteer is only about controlling the browser, we bind it to jest using jest-puppeteer. Playwright provides both the API for controlling the browser and the test runner. It can be bound to jest, but that's not the route recommended by the project.

The test runner API offers similar concept as Jest, but in a slightly different API: test instead of it, and the describe, beforeAll, beforeEach methods are accessed on that test object rather than as argument. It also uses a slightly different API for configuring the use of JavaScript

Each test gets its own isolated "BrowserContext", which includes a fresh page, so that ensures tests don't leak from one to another as far as I understand. Gotcha is that the page is not accessible in test.beforeAll, though, as it is created for each test, so to be reserved for non in-browser initialisation. If some initialisation need to happen on the page, it'll need to be in test.beforeEach (or manually creating a shared page, which is also a possibility, but less ideal).

There's no global page variable. You get reference to the page through the arguments of the test function. This makes things quite neat and make sure test logic runs on the relevant page.

Runner configuration

I've mostly stuck to the pre-configured default form the initialisation. They did the job to get looking at how things would work. They also configure some decent defaults, especially retrying and capturing trace for debugging on CI (more later).

The runner can be configured to automatically start a server, after checking if one is already running. It gives flexibility to npm start or not before running the tests. When configured to run a server, the API to control the browser will use it as base for resolving URLs, allowing us to not have to juggle with the server's port, which is nice.

Working locally

The runner doesn't have a watch mode, which is a major pain point for a test runner 😭 Nothing that nodemon cannot fix though PWDEBUG=0 PW_REPORTER_HTML_OPEN=never npx nodemon --watch "src/**/*.spec.{cjs,js,mjs}" --exec "npx playwright test --project chromium". We'll probably want that as an npm script though.

Unless told otherwise, the runner will automatically open an HTML report of the result if they fail when ran locally. I added some configuration to get some flexibility on that, as it can be a bit annoying and open lots of browser windows. The HTML reports are saved anyway and can be open with npx playwright show-report.

The test runner offers a debug mode (PWDEBUG=1 or --debug) which will pause at each interaction with the browser (locating an element or actual interaction), allowing to go step by step locally.

The test runner can capture traces of the interactions with the browser to replay them afterwards. Not necessarily useful locally thanks to the debug mode, outside of sharing with another dev maybe. It's more something useful on CI to investigate test failures.

Writing tests

The API for controlling browser is very very close to Puppeteer's. To the point that porting the tests could get started by [moving them across and replacing the test runner API only, mostly][port-tests].

The API is able to control 3 different browser engines: Chromium, Firefox and Webkit. Note the mention of Chromium and Webkit, not Chrome (though if Chrome/Edge is installed it can run against it) and Safari. It still gives us a wider target than Puppeteer.

Past that, Playwrights offers some neat additions that make the tests clearer:

• clear API to locate elements. Especially by roles and accessible name, providing a closer experience to what our users will do than CSS selectors.
• matchers designed to test frontend code: toBeVisible(), toHaveAttribute(attributeName, value), toHaveCSS(propertyName, value), toHaveId(), toHaveText() and of course toHaveClass() (though that one is a little unfortunately implemented and does string comparison against the whole value rather than checking individual classes, though RegExp can alleviate that. Probably worth declaring our own toContainClass that abstracts that up).

Playwright supports snapshot matching. This would allow to check for regressions in the HTML of the component. I didn't explore this further, leaving our snapshot tests still to rely on Cheerio. One change is that snapshots are stored as individual files rather than a single file as of now. While Playwright points to Jest's expect API for its documentation, I couldn't figure a way to get them stored as a single file.

[not explored] Playwright also offers a way to compare screenshots.

On CI

Test reports report can be uploaded as artifact of the Github action. Once downloaded and unziped, they can be opened to access the report with npx playwright show-report like a local report. They'll contain the traces for failing tests which can be then investigated in the browser.

Playwright also offers a Progressive Web App for loading the traces, that may smoothen the workflow if we upload only the traces as Github action artifacts.

Perf wise, the PR runs a good few workflow so we can compare running times inside Github actions (see the Changes for description). Clicking "Checks" than "Tests" will give a more palatable view than the list of checks at the bottom of this PR due to the long names.

First notable thing is that Playwright will download the 3 browsers and it takes a little bit. Something to keep in mind when looking at the different checks, as 40-ish seconds of the Playwright runs are down to installing browsers. Hopefully a bit of caching can speed that up.

Even when running a single browser and using 2 workers (matching Github actions), Playwright does run slower than our tests with Puppeteer (comparing "Puppeteer Comparison" times with "Playwright - Matrix , Browser Test (chromium)"), whether it runs a both files or a single one. Goes from ~25s to ~35s with the two files, but hard to say how that will scale up with the number of files (is it the "launch" that's creating the increase and would be a one-off-ish thing, or each test running slower?).

Running multiple browsers unsurprisingly takes longer, especially as Playwright runs only a single worker (50% of available cores, I think), so we'd likely want to create a matrix to parallelise each browser run.

The hardest perf impact is moving template tests from Cheerio to Playwright. Spinning up a browser moves from 6s to 20ish seconds. Again no idea how that would scale, so the gain from a more fluent API for writing the tests will have to be weighted against the performance.

[playwright-port-tests]: alphagov/govuk-frontend@c3ab8d2 (#3064)

Steps trail

• Set up Playwright in the repo
• Port the CharacterCount JS component tests to Playwright
• Tweak Github workflow configuration so it runs on Github similarly to the other test
  Adds 3 extra jobs to compare timings:
  - Playwright Bulk to run all Playwright tests in one call
  - Playwright Matrix that splits them per browser
  - Puppeteer comparison which runs the CharacterCount tests in isolation for comparison
• Port the Checkboxes Nunjucks test to Playwright
• Port the CharacterCount JS code to Playwright locators/assertions rather than Puppeteer's
• Add Github workflow test that only runs the CharacterCount JS and Checkboxes Nunjucks component tests with Puppeteer for timing comparison. We may want to look into using Playwright's Docker container to have pre-installed browsers

Left out for now:

• Port Checkboxes snapshot tests to Playwright? Would require rewriting htmlWithClassName, that isolates the markup related to a component specifically using the element's classnames.
• Check how things go on Windows?

[romaricpascal]

Closing the spike PR. Future work will be tracked by this epic: #3278