[BUG] Screenshot tests differences between headless and headful

[bvaughn]

Context:

• Playwright Version: 1.17.1
• Operating System: Mac
• Node.js version: v14.17.0
• Browser: Chromium
• Extra: MacBook Pro with retina display and external Dell U3417W

Code Snippet

Code to reproduce this problem is available here:
https://github.com/bvaughn/planner

This is a planning app that renders to an HTML canvas.

Repro steps

1. Checkout repo
2. Run yarn install and yarn playwright:install
3. Run yarn start
4. Run yarn test:e2e (in another terminal) and observe success
5. Edit playwright.config.js to change headless to false
6. Re-run yarn test:e2e and observe failures

Describe the bug

There are subtle text rendering differences between headless and headful runs that cause tests to fail. I've experimented with increasing the expect.toMatchSnapshot.threshold value up to 0.9 but at this point, legitimate failures easily slip through. (e.g. I can change the date header formats to add a "1" and the snapshot doesn't even pick it up.)

[pavelfeldman]

I can repro this. Missing step:

2.5. Run yarn start

[pavelfeldman]

@bvaughn: sent you a PR with minor test fixes: bvaughn/planner#14

[bvaughn]

Oh that’s very kind. Thank you!

[dgozman]

Let's see whether we can fix the situation with Page.setFontFamilies.

[yury-s]

For reference, upstream patch: https://chromium-review.googlesource.com/c/chromium/src/+/3388550