[BUG] Chromium and webkit headless/head rendering differences

[manuman94]

System:

• OS: Linux 5.4 Ubuntu 20.04.3 LTS (Focal Fossa)
• Memory: 1.12 GB / 15.45 GB

Binaries:

• Node: 14.18.0 - /usr/local/bin/node
• Yarn: 1.22.5 - /usr/bin/yarn
• npm: 8.5.1 - /usr/local/bin/npm

Languages:

• Bash: 5.0.17 - /bin/bash

Playwright env.:

• Playwright Version: 1.19.2
• Browser: All

The bug

Hi there! We are testing a kind of "analytics" web where some charts and diagrams are shown using amCharts library. The first approach is being made with snapshot testing.

The issue comes with head/headless option, as we develop with headless option set to false and the CI environment uses headless executions.

When the snapshots are made with headless option enabled and then we launch the tests in headful mode the tests in Chromium and Webkit are failing, the images are not equal. This doesn't happen with Firefox, that makes identically component screenshots.

Evidences

We made a simple project that enters an amChart example and makes a snapshot of a pie chart component. The test consist on comparing a snapshot made with headless option set to true and then repeat the test in headful mode (and vice versa).

https://github.com/manuman94/playwright-head-vs-headless-rendering

Further, I tried to compare the images using pixelmatch library (which Playwright is using too I believe) and get the image difference. Here you can see the both images and the difference image:
Headless:

HeadFul:

Difference:

Previous issues

There was an issue opened on Jan 4th 2021 (#11177) regarding this issue, but as for today is closed and a pull request was created to solve the font family problems.

There is another closed issue (#5371) but it seems that it was reporting a completely different problem.

Side notes

• Consequent screenshots in the same mode (headless or headful) are identical. This confirms that the amCharts library renders its components in the same way every time.
• The font family is not the only issue as the difference image shows that some pixels part of the chart are different too.
• Firefox does not have this difference.
• Is there any way of obtaining the result image using the matcher .toMatchSnapshot? The method signature tells that it returns a Buffer but I'm not able to obtain the difference image. The one I got is using pixelmatch function directly.

Thank you so much, greetings.

[yury-s]

Do you see the screenshots that don't include <canvas/> element?

Looks like another issue caused by different anti-aliasing behavior is different between headed and headless modes. Unfortunately there is no good way to make the browsers produce identical images in both modes. We are adding additional settings to toMatchSnapshot matcher in the next release (see #12167), namely pixelCount and pixelRatio. They will enable you to increase the tolerance level of the diff algorithm to allow for small discrepancies like this. You can already give it a try with npm i @playwright/test@next

[manuman94]

Hi @yury-s,

Yes, we have a test that uses SVGs and the screenshots doesn't match between headful and headless. In fact, the screenshots above are of an SVG, it's not a canvas object.

Short HTML code behind the screenshots:

Okay! So the only way to avoid this issue is to play with tolerance, as I understand it is a browser issue.

Thank you so much for your support.

Greetings.

[aslushnikov]

@manuman94 the maxDiffPixels and maxDiffPixelRatio were released yesterday in 1.20 release.

[aslushnikov]

Summary: browsers do render differently in headless and headed modes, so comparing headless screenshots with headful
screenshots becomes complicated.

Rendering issues include:

• different font stacks for headless vs headful (attempts are made to fix in chrome)
• different antialiasing in headless vs headful (with the amchart example demonstrating the issue)
• scrollbars - hidden in headless and might be shown in headful.

Our options:

• can we align headless vs headful rendering across the board?
• shall we consider including headlessness to the upcoming screenshotsDir path?
  • if yes, then how?! do we do a resolver?
• shall we introduce an option to enable scrollbars in headless mode?

[amrsa1]

@aslushnikov @dgozman
Seems that maxDiffPixels doesn't have any effect on the test

the 15 pixels are the difference between headed and headless mode

[aslushnikov]

the 15 pixels are the difference between headed and headless mode

maxDiffPixels only comes into play when image dimensions are equal.

[benwiley4000]

@aslushnikov The problem with our case is that headed vs headless renders the image with one pixel difference of dimensions. I'm not sure I understand how maxDiffPixelRatio will work. Does it also require maxDiffPixels to be set? Or does it replace this?

[cassmtnr]

I am working with a defect where a component shouldn't display a scrollbar, when I run with headed mode on Chromium or Firefox it shows it and I can throw the error with toMatchSnapshot(), but it doesn't show up when it's running in headless mode (local or in CI).

Having a option to enable scrollbars on headless mode would make total sense as we're testing Visual here.

@aslushnikov it seems you already have an prepared option for that in browser_patches/firefox/juggler/content/main.js#L61 is it possible to make it available as option?

[gselsidi]

what is the status on this? Noticing these issues when creating baseline images from headless / headed and vice versa. Normally I'm able to get away with an insane .0001 maxpixeldifference!!

When running cross headed / headless not even .01 get's them to pass

[gselsidi]

@aslushnikov

I'm noticing maxDiffPixels: 5, is not working.... On ci there is a 1 pixel difference of the snapshot say 1x1 vs 1x2 and it fails. Tried 1000 pixels still fails

Expected an image 60px by 125px, received 60px by 124px......

[IvanAbramow]

@aslushnikov
Hello! I have exactly the same problem, the size of the element differs by 1 - 2 pixels.
Screenshot comparison failed: Expected an image 640px by 694px, received 640px by 696px.

This appears for both page.screenshot and element.screenshot.

For example, in jest-image-snapshot package https://www.npmjs.com/package/jest-image-snapshot
there is an allowSizeMismatch parameter that allows you to solve this problem.

@playwright/test 1.27.1

Please find a solution to fix this problem as soon as possible. Due to this problem, I can not completely switch to playwright without using jest

[dgtlmoon]

Please find a solution to fix this problem as soon as possible.

I think this is an inappropriate comment considering that the software is provided to you free of charge and without warranty

[IvanAbramow]

Please find a solution to fix this problem as soon as possible.

I think this is an inappropriate comment considering that the software is provided to you free of charge and without warranty

I agree with you, I don't mean to offend the playwright team. My mistake. I really love this product and enjoy using it. I just found a few requests about this problem, so I asked to try to solve it.

[eveltman-twinfield]

Would the new Chrome headless mode (not yet officially supported in Playwright) solve this issue?
https://developer.chrome.com/articles/new-headless/#whats-new-in-headless

As I understand it, the performance drops quite a bit, but for me, if the rendering becomes identical to the headful browser, and if it increases the usability of visual testing (developing in headful mode and running in CI in headless mode), then I would prefer the new headless mode of Chrome.

[JonMarbach]

aslushnikov +1 to "shall we introduce an option to enable scrollbars in headless mode?"

But perhaps more importantly, the new headless mode that @eveltman-twinfield mentioned above sounds even better. A relevant quote from that article: "This created a confusing situation where any automated browser test might pass in headful mode but fail in Headless mode, or vice versa — a major pain point for automation engineers."

[rgustavsson]

What's needed to try out the new chrome --headless=new flag?
@eveltman-twinfield mentioned that this isn't officially supported in Playwright yet, but should I be able to try it by using the latest Playwright version, setting browser launch option Headless to false, and then pass --headless=new in the Args browser launch option? :)

EDIT:
Above worked out for me.
Screenshot comparison now acts as expected for us no matter if we run headless (=new) or with head.. 👏

[chartinger]

Since b21c81d there is a new env var option PLAYWRIGHT_CHROMIUM_USE_HEADLESS_NEW but it seems to be undocumented (?)

[afonte15]

Hi, are there any updates on this bug. I am currently running into this issue between headless and headed modes. I am currently experiencing this issue.

[oliverp]

Hello, im running in this issue with the width difference in headless creation & headless test. So taking snapshots with a headless environment on the same machine and testing against those snapshots on the same machine produces :

Expected an image 1125px by 229px, received 1133px by 229px.

I tested to define the viewport without success.
I tested chrome without success.

I guess I will crop the resulting snapshots, so the test might work.

[barbolo]

Since b21c81d there is a new env var option PLAYWRIGHT_CHROMIUM_USE_HEADLESS_NEW but it seems to be undocumented (?)

https://developer.chrome.com/articles/new-headless/

[gselsidi]

would this solve anything about cross architecture differences? Because in the past even CPU differences for the same architecture would cause slight differences and it would fail.

So if i author on say Mac and then run on Linux in CI would this new chrome feature resolve those issues? Has anyone tried it?