feat: images of difference sizes cause instant test failure

[dhulme]

Sometimes Cypress takes screenshots of slightly differences sizes during the same test. If the screnshots are different sizes the test fails. But I notice in the code that it calls alignImagesToSameSize function if they are different sizes to try and do the comparison on resized images. If the test always fails what is the reason for alignImagesToSameSize? Perhaps it should attempt to continue with the comparison after resizing the images?

Example of the diff we see in screenshot size between runs:

[FRSgit]

Hi @dhulme,

Code indeed calls alignImagesToSameSize method to make the images used for the pixelDiff to have the same dimensions. So unless you get some explicit error in the console I would say that this part of code should work well.
I think there might be a big enough difference between new and old image and file diff fails.

Also - why the cypress creates differently sized screenshots? Are there any differences between environments the screenshots are created on? I'm asking, because it doesn't happen in the projects where I'm using this plugin - I might need more info to reproduce that situation.

Can you please post here:

• the dump of console output or screenshot from Cypress UI visible when running tests,
• the .diff.png file generated by the plugin.

Also could you add additional information about your environment:

OS and version: [e.g. Windows 10 build. 19043.1319]
Browser and version [e.g. chrome 22]
Cypress version [e.g. 8.6.0]
Node version: [e.g. 16.0.1]

[dhulme]

Hi @FRSgit, thanks for your reply.

When alignImagesToSameSize is called, isImgSizeDifferent willl be true so it fails automatically on line 166, ignoring the result of pixelmatch? So even if the resized images were below tolerance, it will still fail?

I'm not sure why Cypress is captuing different sized images. It only happens on a couple of my components, The images are all captured on a Linux container in a pipeline (cypress/browsers:node16.14.2-slim-chrome103-ff102), so they should be consistent. I will investigate further. I'm on Cypress 10.4.0.

[FRSgit]

When alignImagesToSameSize is called, isImgSizeDifferent willl be true so it fails automatically on line 166, ignoring the result of pixelmatch? So even if the resized images were below tolerance, it will still fail?

My bad @dhulme, you're right - if images have different sizes the test will always fail. The image size check can be removed, however that would be a breaking change - I do not want to make without giving other possibility to react on it.

I'm not sure why Cypress is captuing different sized images. It only happens on a couple of my components, The images are all captured on a Linux container in a pipeline (cypress/browsers:node16.14.2-slim-chrome103-ff102), so they should be consistent. I will investigate further. I'm on Cypress 10.4.0.

Are you capturing images during E2E or component-testing?
If during CT then you might want to try to make screenshots of the whole viewport (scale in each test using viewport command so it fits your component).

[dhulme]

My bad @dhulme, you're right - if images have different sizes the test will always fail. The image size check can be removed, however that would be a breaking change - I do not want to make without giving other possibility to react on it.

Ok thanks, that makes sense. What would be the process for collecting feedback on the change?

Are you capturing images during E2E or component-testing?
If during CT then you might want to try to make screenshots of the whole viewport (scale in each test using viewport command so it fits your component).

This is in component testing yes. In most cases I've been screenshoting the element in question e.g. button, not the whole viewport. Are you aware of issues with screenshotting elements rather than the viewport?

[FRSgit]

@dhulme As I won't have time to look into the issue this week I would let it sit here for the time being and see if anybody comes around. If nobody will respond - then it will be up to us 2 (the library doesn't have that much of an audience) 🤷
In the meantime you can give it a try with a PR if you want to - it will surely speed up the process of shipping it!

This is in component testing yes. In most cases I've been screenshoting the element in question e.g. button, not the whole viewport. Are you aware of issues with screenshotting elements rather than the viewport?

No, I'm not. Quite the contrary - I'm using this plugin in projects where elements are being image targets and it works well.
Even in this repo we run on the CI a test that makes a screenshot of a "description" element.
I would check if there isn't anything interfering with how your element gets rendered. Maybe asynchronously loaded CSS? Or not every time font is being rendered fast enough? Other elements are pushing this one throughout the screen? Maybe you're able to pinpoint the problem in your codebase.

[oguzhantx]

I am having same screenshot size issue with component tests as well. Somehow cypress is not picking up the specified view port when it comes to taking a screenshots. Also having a hard time to match locally uploaded screenshots to match the ones in the ci pipeline. Any suggestion on this?

[FRSgit]

Hey @oguzhantx!
Can you please open up a new bug request for each of the issues you've mentioned? 🙏 I believe that viewport/screenshot size issues could be tried to looked up on the cypress repository as well.

Regarding mismatch between local and ci screenshots please provide additional data in the new issue you create - example screenshots or at least description what's different between them. And also - elaborate please on what does does it mean to "match locally uploaded screenshots to match the ones in the ci pipeline"? How do you try to do such matching and what do you want to achieve as a result?
Cheers 👋

[oguzhantx]

`@

Hey @oguzhantx! Can you please open up a new bug request for each of the issues you've mentioned? 🙏 I believe that viewport/screenshot size issues could be tried to looked up on the cypress repository as well.

Regarding mismatch between local and ci screenshots please provide additional data in the new issue you create - example screenshots or at least description what's different between them. And also - elaborate please on what does does it mean to "match locally uploaded screenshots to match the ones in the ci pipeline"? How do you try to do such matching and what do you want to achieve as a result? Cheers 👋

I was able to fix the first screenshot problem. looks like setting up the viewport in config file or beforeeach() block does not set the screenshot size by itself. For the screenshot size, I had to tell screenshot to use viewport as well. so this worked --> Cypress.Screenshot.defaults({ capture: 'viewport' });

When I say mismatch, I meant uploading the base screenshots from your local to repo then you run your tests in ci. then test fails in ci because screenshots that are taken on my machine is different than the ones in the ci. Here is the sample failure screenshot. (blocked some content)

[FRSgit]

@oguzhantx Great to hear that you've been able to fix the first issue. I've added info about that to the README - could you review the PR to check if I got it correctly please?

Regading the mismatch - screenshot done locally will almost always be different than the one on the CI. There are differences in the OS (e.g. every OS renders fonts a bit differently), in headless vs windowed mode and other things. The main flow for working with the screenshots is to generate & compare them on the CI (and maybe generate & compare other copy of screenshots for better developer experience). If you feel that there is still something left regarding this topic - please open up a new issue as it's not connected with the one created by @dhulme

[FRSgit]

Hey @dhulme!
I've just created new PR resolving this issue: #108. I've published it to npm under 2.0.0-alpha.0, can you give it a try please? 🙏

[dhulme]

Hi @FRSgit, sorry for the delay. That's working great, thanks!

[FRSgit]

Uf, I started wondering what should I do with this PR 😄 Thanks for confirming that it works @dhulme!
I'll do a full 2.0.0 release later today

[FRSgit]

Released 🎉

[idasilva1]

@oguzhantx Great to hear that you've been able to fix the first issue. I've added info about that to the README - could you review the PR to check if I got it correctly please?

Regading the mismatch - screenshot done locally will almost always be different than the one on the CI. There are differences in the OS (e.g. every OS renders fonts a bit differently), in headless vs windowed mode and other things. The main flow for working with the screenshots is to generate & compare them on the CI (and maybe generate & compare other copy of screenshots for better developer experience). If you feel that there is still something left regarding this topic - please open up a new issue as it's not connected with the one created by @dhulme

@FRSgit I'm actually having the same issue running cypress in headless mode but changing monitors.
If I run the matchImage for the first time, in headless, using one monitor, when I run it with a different monitor, even in headless, the snapshot fails with just some fonts mismatching. Any idea what this could be?

[FRSgit]

@idasilva1 unfortunately the screen density has an effect on the rendering of a webpage - which might be causing the test to fail :(
Also, it's possible to force the same resolution for different screens/modes only in chrome testing as the only solution to this issue works for Chrome-only.

Cypress says this issue was fixed in 8.0.0, but well - not completely as can be seen here, here and here.

From my experience I would propose to:

• setup capture: 'viewport' configuration (more details in our FAQ section),
• always try to make a screenshot of a part of your screen (never make a screenshot of a full page with cy.matchImage() as the size of the cypress window might differ).