Taking screenshots during cypress open versus cypress run result in different screenshot sizes

[caaldrid]

Current behavior:

Screenshots are not taken at the height and width set prior for the viewport even when using the setting the capture to be the viewport. This interestingly only happens if you trigger the test from the runner and not if you trigger the test from terminal.

Desired behavior:

For the screenshot height and width to match the height and width that the viewport is set to both when the test is triggered by the runner and when the test is triggered in the terminal.

Steps to reproduce: (app code and test code)

• set code in test to take screenshot like so:

   const visaulDiffWidth = 1280;
   const visualDiffHight = 720;
   
   options.capture = "viewport";
   
   cy.viewport(visaulDiffWidth, visualDiffHight);
   
   cy.wrap(subject, { log: false })
     .screenshot(screenshotTitle, options)
     .then(() => {
       cy.viewport(
         Cypress.config("viewportWidth"),
         Cypress.config("viewportHeight")
     );
   
     cy.task("imageMatch", taskData, { log: false });
   });

• Launch cypress
• Start a test that takes a screenshot with the code above manually in the runner. (Screenshot will match whole display width and a height)
• Exit runner
• Start test from command line with yarn cypress run --spec '<path to test file>' (Screenshot will match the viewport)

Versions

Cypress 3.0.0
macOS Mojave 10.14.2
node v8.11.4
yarn 1.5.1

[jennifer-shehane]

Ok, so the base code in question is something like this:

 const visaulDiffWidth = 1280;
 const visualDiffHight = 720;
 
 options.capture = "viewport";
 
 cy.viewport(visaulDiffWidth, visualDiffHight);
 
 cy.document().then((doc) => {
   cy.wrap(subject, { log: false })
     .screenshot(screenshotTitle, options)
     .then(() => {
       cy.viewport(
         Cypress.config("viewportWidth"),
         Cypress.config("viewportHeight")
     );
  })

When you chain .wrap() to .screenshot(), it's essentially telling the screenshot command to restrict the screenshot to the document you pass in. Is there a reason you are doing this as opposed to just using cy.screenshot() by itself?

[Saulis]

@jennifer-shehane I can verify the behaviour regardless of any custom commands being used:

it('test', () => {
  cy.visit('https://www.google.com')

  cy.viewport(1280, 768)

  cy.screenshot('google')
})

Running cypress run with headless Electron produces screenshots in 1280x768 while as running cypress open both Electron (59) and Chrome (74) produce screenshots in 2560x1536.

Setting viewport(1600, 900) produces screenshots in 3200x1800 when running headed. So the screenshots are taken with 2x the viewport size for some reason.

I'm running:
Cypress v3.2.0
MacOS 10.14.4
Yarn 1.12.3
Node v10.15.0

[Lakitna]

Could be display scaling. I don't know any macs that don't do display scaling due to their high pixel density screens.

[giltayar]

Yup. Seeing the same here. Very weird and inconsistent, BTW. Not sure I can see any logic, but sometimes the screenshot is the size of the viewport, and sometimes just the size the window is opened on. Interestingly enough, this sometimes happens in the runner too.

[edelgado]

I'm having this same issue. Running the specs with cypress run generates screenshots that are doubled in size than those generated with cypress open:

Error: Image size (1872x1256) different than saved snapshot size (936x628).

Any ideas? 😕

[edelgado]

@jennifer-shehane do we have any updates on this issue?

Other folks have reported this issue in the cypress-image-snapshot plugin (jaredpalmer/cypress-image-snapshot#67), though it seems this is an issue with Cypress itself.

[stefanoTron]

same here @edelgado maybe @chrisbreiding or @brian-mann could give us a hand here?

[jennifer-shehane]

Hey @edelgado you mentioned that your issue is that during cypress run the screenshots are double the size than cypress open, but the original issue is that cypress open screenshots were double the size of cypress run. So you are experiencing a different issue than is described.

This issue of the screenshots being smaller in cypress run than in cypress open is what this issue was tracking.

I can recreate this behavior from this test:

it('test', () => {
  cy.visit('https://www.google.com')
  cy.viewport(1280, 768)
  cy.screenshot('google')
})

In cypress open - this is the picture that was taken on my Mac Mojave version 10.14.6 which is 2560 × 1536 in width.

If I do cypress run, then the picture taken is set to the max size of the display which is using Xvfb and is set to 1280x720 as explained in this issue - #2102

The issue appears to be in the scaling as described in this comment: #2102 (comment)

I suppose this is technically a separate problem from #2102, but all of this is a larger problem with how we do scaling and display size calculations overall.

[valentindotxyz]

Hi @jennifer-shehane

We do not recommend comparing screenshots across environments (aka comparing screenshots taken in cypress open versus cypress run)

We do recommend comparing screenshots taken during cypress run on your local computer versus screenshots taken during cypress run in a CI environment.

While I totally understand the issue, I find it weird that screenshots took during run differ from the ones took with open. How shall we implement screenshot diff in that case? Prevent screenshots from being taken while in open mode and "force" other developers to use the run command for the "good" screenshots to be taken before pushing their changes?

It's a bit against your fast and easy from your tagline "Fast, easy and reliable testing for anything that runs in a browser." 😔

[ajw015]

I will also state I am hitting this issue now where open and run snapshot differently, and would like to see this resolved.

[PawelWesolowski]

In my case the problem was with the Retina display on Macbook. I was getting exactly twice as big screenshots when I used open (2048 width instead of 1024). Then I moved the runner's Chrome instance to my second screen and voilà... open and run give the same results.

[jennifer-shehane]

@valentindotxyz Yes, this is exactly what we're advocating. During cypress open the browser opens headed by default - it has a literal screen and this screen is whatever the developer's machine is - that is the size the screenshots/videos will be taken.

So I could run my application tests on my Mac at resolution 2880x1800 taking screenshots to save to diff against, then Gleb opens up cypress open tomorrow and runs the screenshots to 'diff' against on his Macbook Air - resolution of 2560×1600. Now we're trying to diff the 2560×1600 sized screenshots against the 2880x1800 size screenshots. That's a problem.

cypress run defaults to running headless - there is no screen, it is imaginary, so you can set the size to whatever you want or use the imaginary size that Cypress has set as the default - this is the size the screenshots will be taken. You could diff screenshots using cypress open if you pass the --headless flag, because really the difference is headed versus headless.

To only take screenshots during headless runs - here is a recipe for this: https://on.cypress.io/browser#Screenshot-only-in-headless-browser

Cypress.Commands.overwrite('screenshot', (originalFn, subject, name, options) => {
  // only take screenshots in headless browser
  if (Cypress.browser.isHeadless) {
    // return the original screenshot function
    return originalFn(subject, name, options)
  }

  return cy.log('No screenshot taken when headed')
})

// only takes in headless browser
cy.screenshot()

[bahmutov]

I would never do something like this to you @jennifer-shehane

[Phenomite]

I resolved this by slightly modifying @0xIslamTaha code. In accordance with the supported args:

In cypress\plugins\index.js

on('before:browser:launch', (browser = {}, launchOptions) => {
    if (browser.name === 'chrome') {
      launchOptions.args.push('--window-size=1440,900');
      return launchOptions
    }
    if (browser.name === 'electron') {
      launchOptions.preferences['width'] = 1440;
      launchOptions.preferences['height'] = 900;
      return launchOptions
    }
  })

Mainly the electron part we are interested in because when operating as a runner it wouldn't respect anything but these preferences.

Perhaps make these a potential cypress environment config.

[timarney]

To only take screenshots during headless runs

I found @cypress/skip-test

Usage:

import { onlyOn } from "@cypress/skip-test";

...

sizes.forEach((size) => {
    pages.forEach((page) => {
      onlyOn("headless", () => {
        it(`it should render "${page}" on "${size}" screen'`, () => {
     
...

Output

[Tara-E]

You could diff screenshots using cypress open if you pass the --headless flag, because really the difference is headed versus headless.

I don't see any way to pass the --headless flag to cypress open. I think it would be pretty awesome if that option were added though. The test runner could open as normal and show all of the pretty results and logs etc, but that live preview section on the right isn't needed.

[yanfengliu]

I think this is the issue related to the retina screen on Macs that scale every screenshot to 2x resolution. Cypress has an official solution: https://docs.cypress.io/api/plugins/browser-launch-api.html#Set-screen-size-when-running-headless

In general, the physical device where you take the screenshot on will have an impact on the screenshot. For example, the color profile of the monitor will affect the actual color rendered.

[zuzusik]

cypress run defaults to running headless - there is no screen, it is imaginary, so you can set the size to whatever you want or use the imaginary size that Cypress has set as the default - this is the size the screenshots will be taken.

@jennifer-shehane I don't think this statement is actually correct in a part where it says so you can set the size to whatever you want - we have explicitly set cy.viewport("macbook-16"); in our code and while it works just great in headed mode, it has zero effect on headless mode - it still runs with 1280 screen width

[Orgenus]

cypress run defaults to running headless - there is no screen, it is imaginary, so you can set the size to whatever you want or use the imaginary size that Cypress has set as the default - this is the size the screenshots will be taken.

@jennifer-shehane I don't think this statement is actually correct in a part where it says so you can set the size to whatever you want - we have explicitly set cy.viewport("macbook-16"); in our code and while it works just great in headed mode, it has zero effect on headless mode - it still runs with 1280 screen width

I had the same problem as you though.

launchOptions.args.push("--window-size=1536,960");

I recommend you to try the command solved it for me.

[BillSchumacher]

The fix that was implemented to not need the launch options did not actually fix the problem and the work-around is still needed but no longer works.

When I set the viewport and then call the screenshot command with viewport specified, it should take a screenshot with the exact viewport size I specified and not limit it to my monitor's resolution/scaling on axis that exceeds that value.

Even if the image is scaled a few pixels in either direction, the size must match.

The behavior seems different across platforms (linux/OSX) as well. Headed and headless mode do not work as intended.

[AmirL]

The --force-device-scale-factor=1 fix works for Chrome (though I have to look at the non-Retina browser when I open it in UI mode).

And there is no such option for electron, which results in huge screenshot sizes on the Mac. (even in headless mode).

Don't you think it would be good to have an option for desirable screenshot sizes?