False positives: Cypress passes tests that use the same timeout option object in specific sequence.

[beatrichartz]

Current behavior:

Cypress produces false positives if one test contains a long timeout which is fed in via a variable. Below is a test to reproduce, which fails to produce a failure for the second test on https://google.com.

Desired behavior:

Fail the last contains test.

Steps to reproduce: (app code and test code)

describe('Cypress bug', () => {
  const LONG_TIMEOUT_OPTION = {
    timeout: 10000,
  }

  beforeEach(() => {
    cy.viewport('macbook-15')
  })

  it('should visit Google', () => {
    cy.visit('/')
    cy.contains('About').click()
    cy.contains(
      'Our mission is to organise the world’s information and make it universally accessible and useful.', LONG_TIMEOUT_OPTION)
    cy.contains('See what we build').scrollIntoView().click({ force: true })
    cy.contains(
      'Learn more', LONG_TIMEOUT_OPTION)
  })

  it('should visit Google', () => {
    cy.visit('/')
    cy.contains('About').click()
    cy.contains('Something that the page does not contain', LONG_TIMEOUT_OPTION)
  })
})

Versions

uname -a
Darwin angel 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64

Cypress version 3.6.0
Electron version 73
Chrome version Version 74.0.3729.157 (Official Build) (64-bit)

[jennifer-shehane]

I'm not able to recreate this behavior exactly. Although, this does highlight that the options Object is being mutated as it is passed along.

My example below demonstrates logging of the LONG_TIMEOUT_OPTION.

describe('Cypress bug', () => {
  const LONG_TIMEOUT_OPTION = {
    timeout: 10000,
  }

  it('should visit Google', () => {
    cy.visit('http://www.google.com')
    cy.contains('About').click().then(() => {
      cy.log(LONG_TIMEOUT_OPTION)
    })
    cy.contains(
      'Our mission', LONG_TIMEOUT_OPTION)
  })

  it('should visit Google too', () => {
    cy.visit('https://www.google.com')
    cy.contains('About').click().then(() => {
      cy.log(LONG_TIMEOUT_OPTION)
    })
    cy.contains('Something that the page does not contain', LONG_TIMEOUT_OPTION)
  })
})

In the first log, it logs:

In the second log, it logs:

This looks to be this issue: #3171

---

I can't recreate the original issue however, so I will need more information or a reproducible example. Perhaps the fact that Google's website has changed has some effect on it because I couldn't run the original example exactly.

Please comment with a reproducible example and we will consider reopening.

[beatrichartz]

Thanks for having a look - I've created a repo here: https://github.com/beatrichartz/cypress-false-positive

I've been able to reproduce this with the latest version of Cypress - let me know if you have trouble reproducing it using the repo.

[jennifer-shehane]

Yeah, I wasn't able to reproduce this in either cypress run or cypress open. Followed the instructions in the repo when running. In my example, the first test always failed to find the content you were trying to find.

I VPN'd to Atlanta because initially google was not even displayed in English for me, but there is no text on the page for 'See what we build' in the about page. I then tried VPN'ing into Sydney since that seems to be where you are from and now the correct content is showing since Google's website is largely generated based off of locale.

So, this finally recreates the issue although I would prefer a simple index.html file to test against since at any time Google could change their site and we're left with no reproducible example again.

Something is very oddly specific about this example. I tried to remove several different steps and can't reduce the example any further.

// You must VPN into Sydney, Australia
describe('Cypress bug', () => {
  const LONG_TIMEOUT_OPTION = {
    timeout: 10000,
  }

  it('should visit Google', () => {
    cy.visit('https://www.google.com')
    cy.contains('About').click()
    cy.contains(
      'Our mission', LONG_TIMEOUT_OPTION)
    cy.contains('See what we build', LONG_TIMEOUT_OPTION).scrollIntoView().click({ force: true })
    cy.contains(
      'Learn more', LONG_TIMEOUT_OPTION)
  })

  it('should visit Google 2', () => {
    cy.visit('https://www.google.com')
    cy.contains('About').click()
    cy.contains('Something that the page does not contain', LONG_TIMEOUT_OPTION)
  })
})

I haven't been able to reproduce outside of Google from my easy example.

<!DOCTYPE html>
<html>
<body>
  <button>About</button>
  <p>Our mission</p>
  <div>Learn more</div>
  <p>See what we build</p>
</body>
</html>

describe('Cypress bug', () => {
  const LONG_TIMEOUT_OPTION = {
    timeout: 10000,
  }

  beforeEach(() => {
    cy.viewport('macbook-15')
  })

  it('should visit Google', () => {
    cy.visit('index.html')
    cy.contains('About').click()
    cy.contains(
      'Our mission', LONG_TIMEOUT_OPTION)
    cy.contains('See what we build').scrollIntoView().click({ force: true })
    cy.contains(
      'Learn more', LONG_TIMEOUT_OPTION)
  })

  it('should visit Google', () => {
    cy.visit('index.html')
    cy.contains('About').click()
    cy.contains('Something that the page does not contain', LONG_TIMEOUT_OPTION)
  })
})

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[cypress-app-bot]

This issue has been closed due to inactivity.