Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: flaky system-tests-firefox job #18848

Merged
merged 10 commits into from
Nov 15, 2021

Conversation

davidmunechika
Copy link
Contributor

@davidmunechika davidmunechika commented Nov 9, 2021

User facing changelog

  • Improved the warning message for retrying connection to a browser

Additional details

  • filtered out the warning message for the Firefox browser in the stdout of the system tests (Firefox is known to occasionally take longer than normal to connect).
  • Increased the padding in the screenshot spec to better account for slower machines and prevent flakiness.
  • Filter out the noisy stdout related to Firefox taking a long time to connect in system-tests.ts (similar to what we do for video snapshot errors in Firefox)
  • Increase the padding buffer for slow machines for screenshot test

PR Tasks

  • Have tests been added/updated?
  • Has the original issue or this PR been tagged with a release in ZenHub?
  • [n/a] Has a PR for user-facing changes been opened in cypress-documentation?

@cypress-bot
Copy link
Contributor

cypress-bot bot commented Nov 9, 2021

Thanks for taking the time to open a PR!

@cypress
Copy link

cypress bot commented Nov 9, 2021



Test summary

18680 0 202 7Flakiness 5


Run details

Project cypress
Status Passed
Commit 0b28562
Started Nov 15, 2021 2:08 PM
Ended Nov 15, 2021 2:20 PM
Duration 11:35 💡
OS Linux Debian - 10.10
Browser Multiple

View run in Cypress Dashboard ➡️


Flakiness

commands/net_stubbing_spec.ts Flakiness
1 network stubbing > waiting and aliasing > can timeout waiting on a single request using "alias.request"
2 network stubbing > waiting and aliasing > can timeout waiting on a single request using "alias.request"
commands/navigation_spec.js Flakiness
1 src/cy/commands/navigation > #visit > window immediately resolves and doesn't reload when visiting the same URL with hashes
cypress/proxy-logging-spec.ts Flakiness
1 Proxy Logging > request logging > xhr log has response body/status code
e2e/redirects_spec.js Flakiness
1 redirection > meta > binds to the new page after a timeout

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@davidmunechika davidmunechika self-assigned this Nov 10, 2021
@davidmunechika davidmunechika marked this pull request as ready for review November 10, 2021 18:43
system-tests/lib/system-tests.ts Outdated Show resolved Hide resolved
system-tests/lib/system-tests.ts Outdated Show resolved Hide resolved
system-tests/lib/system-tests.ts Outdated Show resolved Hide resolved
@davidmunechika davidmunechika requested a review from a team as a code owner November 12, 2021 14:55
@davidmunechika davidmunechika requested review from jennifer-shehane and removed request for a team November 12, 2021 14:55
davidmunechika and others added 2 commits November 12, 2021 10:07
Co-authored-by: Zach Bloomquist <github@chary.us>
@flotwig
Copy link
Contributor

flotwig commented Nov 12, 2021

@davidmunechika this needs to be tagged w/ release and have a changelog, since we are changing the stdout, it is user-facing

@jennifer-shehane jennifer-shehane merged commit 941cef1 into develop Nov 15, 2021
@jennifer-shehane jennifer-shehane deleted the issue-18644-flaky-system-tests-firefox branch November 15, 2021 14:44
tgriesser added a commit that referenced this pull request Nov 18, 2021
* develop: (329 commits)
  chore: Update Chrome (stable) to 96.0.4664.45 (#18931)
  fix: Loading of specs with % in the filename (#18877)
  chore: refactor `create` into class `$Cy` (#18715)
  chore: Update Chrome (beta) to 96.0.4664.45 (#18891)
  fix: flaky `system-tests-firefox` job (#18848)
  release 9.0.0
  feat: ensure major release
  have conduit app wait on localhost:3000
  fix install-required-node
  use --legacy-peer-deps
  feat: ensure major release
  fix darwin node install
  chore(driver): fix integration test retry configuration (#18643)
  feat(deps): update dependency electron to v15 🌟 (#18317)
  chore: Bind this correctly when setting response headers with cy.route() (#18859)
  feat: create config package for config validation (#18589)
  chore: patch `winston` to suppress `padLevels` warning (#18824)
  chore: test out major release build
  fix: remove outdated npm registry links (#18727)
  fix: Adding an existing command with `Cypress.Commands.add()` will throw an error (#18587)
  ...
tgriesser added a commit that referenced this pull request Nov 20, 2021
* develop: (52 commits)
  feat: use hoisted yarn install in binary build (#17285)
  fix: compile npm packages for node 12 (#18989)
  fix: show call count even if `cy.stub().log(false)`. (#18907)
  chore: Update TypeScript to 4.4.4 (#18930)
  fix: wrap playground selectors in double quotes if not included (#18442)
  fix: flaky settings_spec test (#18979)
  chore: Update Chrome (stable) to 96.0.4664.45 (#18931)
  fix: Loading of specs with % in the filename (#18877)
  chore: refactor `create` into class `$Cy` (#18715)
  chore: Update Chrome (beta) to 96.0.4664.45 (#18891)
  fix: flaky `system-tests-firefox` job (#18848)
  chore: release @cypress/webpack-preprocessor-v5.10.0
  chore: release @cypress/vue-v3.0.5
  chore: release @cypress/schematic-v1.6.0
  chore: release create-cypress-tests-v1.2.0
  release 9.0.0
  feat: ensure major release
  have conduit app wait on localhost:3000
  fix install-required-node
  use --legacy-peer-deps
  ...
tgriesser added a commit that referenced this pull request Nov 21, 2021
* 10.0-release: (56 commits)
  chore: post-merge cleanup
  feat: use hoisted yarn install in binary build (#17285)
  fix: fix spec list header, "Create specs" prompt, add workspace recommended apollo extension (#18993)
  feat(unify): reporter settings (#18946)
  feat: add devServer to config file (#18962)
  fix: compile npm packages for node 12 (#18989)
  fix: show call count even if `cy.stub().log(false)`. (#18907)
  chore: Update TypeScript to 4.4.4 (#18930)
  fix: wrap playground selectors in double quotes if not included (#18442)
  fix: flaky settings_spec test (#18979)
  chore: Update Chrome (stable) to 96.0.4664.45 (#18931)
  fix: Loading of specs with % in the filename (#18877)
  chore: refactor `create` into class `$Cy` (#18715)
  chore: Update Chrome (beta) to 96.0.4664.45 (#18891)
  fix: flaky `system-tests-firefox` job (#18848)
  chore: release @cypress/webpack-preprocessor-v5.10.0
  chore: release @cypress/vue-v3.0.5
  chore: release @cypress/schematic-v1.6.0
  chore: release create-cypress-tests-v1.2.0
  release 9.0.0
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix flaky system-tests-firefox job
5 participants