-
-
Notifications
You must be signed in to change notification settings - Fork 6.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
allow bail
setting to control when to bail out of a failing test run
#7335
Conversation
Interesting idea! I wonder if we could overload |
I like passing a number to |
bailAfter
setting to control when to bail out of a failing test runbail
setting to control when to bail out of a failing test run
docs/WatchPlugins.md
Outdated
@@ -155,7 +155,7 @@ class MyWatchPlugin { | |||
|
|||
For stability and safety reasons, only part of the global configuration keys can be updated with `updateConfigAndRun`. The current white list is as follows: | |||
|
|||
- [`bail`](configuration.html#bail-boolean) | |||
- [`bail`](configuration.html#bail) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does the link work? You can run yarn start
in the website folder to test links etc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe I'm just doing something wrong, but I've tried yarn start
and yarn build
then yarn start
in /website/ and it doesn't seem to be reflecting any of the changes from these *.md files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I was able to find the link in /build
folder. Updated to match now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to open the next
docs, by default it renders the versioned docs. No need to build :)
You can select the version by clicking the version number in the upper left
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This document gives a 404, though, can yiu double check locally?
https://deploy-preview-7335--jest-preview.netlify.com/docs/en/next/watch-plugins
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So the link itself for configuration#bail-number-boolean
seems to work correctly. But the link you posted, as well as when running locally, and going to /next/watch-plugins
both seem to return page not found. Seems to be partially broken in master: https://jestjs.io/docs/en/next/watch-plugins Looks like it's because of original_id
inside the header. I've removed it because I don't see any references to that property outside of versioned docs and removing it seems to fix that link so we can test the updated link.
@SimenB anything else? I believe I've addressed all of the outstanding review comments. |
if (this._globalConfig.bail && aggregatedResults.numFailedTests !== 0) { | ||
if ( | ||
this._globalConfig.bail !== 0 && | ||
aggregatedResults.numFailedTests >= this._globalConfig.bail |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this will still be true if bail
is explicitly set to false
. Shouldn't we normalize it just in case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There shouldn't be any case where bail
gets to this point that isn't a number: https://github.com/facebook/jest/pull/7335/files#diff-7121e12d406ff9a3060a13ee808632de
value = options[key] ? 1 : 0; | ||
} else if (isNaN(options[key])) { | ||
value = 1; | ||
argv._.push(options[key]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can't we just remove this line? looks fishy
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
@rickhanlonii you have anything to add here? if not, feel free to merge :) |
Codecov Report
@@ Coverage Diff @@
## master #7335 +/- ##
==========================================
+ Coverage 67.15% 67.44% +0.29%
==========================================
Files 247 247
Lines 9502 9511 +9
Branches 5 5
==========================================
+ Hits 6381 6415 +34
+ Misses 3119 3094 -25
Partials 2 2
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM just a couple questions!
Thanks @bookman25, I'll merge when we're green |
Thanks @bookman25!! |
* master: (24 commits) Add `jest.isolateModules` for scoped module initialization (jestjs#6701) Migrate to Babel 7 (jestjs#7016) docs: changed "Great Scott!" link (jestjs#7524) Use reduce instead of filter+map in dependency_resolver (jestjs#7522) Update Configuration.md (jestjs#7455) Support dashed args (jestjs#7497) Allow % based configuration of max workers (jestjs#7494) chore: Standardize filenames: jest-runner pkg (jestjs#7464) allow `bail` setting to control when to bail out of a failing test run (jestjs#7335) Add issue template labels (jestjs#7470) chore: standardize filenames in e2e/babel-plugin-jest-hoist (jestjs#7467) Add node worker-thread support to jest-worker (jestjs#7408) Add `testPathIgnorePatterns` to CLI documentation (jestjs#7440) pretty-format: Omit non-enumerable symbol properties (jestjs#7448) Add Jest Architecture overview to docs. (jestjs#7449) chore: run appveyor tests on node 10 chore: fix failures e2e test for node 8 (jestjs#7446) chore: update docusaurus to v1.6.0 (jestjs#7445) Enhancement/expect-to-be-close-to-with-infinity (jestjs#7444) Update CHANGELOG formatting (jestjs#7429) ...
* standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * standardizing jest-runner filenames to fit facebook standards * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * Add issue template labels (#7470) * Add issue template labels * Lint * standardizing jest-runner filenames to fit facebook standards * allow `bail` setting to control when to bail out of a failing test run (#7335) * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names in e2e/__test__ * standardizing file names in compare-dom-nodes * standardizing file names for coverage-report and coverage-remap * updating case sensitive filenames * renaming files to fit facebook standards * adding case sensitive file changes * resolving dependency issues after renaming case sensitive files. * standardizing file names to fit facebook standards * adjusting file names and snapshots to fit facebook standards * adjusting file names to fit facebook standards * resolves bug with nodeAssertionError not being found by test suite * changing references to match file name * changing file name to check test-jest-circus * readding a previously deleted snapshot * Update e2e/bad-source-map/__tests__/badSourceMap.js comment to reflect new file name Co-Authored-By: GGonryun <amodestduck@gmail.com> * improving file naming conventions letters g & j * changing filename to fit facebook standards * renaming filenames to fit facebook standards for 'L' * renaming filenames to fit facebook standards for 'M' * renaming filenames to fit facebook standards for 'N' * resolving leftover merge conflicts * reverting error in naming by changing filename back to uppercase * renaming filenames 'O' to fit facebook standards. * renaming filenames 'P' to fit facebook standards. * renaming filenames 'R' to fit facebook standards. * renaming filenames 'S' to fit facebook standards. * changing regex to match temporary folder name * renaming filenames 'T' to fit facebook standards. * renaming filenames 'U' to fit facebook standards. * renaming filenames 'V' to fit facebook standards. * renaming filenames 'W' to fit facebook standards. * updating snapshots to reflect standardized file names * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern-temp/package.json * Rename e2e/test-name-pattern/package.json to e2e/test-name-pattern-skipped/package.json * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern.package.json * Rename e2e/test-name-pattern-temp/package.json to e2e/test-name-pattern/package.json * Rename e2e/test-name-pattern.package.json to e2e/test-name-pattern-skipped/package.json * removing left over files and adjusting file names previously missed * deleting more leftover files
* standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * standardizing jest-runner filenames to fit facebook standards * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * Add issue template labels (jestjs#7470) * Add issue template labels * Lint * standardizing jest-runner filenames to fit facebook standards * allow `bail` setting to control when to bail out of a failing test run (jestjs#7335) * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names in e2e/__test__ * standardizing file names in compare-dom-nodes * standardizing file names for coverage-report and coverage-remap * updating case sensitive filenames * renaming files to fit facebook standards * adding case sensitive file changes * resolving dependency issues after renaming case sensitive files. * standardizing file names to fit facebook standards * adjusting file names and snapshots to fit facebook standards * adjusting file names to fit facebook standards * resolves bug with nodeAssertionError not being found by test suite * changing references to match file name * changing file name to check test-jest-circus * readding a previously deleted snapshot * Update e2e/bad-source-map/__tests__/badSourceMap.js comment to reflect new file name Co-Authored-By: GGonryun <amodestduck@gmail.com> * improving file naming conventions letters g & j * changing filename to fit facebook standards * renaming filenames to fit facebook standards for 'L' * renaming filenames to fit facebook standards for 'M' * renaming filenames to fit facebook standards for 'N' * resolving leftover merge conflicts * reverting error in naming by changing filename back to uppercase * renaming filenames 'O' to fit facebook standards. * renaming filenames 'P' to fit facebook standards. * renaming filenames 'R' to fit facebook standards. * renaming filenames 'S' to fit facebook standards. * changing regex to match temporary folder name * renaming filenames 'T' to fit facebook standards. * renaming filenames 'U' to fit facebook standards. * renaming filenames 'V' to fit facebook standards. * renaming filenames 'W' to fit facebook standards. * updating snapshots to reflect standardized file names * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern-temp/package.json * Rename e2e/test-name-pattern/package.json to e2e/test-name-pattern-skipped/package.json * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern.package.json * Rename e2e/test-name-pattern-temp/package.json to e2e/test-name-pattern/package.json * Rename e2e/test-name-pattern.package.json to e2e/test-name-pattern-skipped/package.json * removing left over files and adjusting file names previously missed * deleting more leftover files
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Summary
Sometimes when running a full suite of tests there might be a config change (or some other failure) that leads to a catastrophic failure. Instead of continuing to run the entire suite, have an early exit after say
15
failures, something must be seriously wrong so abort the run.Typically in CI mode we'll want to try to let the tests run so we can see all the failures at one time (which is why bail is not quite enough). But in the event of everything failing there's no need to continue running. We had a case where a branch of code was generating 100GB worth of logs because of test failures, with this change it would have been able to early abort.
Test plan
Unit tests