Jest 24 is slower than Jest 23.6.0 on the same test suite #7811
Comments
vitalibozhko
changed the title
|
Would be great if you could attach a repro that one can checkout and analyze. |
|
We're also seeing quite significant performance regression for Jest 24. This is with 1400 tests on a large code base. Running a custom typescript transformer.
|
|
I'd love to provide repo for a faster resolution but as most people here i can't share private projects. If someone could advise an open source project that uses jest for tests we could tweak it out and see if it happens there. |
|
Here's an open source I built with jest for testing. Runs both traditional units tests and integration tests with live API calls. I definitely noticed significant slow downs when trying to upgrade to jest@24. Unit tests and integration tests typically run in ~10s each, so should be easy to quickly do some verification. https://github.com/mkreiser/ESPN-Fantasy-Football-API |
|
To be more useful, I ran some stats. Unfortunately, ESPN's API appears to be down so I could not run the integration tests (which could be more interesting) as well. env (via
|
| PASSED OPTIONS | TIME |
|---|---|
| (no options) | 3.5s |
| --maxWorkers=2 | 3.4s |
| --no-cache | 8.6s |
| --no-cache --maxWorkers=2 | 6.8s |
| --runInBand | 3.8s |
| --runInBand --no-cache | 8.3s |
jest@24.1:
| PASSED OPTIONS | TIME |
|---|---|
| (no options) | 4.4s |
| --maxWorkers=2 | 4.7s |
| --no-cache | 13.4s |
| --no-cache --maxWorkers=2 | 17.6s |
| --runInBand | 5.1s |
| --runInBand --no-cache | 9.3s |
|
Checked on this repo tests https://github.com/LambdaSchool/React-Testing, results: v24 Verbose mode does not produce consistent timings though, however in non-verbose v24 is on average 1 second slower than v23 (3.9s vs 4.9s on my machine). |
|
@thymikee Can we remove the Windows-tag here? This is definitely present on MacOS and Ubuntu as well. |
|
Jest v24: Jest v23: |
|
Woah, 17s vs 113s, that's huge! Any possibility of sharing that project? I think we need to start really considering adding performance monitoring to Jest. @aaronabramov talked about this in #4471. Node comes with essentially the same performance API the browser does (available since node 8): https://nodejs.org/api/perf_hooks.html, maybe we can use that to measure? Having some way of emitting how long different things takes should be a first step probably (so #4471, but that only talks about what happens inside of tests) |
|
Sure, it is this repo. Currently I do not want to upgrade to v24 due to this difference 😞 |
|
Thanks! On my machine, that takes 35-40 sec on a cold cache, and 7-8 sec on a warm cache. With jest 24, I get 40-45 sec on cold cache and 17-18 sec on warm cache. Definitely shows the problem (although not as severe as on your machine). I'll try to find some time digging into this
Understandable! |
|
Maybe I ran v23 with cache and v24 without 🤔 . But we also have another repo where the tests got noticeably slower after upgrading. I don't have any official timings though. |
|
I suspect Babel 7 significantly contributes to the issue, check out the commit |
|
Tested with
The percentage difference between hot and cold cache is also much greater
Even worse is the difference between
I aborted the --no-cache run on 24.1 after 400 seconds. With 2669/5749 tests 113 / 621 suites done. So the total execution time is probably in the 800-900 seconds range. Edit: Jest 24.8.0 was a big improvement. Slightly faster run time despite more tests. |
|
Is there someone working on this as there is a clear repro? |
|
There is #8032 which has identified one cause of regression |
|
We've recently switched from Babel 6/Jest 23 to Babel 7/Node 24 and have noticed similarly stark speed regressions (from ~150s to ~200s with a clean cache and ~100s to ~160s cached). Doesn't appear to be Babel 7 related, since running Babel compilation by itself is actually faster now. |
|
Hi There, I would like to share my experience here with Jest. PASS test/domain/model/Actor.test.ts (12.193s)
Actor creation
√ should create an actor (5ms)
√ should create an actor will publish ActorCreated event (3ms)
Actor roles
√ should create a role and to be an instance of it (15ms)
√ should publish CreatorRoleAssignedToActor event when add a creator role to an actor (2ms)
√ should publish UserRoleAssignedToActor event when add a user role to an actor (5ms)
√ should create a user actor and publish UserRoleAssignedToActor domain event (1666ms)
Test Suites: 1 passed, 1 total
Tests: 6 passed, 6 total
Snapshots: 0 total
Time: 13.4s
Ran all test suites.
Watch Usage: Press w to show more.npx envinfo --preset jest
npx: installed 1 in 2.847s
System:
OS: Windows 10
CPU: (8) x64 Intel(R) Core(TM) i7-4700HQ CPU @ 2.40GHz
Binaries:
Node: 11.7.0 - C:\Program Files\nodejs\node.EXE
npm: 6.5.0 - C:\Program Files\nodejs\npm.CMDAlso in the package.json "scripts": {
"test": "jest",
"test:watch": "jest --watch",
},
"devDependencies": {
"@types/jest": "^24.0.12",
"jest": "^24.5.0",
"jest-coverage-badges": "^1.1.2",
"jest-junit": "^6.4.0",
"ts-jest": "^24.0.2",
...etc
}jest.config.js module.exports = {
preset: "ts-jest",
testEnvironment: "node",
coverageDirectory: "doc/code-coverage",
collectCoverageFrom: [
"src/**/*.{js,ts}",
"!**/node_modules/**",
"!**/vendor/**",
"!src/index.ts"
],
coverageReporters: ["json", "html", "json-summary", "jest-junit", "lcov"]
};
I'm using it with typescript: in my Then I tried to use mocha npm i -D chai @types/chai mocha @types/mocha sinon @types/sinon sinon-chai @types/sinon-chai source-map-support nycand I put mocha.opts inside 'test' folder --require ts-node/register
--watch-extensions ts
test/**/*.tsand in the package.json: "scripts": {
"test:watch": "mocha --opts test/mocha.opts --watch"
},
...etcI ran the same tests (only I modifed the assertions to use sinon-chai), and it ran amazingly fast (about 1-2 second(s)), Actor creation
√ should create an actor
√ should create an actor will publish ActorCreated event
Actor roles
√ should create a role and to be an instance of it
√ should publish CreatorRoleAssignedToActor event when add a creator role to an actor
√ should publish UserRoleAssignedToActor event when add a user role to an actor
√ should create a user actor and publish UserRoleAssignedToActor domain event (325ms)
6 passing (336ms)Something is not good, and I don't know what is the problem. |
|
Anything we can do regarding this? Any approach advice? |
|
Hopefully today. We just need to make sure it works for FB (best test nothing is broken that exists 🙂) You can use diff --git a/node_modules/@jest/transform/build/ScriptTransformer.js b/node_modules/@jest/transform/build/ScriptTransformer.js
index 8b02912..2191456 100644
--- a/node_modules/@jest/transform/build/ScriptTransformer.js
+++ b/node_modules/@jest/transform/build/ScriptTransformer.js
@@ -199,7 +199,7 @@ const {version: VERSION} = require('../package.json'); // This data structure is
// we need to transform a file. Since ScriptTransformer is instantiated for each
// file we need to keep this object in the local scope of this module.
-const projectCaches = new WeakMap(); // To reset the cache for specific changesets (rather than package version).
+const projectCaches = new Map(); // To reset the cache for specific changesets (rather than package version).
const CACHE_VERSION = '1';
@@ -224,16 +224,17 @@ class ScriptTransformer {
this._config = config;
this._transformCache = new Map();
this._transformConfigCache = new Map();
- let projectCache = projectCaches.get(config);
+ const configString = (0, _fastJsonStableStringify().default)(this._config);
+ let projectCache = projectCaches.get(configString);
if (!projectCache) {
projectCache = {
- configString: (0, _fastJsonStableStringify().default)(this._config),
+ configString,
ignorePatternsRegExp: calcIgnorePatternRegExp(this._config),
transformRegExp: calcTransformRegExp(this._config),
transformedFiles: new Map()
};
- projectCaches.set(config, projectCache);
+ projectCaches.set(configString, projectCache);
}
this._cache = projectCache; |
|
@SimenB this patch is to which version of jest? 25.0.0, or 24.9.0? |
diff --git a/node_modules/@jest/transform/build/ScriptTransformer.js b/node_modules/@jest/transform/build/ScriptTransformer.js
index 0dbc1d7..b595ec6 100644
--- a/node_modules/@jest/transform/build/ScriptTransformer.js
+++ b/node_modules/@jest/transform/build/ScriptTransformer.js
@@ -207,7 +207,7 @@ const _require = require('../package.json'),
// we need to transform a file. Since ScriptTransformer is instantiated for each
// file we need to keep this object in the local scope of this module.
-const projectCaches = new WeakMap(); // To reset the cache for specific changesets (rather than package version).
+const projectCaches = new Map(); // To reset the cache for specific changesets (rather than package version).
const CACHE_VERSION = '1';
@@ -239,16 +239,17 @@ class ScriptTransformer {
this._config = config;
this._transformCache = new Map();
this._transformConfigCache = new Map();
- let projectCache = projectCaches.get(config);
+ const configString = (0, _fastJsonStableStringify().default)(this._config);
+ let projectCache = projectCaches.get(configString);
if (!projectCache) {
projectCache = {
- configString: (0, _fastJsonStableStringify().default)(this._config),
+ configString,
ignorePatternsRegExp: calcIgnorePatternRegExp(this._config),
transformRegExp: calcTransformRegExp(this._config),
transformedFiles: new Map()
};
- projectCaches.set(config, projectCache);
+ projectCaches.set(configString, projectCache);
}
this._cache = projectCache; |
|
A new release of 25 will come next week. You can build from master now and test it out if you want. After the release, if we still have regressions vs Jest 24, please open up a new issue. This issue has more than 100 comments, so it's getting a bit unwieldy... Things we've not yet tried that has been mentioned in this issue is to bundle up our modules to hit the FS less and look into what we do with stack traces. Thank you to everyone who helped track this down! |
|
@SimenB You said "A new release of 25 will come next week. " Any news about the release? thanks |
|
Last I heard it was being tested at FB on Friday. |
|
Would it be possible to release a patch to version 24 while we wait for the version 25 release? |
|
Hi @SimenB ! Thanks for the patch for 24.9! I'm trying to use However, when I run Any pointer for this situation? Thanks a lot! |
|
Happy Friday everyone! Just wondering if anyone has tips on how to patch |
|
any progress on releasing this? |
|
Quick update on #7811 (comment) and #7811 (comment), @ds300 said we just had to do: Can confirm SimenB's patch works for 24.9!! |
Any update on this by any chance? |
|
All I want for Christmas is y...Jest 25 <3 |
|
If Jest 25 is delayed for some reason, would it be possible to release a patch for 24.9 with the fix? @SimenB |
|
For the record, jest@next downloaded on Nov 22nd was extremely slow, but I redownloaded now and it looks much much faster (sorry for the imprecision) |
|
@dpinol curious which version you ended up installing that worked? The version on the npm registry tagged as @SimenB is there any version of jest (either v25+ or a patched v24) that's available on the registry currently which includes this fix? While using something like patch-package is possible, I'd prefer not to fork or build a custom version given that the fix does already exist in master. If not, perhaps the simplest option from your end would be to update the |
|
For people who want to use Steps:
"scripts": {
+ "postinstall": "patch-package"
}
|
|
When is this going to be in a release? |
|
Ran 24.9 vs 25.1 on a local test suite. Amazing work @SimenB and friends! Jest 25.1 Cold: Test Suites: 3 skipped, 218 passed, 218 of 221 total Warm: Test Suites: 3 skipped, 218 passed, 218 of 221 total Jest 24.9 Cold: Test Suites: 3 skipped, 218 passed, 218 of 221 total Warm: Test Suites: 3 skipped, 218 passed, 218 of 221 total |
|
I was just about to post that a stable Jest 25 was just released, and you've already tested it 😀 Fantastic numbers, thanks for sharing @csvan! |
|
I'm not sure if this should be filed as a separate issue at this point, but sadly, 25.1.0 doesn't seem to have resolved the slowdown for us. We skipped version 24 because of the slowdowns we were seeing from 23.4.2. For the whole application (130 test files, 1104 tests) I see tests go from taking 17-18 seconds in v23 to taking 23-24 seconds in v25. Running the slowest test file in our application on its own went from taking 6 seconds in v23 to 8 seconds in v25. |
|
Interesting, would you be able to post a reproduction in a new issue? This issue in particular was about a specific bug introduced in Jest 24 which has been fixed in #8890 |
|
There's a new issue for Jest 25 performance issues: #9457 |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
and others
🐛 Bug Report
Filing bug per #7732 #7732 (comment)
Jest v24 takes 38 seconds vs v23 28 seconds to run 50 suites (296 tests).
To Reproduce
Steps to reproduce the behavior:
Jest v23 setup:
.babelrc
package.json:
Jest v24 setup:
.babelrc
package.json:
Jest config is not changed between v23 and v24:
Expected behavior
Jest v24 running performance matches v23.
Link to repl or repo (highly encouraged)
Not available
Run
npx envinfo --preset jestPaste the results here:
System: OS: Windows 7 CPU: (4) x64 Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz Binaries: Node: 8.12.0 - C:\Program Files\nodejs\node.EXE npm: 6.4.1 - C:\Program Files\nodejs\npm.CMDThe text was updated successfully, but these errors were encountered: