Very poor performance on basic tests

[cozmy]

🐛 Bug Report

I've created a simple test with no configuration. You can find it here. It takes 7.6 seconds just to start jest, and then, on file change, it takes 5 seconds to re-run.
There's clearly something wrong here.

Mocha and Chai, for example, start in around 4 seconds, but afterward, the re-run is almost instant.

It's weird that Jest reports that the test takes 14ms but shows me the result in 5 seconds.

On top of this, if you add more basic tests, the time just seems to go up.

To Reproduce

Clone my minimal repository, yarn install and yarn test.

Expected behavior

It should be faster than this.

Link to repl or repo (highly encouraged)

https://github.com/cozmy/jest-performance

Run npx envinfo --preset jest

Paste the results here:

  System:
    OS: Windows 10
    CPU: (8) x64 Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz
  Binaries:
    Node: 8.12.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.13.0 - C:\Users\CosminCristian\AppData\Roaming\npm\yarn.CMD
    npm: 6.8.0 - C:\Program Files\nodejs\npm.CMD

[jeysal]

Your repo takes way less time for me:
4ms test case
0.7s according to jest
1.4s according to time
Reruns are pretty much instant.

On Windows, startup performance (of any Node process and especially if you do a lot of IO) will always be noticeably slower. You can try upgrading the Node version to 10 or 11 (8 is quite old), I'd expect a 10-20% speedup coming from that.
I'd try to profile the performance issue a bit, but I can't reproduce the terrible performance on my systems :/

[gsteacy]

Test: 8-10ms
Total: 2.4-2.8s

Reruns are very erratic if the rerun was triggered by a file change:

Test: 2-14ms
Total: 0.8-3.3s (typically on the upper end though, <2.5s is uncommon).

If it was triggered manually then it is pretty consistent with the first run. Interestingly, if I run with --no-cache then the first run is similar, but reruns on file changes are 0.8-0.9s and manual reruns are <0.2s. I think this might be because it throws away the timings of each run and causes Jest to run in-band (no extra workers). I usually run in-band on Windows because it's nearly always the fastest option, at least in watch mode. In WSL the performance is not quite as bad (though well short of native Linux) so I typically let Jest do whatever it wants.

As a side note, my runs seems to have been quite a bit faster than yours despite this machine having pretty dated hardware. That's pretty strange in itself.

  System:
    OS: Windows 8.1
    CPU: x64 Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
  Binaries:
    Node: 8.11.3
    Yarn: 1.7.0
    npm: 5.6.0
  npmPackages:
    jest:
      wanted: ^24.1.0
      installed: 24.1.0

[cozmy]

To give you a bit of background, I was trying to set up the test environment on my work laptop and I noticed that my React tests were way to slow. That's when I've discovered the poor performance. I've also tested this on Linux and it's 2x as fast, but still, 3seconds on re-run for some basic tests it's too much.

It's weird that Jest reports that a test runs in 10ms but the reported time is 5seconds.

Even on my work laptop, running the tests on create-react-app are slow, on re-run:

I guess this issue is a duplicate of #7619 and #6694 so most likely it can be closed.

[fabiosantoscode]

Mocha doesn't take 4 seconds to initialise in this case. I changed your test to do require('assert').equal(1, 112) and it took less than 200ms:

fabio@fabio-thinkpad ♥  time mocha index.spec.js 


  jest performance
    1) should be fast


  0 passing (6ms)
  1 failing

  1) jest performance
       should be fast:

      AssertionError [ERR_ASSERTION]: 1 == 112
      + expected - actual

      -1
      +112
      
      at Context.<anonymous> (index.spec.js:3:23)




real    0m0.173s
user    0m0.153s
sys     0m0.024s

Which by the way, if you add another test it doesn't take 173ms * 2 to run.

I've never seen a testing framework that takes more than 500ms to perform a single equality check. There must be some configuration option we can set here, and if there is, I would like to make the case for it to be in the defaults.

[fabiosantoscode]

Just copied the original test 50 times to make sure. Results:

real    0m0.193s
user    0m0.157s
sys     0m0.040s

[natealcedo]

Tests run fine for me.

[jeysal]

I tried this out on my Mac and got run times of 3-4s, so a bit closer to what you are describing. There is nothing that takes way longer than expected in the profile though, as I said it's just that IO performance is very OS-dependent, most of this is require loading modules:

One thing caught my eye in the flame chart though

Setting up a JSDOM environment is very expensive (the middle part of the chart). Setting testEnvironment to node if the tests don't need it cut the test time in half for me. I find it a bit unfortunate that the default is jsdom. @SimenB has there been any discussion on the default testEnvironment?

[jeysal]

Also, I noticed that watch mode is not running inBand, which I think it should in this case. After setting the test environment to node and passing --watchAll --runInBand reruns are pretty much instant. I'll try to find out by Jest doesn't decide to runInBand automatically in this case.

[SimenB]

@SimenB has there been any discussion on the default testEnvironment?

Yes, #5815. If you use jest --init, it defaults to node as well. I'd forgotten #5815 was accepted, would be nice to get done for 25.

[jeysal]

After some investigation, the reason why Jest does not run the test inBand is the following:

• On the first run, we do not know whether the one test will be fast, so we decide not to runInBand in order to keep the main thread unblocked and make sure the TTY remains responsive
• Because this first run is not inBand, the overhead causes it to take about 2.2s on my device (even without JSDOM)
• On rerun, Jest sees that the test took longer than 1s (the current SLOW_TEST_TIME threshold), so it again decides not to runInBand
• You will never get out of this until you force runInBand once (and do one rerun, because the first runInBand execution also has to load everything which is slow), which causes the test time to go down to <100ms

I don't have a good solution to this other than making the threshold more generous right now

[SimenB]

Because this first run is not inBand, the overhead causes it to take about 2.2s on my device (even without JSDOM)

Can you test with node worker threads? Still some overhead, but maybe not so much

[jeysal]

@SimenB no noticeable change. jasmine2() has 1.5s total time when it initially requires all the modules, it's not really the worker overhead itself. In band the modules will be loaded after the first run so subsequent runs are almost instant.

[jeysal]

(jest-circus is not significantly faster to load either)

[jeysal]

There's no mechanism in place to reuse workers across watch runs, is there? Would it be unsafe to do so in case there is some kind of leftover state in the worker?

[SimenB]

Workers are always reused, do you mean the sandbox? If so, no - we want every test to run in isolation

However, we want to lift jasmine/circus out of the sandbox and inject some integration points. Might help?

[jeysal]

Workers are always reused

That's not what I'm seeing though, this is the first part of a rerun:

[jeysal]

This is (like some other issues mentioned) definitely a regression introduced in 23.0.0
In 22.4.4, jasmine2() only takes about 600ms (less than half the time)

[SimenB]

That should be fixed - the worker pool should be reused.

Could you bisect?

[jeysal]

Will do, but it's not what causes the performance issue here. After quite a bit of profiling I've determined the culprit to be micromatch@3. I'll open a PR where we can discuss how to handle this.

[SimenB]

I've determined the culprit to be micromatch@3

Really? Interesting! Looking forward to the analysis 🙂

[cozmy]

Still no conclusion? :(

[jeysal]

See #8032 / #8046 for the latest info

[Thristhart]

any updates now that micromatch@4 has released?

[SimenB]

Seeing as it's a major, we'll probably upgrade in Jest 25 to be on the safe side

[nasreddineskandrani]

@SimenB i took a look in diagonal to all the posts @jeysal posted. great work!!

questions

1. Can we put this micromatch fix at the begginning of the work for jest25? so we can play with it and see if it cover the major problem present on windows machines or needs more investigations?

2. You have a branch next for futur release or it's master to contain the micromatch fix ?

[SimenB]

Assuming no breaking changes, you can use yarn resolutions to test yourself.

https://yarnpkg.com/lang/en/docs/selective-version-resolutions/

[playfulpachyderm]

I tried upgrading from v24.9.0 to v26.6.3, and found a slight performance decrease (test suite went from about 17s to 20s). Most of the extra time was at the beginning, there's still a solid 8 to 10 second delay before the first test starts running. There was also an additional pause of about a second or two at the end, after reporting coverage stats, which wasn't there in the earlier version.

I don't know much about profiling, but I read this thread with interest because I was hoping to get my front-end tests to run something closer to as fast as my backend (python) tests, which run in about 3s for an enormously larger and more complicated application.

I run a Ubuntu VM on windows, node v12.20.0.

[trenttobler]

I too have been struggling with Jest performance on our production react app. I have been working to try and implement a "Red Green Refactor" code workflow in our production code base, but the time taken to run tests severely impacts the ability to do this flow efficiently.

I have created a repo to explore this further and try to find other options to improve the performance. See https://github.com/trenttobler/ts-unit-test-loadtesting

Also in this project is second folder with identical tests, except using a mocha+chai+nyc setup to provide evidence that faster testing time should in theory be possible.

I'm looking for any suggestions on strategies or approaches that can be applied to this project, specifically, to get the jest run time to somewhat closer to mocha. This test repository / suite is large (~27K individual assertions), but simulates the scope and raw test counts I'm targeting in our production app.

• jest takes ~15s to run the tests on my machine.
• mocha+chai takes ~4s to run the tests on my machine.
• our production app, with similar range of test suite size, takes 20+ s after all possible improvements I have been able to find online -- I started from a baseline 60+ s run time.

Note: I created the jest test folder using npx create-react-app . --template typescript, followed by adding the stub file, and loadtest folders. I did not remove the existing sample tests created by this template, but do not believe removing them would improve much the actual timings. Also, the functionality demonstrated in this sample is useful, as well, since the production app will need similar features as demonstrated in the sample, to run correctly.

I welcome any help, suggestions, or advice for strategies or techniques to improve the jest performance and get closer to the mocha+chai+nyc timings.

UPDATE: using a '--watch' model, using git commits to control test coverage, got the original tests to almost acceptable rates (~10 seconds for test and coverage report) I would prefer something on the order of 1-2 seconds, but I think this is something we can at least work with.

[mlarcher]

@Moumouls @jeysal Could you please tell me how you generate those flame charts from the jest workers ?

[Moumouls]

@mlarcher here an example: https://medium.com/@paul_irish/debugging-node-js-nightlies-with-chrome-devtools-7c4a1b95ae27

[mlarcher]

@Moumouls thank you for the link !
I can debug the script with the method described there, but the "profiler" tab of chrome-devtools is empty, so I fail to see how to capture a cpuprofile.

I managed to get a .cpuprofile file with "node --cpu-prof ./node_modules/.bin/react-scripts..." but it only shows the calls up to "spawnSync", not what happens in the worker processes.

Could you be more specific about how you got those flamegraphs ?

[nicolaslohrer]

@mlarcher, I've fought with the exact same thing yesterday. There seems to be a bug in Chrome right now. But I got it to work using the latest Chrome Beta version.

[mlarcher]

@nicolasschabram thanks for the tip, it really helps ! 🙂
I can now generate cpuprofile files directly from the chrome beta devtools, but I am facing the same issue : the profile doesn't show much, only a few levels deep until a spawnSync 😕

[mlarcher]

It appeared that create-react-app was making things harder than it should. After ejecting, I was able to run jest directly (with node --inspect scripts/test.js --colors --watchAll=false --verbose) and I was able to inspect the various workers using the logic described in https://indepth.dev/posts/1186/how-to-debug-a-child-process-in-node-and-gatsby-js-with-chrome#patching-jest-workers-to-debug-child-process-in-node i.e. patching the child_process.fork call in node_modules\jest-worker\build\workers\ChildProcessWorker.js with something like

const execArgv = process.execArgv.filter(value=>!value.includes('inspect-brk'));
const randromNumber = Math.floor(Math.random() * 9 + 1);
execArgv.push("--inspect-brk=:700" + randromNumber);

and allowing those ports to be discovered by chrome devTools by declaring them in "Discover network targets"

Now to dig into the output and try to figure out where the time is spent 🤠

[RebecaLozrey]

Add the possibility to run react as a BUILT VERSION instead as development version to avoid all the validations that react does when one do: NPM START

[nicusorflorin]

Any chance of this being fixed?
I am struggling with slow jest on a Windows 10 + Webstorm + WSL2 Ubuntu node interpreter.
For me it takes more than 3 minutes to run a single test with "isolatedModules": true

[nicewaytodoit]

2 years have gone since I created this issue #7619 I believe the only way to fix this is just to use Mocha/Chai/Jasmine... or anything else but Jest! :/

[RebecaLozrey]

I used --maxWorkers=1 Saludos. Ing. Rebeca López Reynoso cel. 04455- 3294-6671 Le lun. 15 mars 2021 à 11:39, Nick ***@***.***> a écrit :

…

Any chance of this being fixed? I am struggling with slow jest on a Windows 10 + Webstorm + WSL2 Ubuntu node interpreter. For me it takes more than 3 minutes to run a single test with "isolatedModules": true — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7963 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG5GYVBIGS4CY7P7OTGMN3TDZA5HANCNFSM4GZXKNIA> .

[chrisribe]

@RebecaLozrey I think this blog post covers the details better.
https://dev.to/vantanev/make-your-jest-tests-up-to-20-faster-by-changing-a-single-setting-i36

I set my config to

"maxWorkers": "50%"

Gave me some improvement.

[Jazekss]

I have same problem with testing speed (time result in pic) i try to configure package.json to add:
"test": "jest --env=mode --color test --watch --runInBand" (two last options) and test time is bit better, but is not good, and in envinfo for VSC i check system and get this (in pic), but i use Windows 10 Pro N x64 (20h2) and my pc is good, 16Gb RAM and GPU ir 2Gb, where is problem? Please help me, i tried what write @cozmy its helped a bit

[spencer-shupe-qhr]

We found the most improvement by using something like

import fg from 'fast-glob';

describe('all specs', () => {
  const specs = fg.sync('src/app/**/*.spec.ts');

  specs.forEach(path => require(path));
});

to run all the specs as one file on our CI environment. Naturally we had lots of tests leaking into one another using this method, but after cleaning that up and following the suggestion from this thead #9457 (comment) in a beforeEach, we reduced our test time by a factor of about 5 (angular 12, >300 spec files and >3000 tests in total).

[hqtoan94]

From me, when using jest to run my unittests, there are 2 problems I've faced with:

• ts-jest takes lot of time for transform ts, tsx files
• my code performs require other modules like a lot, even importing modules that not belong to the test that I've picked for the profiling.

My current approach are:

• Use tsc command to check types of the code, switch to use swc-node/jest to transform the test (reduce run time for the first time because jest does cache the transformed code after the first run)
• Prevent huge import such as import * as A from 'a';, try best to import only what I need, this helped me to reduce the run time about 30%

Hope that jest will introduce us a way to cache imported modules to share access our tests, instead of require it again and again on each test file.

[nthornton2010]

Same problem here with tests finishing in milliseconds, but the test file taking a very long time to finish.

[reinaldo-teixeira]

Same problem... I tried to increase the number of workers but the high time keeps 🙁

[RebecaLozrey]

Read my post if is still there

…

On Fri, Jan 6, 2023, 1:41 PM Reinaldo Teixeira ***@***.***> wrote: Same problem... I tried to increase the number of workers but the high time keeps 🙁 — Reply to this email directly, view it on GitHub <#7963 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG5GYTCIEOCYMBLQEDEBNDWRBYQDANCNFSM4GZXKNIA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tiennguyen1293]

any update on this issue :(

[Gu7z]

Also having this problem

I've created a simple example to check if the problem was my repo and seems that a fresh jest/babel/testing library had the same problem
My config files:

See babel.config.js

Some Javascript

module.exports = { presets: ["@babel/preset-env", "@babel/preset-react"] };

See jest.config.js

I'm testing with node environment and jsx environment

jest-node.config.js

module.exports = {
slowTestThreshold: 0,
testEnvironment: "node",
testMatch: ["<rootDir>/**/*.spec.jsx", "<rootDir>/**/*.spec.js"],
moduleDirectories: ["node_modules"],
testPathIgnorePatterns: ["<rootDir>/node_modules/"],
};

jest-jsx.config.js

module.exports = {
transform: {
  "^.+\\.(js|jsx|ts|tsx)$": "babel-jest",
},
slowTestThreshold: 0,
testEnvironment: "jsdom",
setupFilesAfterEnv: ["<rootDir>/setup.js"],
testMatch: ["<rootDir>/**/*.spec.jsx", "<rootDir>/**/*.spec.js"],
moduleDirectories: ["node_modules"],
testPathIgnorePatterns: ["<rootDir>/node_modules/"],
};

My Specs

No Jsx

describe("testing", () => {
  it("slow spec", () => {
    expect(1).toBe(2);
  });
});

Jsx

import React from "react";
import { render, screen } from "@testing-library/react";

const Test = () => {
  return (
    <>
      <span data-testid="not-empty">
        <span data-testid="empty"></span>
      </span>
    </>
  );
};

test("uses jest-dom", () => {
  render(<Test />);

  expect(screen.queryByTestId("not-empty")).toBeEmptyDOMElement();
});

Results

• I'm doing this test disconsidering cache and runing inBand to reproduce a CI that downloads a repo in every execution and has only one thread. This is my scenario here.
• Boths specs were meant to break

I'm using those two commands

yarn jest -c jest-node.config.js --runInBand --no-cache --verbose node_specs/testing.spec.js
yarn jest -c jest-jsx.config.js --runInBand --no-cache --verbose jsx_specs/test.spec.jsx

Node

See

JSX

See

As you can see a simple spec with just a expect(1).toBe(2) excutes in 8ms but the spec took more than 1 second to run

I also tried to run both node/jsx two times to check if was a startup problem or maybe jest caching, the specs decreased the time but not even closes to the time of the test itself

Timming

Made it work to focus on the timming

Node

JSX

Does anyone know why is it happening?