Seems to be much slower than Mocha

[paldepind]

I currently use a testing setup for my projects that uses Mocha, ts-node, and nyc. In several of them I've tried switching to Jest. But ts-jest is consistently significantly slower 😢

I've added Jest to this project where you can see the difference.

These are the times I'm seeing in the project:

• Current setup: time npm test 4.05 seconds
• Jest: time npm run test-jest 12.34 seconds

Note that my tests themselves takes about 3-4 seconds. So Mocha adds a very small overhead where Jest is noticeably slower. I've tried in larger projects as well where the difference is even bigger.

For your convenience, here are the commands to reproduce:

git clone https://github.com/funkia/list
cd list
npm install
time npm test
time npm run test-jest

[kulshekhar]

@paldepind thanks for creating the repo to test on.

I tried this out and ts-jest does seem slower. I'm getting results of 7s and 14s for the two tests on a test machine. I definitely want to see this improve!

Note: This issue is not the same as #259. The repo linked in this issue takes the same time for ts-jest versions 20.0.4 and 20.0.7. If you're facing the same issue as #259, please create a minimal repo that reproduces that issue and we can open that issue and work on it

[paldepind]

@kulshekhar

Thank you for the reply. It would be awesome if the overhead could be broad down. I'd love to use Jest :)

thanks for creating the repo to test on.

You're welcome. If having an additional repository to test on might be helpful I can setup Jest on another open source project of mine as well.

[kulshekhar]

I've looked into this a bit and it seems there's some issue with caching.

Even with a bare minimum transformer that just returns valid js code (without any compilation overhead), jest seems to take 8+ seconds on all runs where mocha takes 7s on the first run and 4s for subsequent runs. This can be seen in the js branch of the fork of the above repo

I'm not sure this caching issue is caused by the implementation here or if it is something that needs to be fixed in jest

[kulshekhar]

I've posted a question on stackoverflow to see if we can get some help on this

[paldepind]

@kulshekhar What about opening an issue in the Jest repo? ts-node is clearly very important to Jest (it's mentioned on the front page and in the official docs) so I imagine they'd be willing to help out 😄 Especially if the performance issue is caused by Jest.

[kulshekhar]

It seems they only want feature requests and bug reports on their issue tracker and I'm not yet sure this is a bug. That's why I've posted on stackoverflow. Hopefully we'll get some help there.

Improving the performance of the simple transformer, which does absolutely no work, should help improve things for ts-jest as well.

[GeeWee]

I think we can definitely report this to the jest repo directly - it's a reasonably nice small reproduction on stackoverflow.

[kulshekhar]

I just tried this without any transformer and the results are the same. The setup in the linked repo is slower with jest than with mocha even without ts-jest. I think this issue should be closed and one opened in the Jest repo.

@paldepind thoughts?

[tvald]

We've identified the issue with ts-jest's caching implementation: in preprocessor.ts:getCacheKey(), tsconfig.json is loaded and wildcard paths (eg, include: ['src/**/*']) are resolved.

This results in an expensive walk of the entire source directory for each typescript file in the source directory, causing exponential growth in running time as a codebase expands. With our large codebase, this caused a no-op test to take more than 7 seconds to complete.

I'll submit a patch shortly that caches in preprocessor.ts. I'm not sure it would be possible to cache within getTSConfig():utils.ts, since it seems to be called with different parameters from various places in the code. Eventually, it should be ensured that tsconfig.json is only loaded and paths resolved once.

[tvald]

After further profiling, it seems that getTSConfig() needs additional optimization. Our full test suite took 100s before #286 was merged, and now runs at 45s. However, the hack that I first proposed for #286 in a8de326 enables our test suite to run in 9s. We'll try to diagnose it further and come up with another PR.

[kulshekhar]

@tvald

Our full test suite took 100s before #286 was merged, and now runs at 45s

I suspect that this particular improvement might have been due to #292 which was published in 20.0.10. Other users have reported a similar improvement (see #259)

A quick way of confirming/refuting this would be to test against 20.0.9.

[kulshekhar]

I'm closing this because, at this stage, it looks like this can't be fixed in ts-jest.

We can use this issue to continue the discussion and, if necessary, reopen it down the road.

[paldepind]

@kulshekhar Should I open a bug in the Jest repo?

[kulshekhar]

@paldepind It might help. I'll leave my fork of your repo up if it helps. The js branch has the absolute minimal version that can demonstrate this issue

[GeeWee]

Would love a PR with the performance implications you're suggesting @tvald

[tvald]

Further investigation shows that the jump from 45s to 10s was due to sourcemap caching on subsequent runs thanks to #292. 10s is still fairly lengthy for this particular test, so we'll continue to investigate in our spare time. (35s for sourcemaps is also rather hefty, even if only on first run.)

[tvald]

Our final conclusion is that there's simply a lot of overhead with large codebases.

jest resolves modules via the resolve package, which performs ~3x fs.statSync per file at a cost of ~5ms per file. With 1000 files, this adds up to ~5s. Babel takes up another three seconds, leaving the individual test to run in less than a second.

[GeeWee]

Guess there's nothing more we can do here then.