-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Vitest runs tests 3x slower than Jest with threads: true (default) setting #579
Comments
Vitest is also about 2-3x slower than @web/test-runner on the same project. But also, it reports the time incorrectly: when it says that tests ran for 7 seconds, the |
@cbxp User time will be higher because Vitest is multi-threaded. |
A slash was missing in the close-tag.
One of the reasons why Jest is fast is cache. I added
Another benchmark. https://medium.com/dailyjs/javascript-test-runners-benchmark-3a78d4117b4 |
My team in the process of migrating a large test suite (6073 tests) from jest to vitest. With jest, the full suite runs in 3m47s. With the latest vitest (0.18.0) it runs in 10m7s. We really want to continue with vitest but this kind of a slow down is making it challenging. We don't want to disable isolation, as it's crucial for reliability in a test suite this large. Is there any plan to improve the performance vitest, or is jest doing something fundamentally different that vitest won't be able to replicate? @antfu @sheremet-va @Demivan |
@pmdarrow I'm investigating switching to But I did not have time to work on this recently (there is the war going on) |
I don't think we can do it without getting the same problems Jest has, because VM has terrible support for ESM. I already tried it. |
You can get speed boost by disabling isolation. Bear in mind tho, that it will make |
After investigating it more and checking with @sheremet-va i don't think it can be implemented with current state of Lets hope ShadowRealm implementation will be better then |
So is the summary that we're stuck with the performance where it's at right now, unless we want to sacrifice isolation? If so, I think this should be published in the docs somewhere. I know the docs say disabling isolation can improve performance, but it doesn't communicate that comparable tools (i.e. Jest) are 3x faster even with isolation. It would help to know up front why vite is slower by default and what is blocking it from becoming faster ( |
That is an incorrect statement. It really deps on what and how your tests are written. You can't say a man walks faster than cars without the context (in a traffic jam or something it might be true). The reason why this is happening commonly due to you are using some giant dependencies, where in isolation, they will be executed multiple times for each tests. Vitest is on-demand like Vite, while Jest will bundle the code, so in some scenario it would be more efficient than Vitest. While indeed there is some space for optimization like pre-bundle the deps, or waiting for |
Can you please elaborate a bit on how/where exactly Jest bundles anything? AFAIK Jest transformers are invoked lazily whenever a file is being loaded by other files during a test file execution.
The thing is that such "unit" tests (deeply mocked) tests are fast in both Jest and Vitest (the difference is really minor, and not always in favour of the same framework). However, in addition, Jest is isolating much better, so the trade-offs between speed and correctness case are relevant for Vitest only, while with Jest one can get both in this case. Unfortunately the isolation with the use |
Is there anyway of debugging |
Like @thebuilder and @pmdarrow we have also experienced a 3x slowdown of tests after migrating to vitest from jest, our large CI unit test suite now takes 15 mins instead of 5 when running on GHA. |
Ok, I can have a look at this. Can anyone set up a minimal repo (2~3 test files, or the less possible) of how the slowdown is being observed (as I personally do not face such a slowdown when doing unit testing or testing Vue)? Thanks |
@antfu Maybe https://github.com/EvHaus/jest-vs-jasmine can help. It's more than 2~3 test files, but I think you can easily just strip out the test suites and reduce them. The latest benchmark shows:
|
Hm, looking at the repo, I wonder if the problem is in Disabling css with // plugin.js
{
name: "remove-css",
enforce: "post",
transform(code, id) {
if (id.endsWith(".module.less")) {
const code = `
export default new Proxy(
{},
{
get(_, style) {
return style;
},
}
);`;
return {
code: code,
};
}
},
}, I have this result on M1 Air 8 GB:
Maybe you should also benchmark Considering CSS, I don't see a lot of changes unfortunately. So It seems the problem lies with how tinypool fires up isolated workers. |
@antfu is https://github.com/EvHaus/jest-vs-jasmine sufficient for a reproduction or would you like it to be trimmed down further? |
I trimmed a bit locally. Flow + Less is a bit less common tbh, but yes with your tests I can confirm it surfaced some bottleneck of Vitest, where I am still investigating. |
@antfu is there anything that we could do to help move this forward? |
@phifa I don't think that's related to this discussion - that's just vitest using as many cores as possible. If you don't want this behaviour, you can turn it off: https://vitest.dev/guide/features.html#threads. |
Same experience here. Jest ran my test-suite in about 3-5 minutes, after migrating to vitest it now takes 10-17 minutes 😞 I'm sure it would be a lot faster if I disabled threads, but it's not worth dealing with all the potential issues that comes with that. I do have |
I wonder, if enabling |
@AriPerkkio Really great to see the performance improvements from #3006. Moving the 5000 test above from 5667.68s to 4.63s is huge! Thanks for the effort on that. |
I've noticed that one of the reasons why there is a difference between Jest and Vitest is the fact that Vitest imports |
I guess it wouldn’t do it when testing pure node code? Does it mean that the issues discussed here do not apply to testing server code? |
Yep, it depends on the environment. You can check out my previous comment #579 (comment) |
Uff, that's a lot. I guess when you combine this with testing library, it really adds up. |
Why not import |
Each test file runs inside a separate worker. You cannot transfer JSDOM constructor between worker threads as far as I know. |
Hi guys! Our project has a lot of files that are connected during the test (utils, components, openApi interfaces). Because of what, even one test takes a very long time (
I run the test in debug mode with
Is it possible to merge the imported files into one so that Vitest does not waste time downloading them one by one? |
SUMMARYI've written up a summary of all the advice given in this GitHub Issue as a time saver for those coming here in the future. Improving Vitest Performance |
|
Using v0.31.4. The situation with me seems a bit worse than what I read here. I have 55 tests spread across 19 files. On my machine, these tests run in 28 seconds. However, on Azure DevOps (Pipeline), they take over 30 minutes (I gave up waiting). I tried using the With the I'm using tools like testing-library, msw, material-ui, react-hook-form. |
If a test that takes 30 seconds locally takes over 30 minutes on your server, the first thing you need to check is that the test is written correctly. |
After your comment, I realized there was a test with using I ended up being able to make it work without any changes to the tests, using the option |
I also noticed that after moving from Jest to Vitest, my tests were taking 2-3x more time to run. After doing some experiments lately, one thing that drastically reduced the time to run all specs was to disable experimental fetch ( Before: 126.28s |
Vitest 0.34.0 is released with the new |
I updated the benchmark to use vitest@0.34.1
vitest@0.34.1 with
The test previous ran in 101.259s but now runs in 60.171s, which is also identical to Jest's speed! Side-note: Looks like using I think based on this excellent new development we can probably close this ticket? |
The fastest way to run tests is still using only
We still need a way to run tests using VM inside the forked process to support |
My bad. I meant |
Yes, it has no effect: https://vitest.dev/config/#isolate |
Conceptually,
|
We are using msw on React test and we are facing this issue when enabling
|
I will ask this again: If you have any issues with the new pool, please open a separate issue instead of commenting here. In your case, jsdom doesn't provide I am locking this issue. Open new issues if you have problems when running |
More tips from @TheJaredWilcurt: https://dev.to/thejaredwilcurt/improving-vitest-performance-42c6 |
Describe the bug
With the latest Vitest/Vite dependencies, Vitest is ~3 times slower than Jest in https://github.com/EvHaus/jest-vs-jasmine (more or less like a typical front-end repo) with the
threads: true
(default) setting.Related comment #229 (comment). That issue was closed but
threads: true
is still 3-4 slower than Jest after the issue was closed and the repro dependencies updated).The issue was raised in Discord chat as well.
With
threads: false
Vitest is ~2 times faster, but the cost of it - there's no real isolation. Withthreads: false
it's unfortunately not even close what Jest is doing in terms or resetting state, so not even sure if makes sense to comparethreads: false
with Jest (maybe threads: false should be compared withuvu
or other "run-once-and-forget" runners with no real isolation). In a variety of projects that we have worked on, even with pure stateless components, the proper isolation is a super important concern. Especially in watch mode where introducing some unwanted state/mocking or polluting global/process state is simply an expected thing to happen due to the nature of various incremental changes (that are not always good or final) that simply break not properly isolated watch mode.Reproduction
(if required, install
hyperfine
)System Info
Used Package Manager
npm
Validations
The text was updated successfully, but these errors were encountered: