Vitest 3 runs 3 times slower than Vitest 2

[TheJaredWilcurt]

Describe the bug

My unit tests took ~12 seconds on the latest v2 of Vitest, then switching to v3, they now take ~36.5 seconds to run locally. Importantly they went from 3.2 seconds to 3.0 seconds on CI (GHA).

My assumption is this local slowdown is caused by the live update "Duration" timer constantly refreshing the terminal multiple times a second. From my experience in the past, these constant update severely slow down running tasks. On Windows for example, minimizing a command prompt will cause it to run faster as it stops performing repaints. On Ubuntu, minimizing the terminal actually caused it to go much slower (~65 seconds). I assume minimized windows on Ubuntu are given less resources?

I looked around but couldn't find any way to turn the "Duration" field off when running tests. This would be a good option even if it is unrelated to performance because I just don't like it. It's annoying and distracting to have something constantly flashing. I don't care how long it's been running, I care how long it took at the end (and only very rarely).

Reproduction

This is the library repo I'm working on where I noticed this:

• https://github.com/tjw-lint/vue3-snapshot-serializer

System Info

12 second details:

  System:
    OS: Linux 6.8 Ubuntu 24.04.1 LTS 24.04.1 LTS (Noble Numbat)
    CPU: (2) x64 Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
    Memory: 4.84 GB / 7.71 GB
    Container: Yes
    Shell: 5.2.21 - /bin/bash
  Binaries:
    Node: 23.6.0 - ~/.volta/tools/image/node/23.6.0/bin/node
    npm: 11.0.0 - ~/.volta/tools/image/npm/11.0.0/bin/npm
  npmPackages:
    @vitejs/plugin-vue: ^5.2.1 => 5.2.1 
    @vitest/coverage-v8: ^2.1.8 => 2.1.8 
    vitest: ^2.1.8 => 2.1.8 


36 second details:

  System:
    OS: Linux 6.8 Ubuntu 24.04.1 LTS 24.04.1 LTS (Noble Numbat)
    CPU: (2) x64 Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
    Memory: 4.82 GB / 7.71 GB
    Container: Yes
    Shell: 5.2.21 - /bin/bash
  Binaries:
    Node: 23.6.0 - ~/.volta/tools/image/node/23.6.0/bin/node
    npm: 11.0.0 - ~/.volta/tools/image/npm/11.0.0/bin/npm
  npmPackages:
    @vitejs/plugin-vue: ^5.2.1 => 5.2.1 
    @vitest/coverage-v8: ^3.0.2 => 3.0.2 
    vitest: ^3.0.2 => 3.0.2

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[boldurean]

We upgraded it also to v3 and the tests execution went from 17s locally to almost 70s
Downgraded back to v2 :|

[AriPerkkio]

I looked around but couldn't find any way to turn the "Duration" field off when running tests. This would be a good option even if it is unrelated to performance because I just don't like it. It's annoying and distracting to have something constantly flashing. I don't care how long it's been running, I care how long it took at the end (and only very rarely).

By annoying and distracting flashing, do you mean the summary? Use following to disable it: https://vitest.dev/guide/reporters.html#built-in-reporters

You can disable the summary by configuring the reporter:

export default defineConfig({
  test: {
    reporters: [
      ['default', { summary: false }]
    ]
  },
})

[TheJaredWilcurt]

Confirmed, applying the summary: false removes the "Duration" and also fixes the performance issue. @boldurean try it out, you may be able to upgrade after all.

Though honestly, I'd rather have the summary on, but the duration off. Since the duration is what makes the performance slow, and also is annoying to look at. Basically "exactly like it was in v2".

[AriPerkkio]

How did you narrow down the root cause to the Duration field? Do you have any *.cpuprofile files that indicate it? We could probably extend the configuration to allow something like ['default', { summary: { duration: false } }] if that helps.

Testing your reproduction on Github Codespace with time pnpm run test --run looks good to me. How can I reproduce this slowness?

• 2.1.8:

  • time reports: real 0m7.287s, user 0m8.109s
  • Vitest reports: Duration 6.34s (transform 217ms, setup 17ms, collect 443ms, tests 519ms, environment 1.92s, prepare 1.78s)

• 3.0.2:

  • time reports: real 0m7.460s, user 0m8.249s
  • Vitest reports: Duration 6.38s (transform 212ms, setup 16ms, collect 439ms, tests 479ms, environment 2.02s, prepare 1.84s)

Does this recording demonstrate the annoying and distracting flashing you mentioned? I'm not sure I see it but might be just biased. 🤔

vitest-maybe-flickering.webm

[AriPerkkio]

We also have a test suite where we run 5000 test files with total of 20000 test cases. This test suite enables these interactively updating reporters even on CI (using node-pty). Then we check that the continuous updates are not blocking main thread, which could cause test slowness like seen in vitest@v1 at #4710.

• Vitest 2.1.8 on CI:

  • Tests took 33.900369
  • Slowest function { name: '(idle) }, meaning that main thread was not blocked by reporter ✅
  • https://github.com/vitest-tests/reporters-large/actions/runs/12842231985/job/35812876647

• Vitest 3.0.2 on CI:

  • Tests took 18.138398 ✅
  • Slowest function { name: '(idle) } ✅
  • https://github.com/vitest-tests/reporters-large/actions/runs/12842227828/job/35812868160

• Vitest 2.1.8 locally (M2, 16GB)

  • Tests took 13.025625
  • Slowest function { name: '(idle) } ✅

• Vitest 3.0.2 locally (M2, 16GB)

  • Tests took 10.021416 ✅
  • Slowest function { name: '(idle) } ✅

[fmal]

Confirmed, applying the summary: false removes the "Duration" and also fixes the performance issue. @boldurean try it out, you may be able to upgrade after all.

Though honestly, I'd rather have the summary on, but the duration off. Since the duration is what makes the performance slow, and also is annoying to look at. Basically "exactly like it was in v2".

the live update of "Duration" timer also caused some issues for me when running tests inside pre-push hook and logging results to stdout. Each update creates a new entry in stdout creating lots of noise.

[TheJaredWilcurt]

You can see the redraws/repaints being done by the terminal/command prompt. Because the duration is updating so frequently, it is causing flashing to occur more often. My assumption is it is also related to the performance issue, as I've seen this same outcome from other CLI's in the past. Also turning summary off seems to confirm this suspicion.

---

I absolutely want a summary: { duration: false } option. I think there may even be a case for making that the default.

[AriPerkkio]

In Vitest 2 we used to repaint whole terminal window every 16ms. Now the repainting is limited to only the summary. We've actually seen improved reporting performance now.

What terminal is that? I have some improvement ideas but before that I need to be able to reproduce this issue. Your reproduction works fine on Github Codespaces, running on Ubuntu. I would expect that to match your reported environment Linux 6.8 Ubuntu 24.04.1 LTS. 🤔

the live update of "Duration" timer also caused some issues for me when running tests inside pre-push hook

@fmal please create new issues for reports that are unrelated to this one. Let's keep this one focusing on Vitest 3 performance only, as the title says.

[TheJaredWilcurt]

Original report is on an old laptop running Ubuntu in a VM. So it's a much slower than normal environment to begin with. Using default terminal, no tweaks/modifications.

The GIF was created on Windows using cmd.exe just because it was easier to record a GIF there. But the outcome is still the same (the flashing). The performance on Windows doesn't seem impacted, but that may just be because Windows is different, or it's a much more powerful machine.

[AriPerkkio]

There's preview release that you can try now:

npm i https://pkg.pr.new/vitest@7291

#7291 (comment)

[TheJaredWilcurt]

Trying with npm i https://pkg.pr.new/vitest@7291.

• 15.16s
• 16.34s
• 14.89s

Setting { summary: false } with it gives:

• 16.67s
• 14.37s
• 14.45s

So about the same. So this PR removes the difference in performance between enabling/disabling Summary.

---

Switching back to 2.x.x:

• 14.51s
• 14.78s
• 14.31s

---

Switching back to 3.0.2:

• 37.34s
• 36.96s
• 34.95s

---

So this seems fixed to me!

[TheJaredWilcurt]

Also the "flashing" seems to have gone away too, so I no longer need a duration: false option.

[AriPerkkio]

Awesome, thanks for testing. If possible, could you record similar gif as in #7285 (comment) with the preview release? I'm just interested to see how it looks now. I'm unable to reproduce that kind of flickering terminal output myself.

[TheJaredWilcurt]

No flashing at all on Windows. On the Ubuntu VM, there is some, but it's very minor and not really any different from any other CLI usage.

[boldurean]

Confirmed, applying the summary: false removes the "Duration" and also fixes the performance issue. @boldurean try it out, you may be able to upgrade after all.

Though honestly, I'd rather have the summary on, but the duration off. Since the duration is what makes the performance slow, and also is annoying to look at. Basically "exactly like it was in v2".

Yip that improves the speed. Thank you!

[TheJaredWilcurt]

@boldurean cool, then you had the same issue as me. There is a PR up to resolve it (#7291), so you won't need to turn off the summary after that is merged/released.