Profiling tests in Vitest

[timhn-bm]

Hi 👋

I am trying to do some performance profiling on our tests to identify bottlenecks and overall improve our testing strategy.

I am finding it quite difficult and cumbersome to do this profiling. I am wondering if I'm doing things wrong and if there is a better approach than mine with Vitest.

Below is the ideal approach I would want to follow, but does not work with Vitest

This is what I did with Jest for instance, which worked. But not with Vitest

• Run vitest with node --inspect, with tests not running in parallel. For instance
  node --inspect ./node_modules/vitest/vitest.mjs <test-name> --no-file-parallelism --pool=threads --poolOptions.threads.singleThread

• Use ndb in the Dev Tools, run a performance profile and analyse the flamecharts

Unfortunately, as we you can see in the screenshot, we mostly see idle blocks, and nothing related to the tests execution.

Test Profiling, from node --inspect

Why is that happening ?

I don't fully understand why this is the case, but from what I read here and here, Vitest run tests with worker threads or child processes. What we see in the flamecharts is only the execution of the main process (and its main thread ?). This one is responsible for spawning the worker threads / child processes (depending on the pool option). Hence we don't see the actual test execution.

NB: please correct if anything is inexact, or simply wrong. This is what I've understood from Vitest execution so far.

A solution ?

The only solution that worked from me - described here - is to generate cpuprofile files using node --cpu-prof. And then import these files in the Dev Tools. You can see the result in the screenshot below:

Test Profiling, from a cpuprofile file

NB:

• running node --cpu-prof generates multiple cpuprofile files. You'd have to import the right one that contains the test execution
• I am not entirely sure why, but this technique only works with threads, and not forks

Conclusion:
This solution works, but is very cumbersome. I'd love to be able to rerun my test in one save, and obtain the flamecharts immediately. Do you have recommendations to better do profiling ? Is there something that can be done at Vitest codebase level to be able to do this ?

Thanks a lot ❤️

[AriPerkkio]

Edit: There's now https://vitest.dev/guide/profiling-test-performance.html#test-runner

running node --cpu-prof generates multiple cpuprofile files. You'd have to import the right one that contains the test execution

To generate only a single cpuprofile you could use following configuration:

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    pool: "threads",
    poolOptions: {
      threads: {
        singleThread: true,
        isolate: false,
        execArgv: ["--cpu-prof", "--cpu-prof-dir=./profiling"],
      },
    },
  },
});

I am not entirely sure why, but this technique only works with threads, and not forks

I'm not sure why --cpu-prof doesn't work for forks (node:child_process) here. 🤔

But it looks like --prof works:

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    pool: "forks",
    poolOptions: {
      forks: {
        singleFork: true,
        isolate: false,
        execArgv: ["--prof"],
      },
    },
  },
});

There's quite much on the post description but hopefully this helps.

[tankorsmash]

Could you share the specific command to include --cpu-prof? NODE_OPTIONS doesn't support it. Thank you!

edit: node --cpu-prof .\node_modules\vitest\dist\cli.js

[AriPerkkio]

@tankorsmash that won't record profile of test runners, only the main thread. See https://vitest.dev/guide/profiling-test-performance.html#test-runner

[tankorsmash]

@AriPerkkio Thank you. Part of my issue ended up being that the vitest docs defaulted to 'forks' for the pool config, but since I'm on vitest v1.3.0 vitest run uses the 'thread' pool by default, so my settings weren't getting used. Had I read more, I'd have noticed sooner. Thank you!

[AriPerkkio]

but vitest run uses the 'thread' pool by default

Since v2 Vitest uses pool: 'forks' by default:

• fix!: change default pool to 'forks' #5047

[tankorsmash]

Ah, I'm on 1.3.0 still, perfect! Thanks