When using multiple Vitest configurations, running the tests runs outdated code

[Fheuef]

Describe the bug

I have added two different configuration files in the same project : vitest.config.ts and vitest_sometimes.config.ts. The first file excludes some test files that I do not want to run by default - i. e. some performance benchmark tests that take a very long time - the latter includes only those tests.

When I want to run the "sometimes" set of tests, I specify the config file in the CLI, or use the extension to select the "sometimes" profile or run them manually.

However, the extension does not refresh the contents of the default set of tests, nor the code that they import, when running the tests. Instead, it seems to use cached code and so it runs outdated tests and shows results that do not match the current files. In order to use the newly saved code, I have to manually click the "Refresh" button, or restart VS Code.

This happens only in the test files intended for the default config, and the "sometimes" tests work perfectly !

(Note: the default file does not specify include paths and uses the default settings. I have tried to explicitly import the default test files and got the same results.)

This bug can be quite misleading, and has already caused me to commit broken code after seeing the tests pass - though I suppose that's on me for not automating the tests on commit.

The issue seems similar to #469, though it was fixed a month ago. However, this one was using one config file per project, whereas my issue uses multiple files in the same repo.

Reproduction

https://github.com/Fheuef/issue-vitest-config

Here are the config files for quick reference:

vitest.config.ts

export default defineConfig({
  test: {
    exclude: [...configDefaults.exclude, "**/*sometimes.test.ts"],
  },
});

vitest_sometimes.config.ts

export default defineConfig({
  test: {
    include: ["**/*sometimes.test.ts"],
  },
});

Output

[INFO 6:14:49 PM] [v1.6.9] Vitest extension is activated because Vitest is installed or there is a Vite/Vitest config file in the workspace.
[INFO 6:14:50 PM] [API] Running Vitest v2.1.5 (VitestConfig/vitest.config.ts) with Node.js@v20.11.0: C:\Program Files\nodejs\node.EXE 
[INFO 6:14:50 PM] [API] Running Vitest v2.1.5 (VitestConfig/vitest_sometimes.config.ts) with Node.js@v20.11.0: C:\Program Files\nodejs\node.EXE 
[INFO 6:14:51 PM] [API] Vitest v2.1.5 (VitestConfig/vitest.config.ts) child process 23468 created
[INFO 6:14:51 PM] [API] Vitest v2.1.5 (VitestConfig/vitest_sometimes.config.ts) child process 9668 created
[INFO 6:14:51 PM] [VSCODE] Watching VitestConfig with pattern **/*
[INFO 6:15:08 PM] [API] Collecting tests: src/default.test.ts
[6:15:08 PM] Not starting the runner because tests are being collected for src/default.test.ts
[6:15:08 PM] There is no test run for "default.test.ts"
[6:15:08 PM] No test run to finish for src/default.test.ts
[6:15:08 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[INFO 6:15:10 PM] [API] Collecting tests: src/sometimes.test.ts
[6:15:10 PM] Not starting the runner because tests are being collected for src/sometimes.test.ts
[6:15:10 PM] There is no test run for "sometimes.test.ts"
[6:15:10 PM] No test run to finish for src/sometimes.test.ts
[6:15:11 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:15:17 PM] [VSCODE] File changed: src\default.test.ts
[6:15:21 PM] [VSCODE] File changed: src\sometimes.test.ts
[INFO 6:15:21 PM] [Worker] Collecting tests due to file changes: src/sometimes.test.t
[6:15:21 PM] Not starting the runner because tests are being collected for src/sometimes.test.ts
[6:15:21 PM] There is no test run for "sometimes.test.ts"
[6:15:21 PM] No test run to finish for src/sometimes.test.ts
[6:15:21 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:15:34 PM] [VSCODE] File changed: src\sum.ts
[6:15:45 PM] Initiating deferred test run
[INFO 6:15:45 PM] Running 1 file(s): src
[6:15:45 PM] The runner is starting because tests src/default.test.ts were started due to a file change
[6:15:45 PM] Enqueuing "does not refresh tests"
[6:15:45 PM] Enqueuing "does not see new tests"
[6:15:45 PM] No task result for "default.test.ts", ignoring
[6:15:45 PM] Enqueuing "does not refresh tests" because it was just collected
[6:15:45 PM] Enqueuing "does not see new tests" because it was just collected
[6:15:45 PM] No errors found for "default.test.ts"
[6:15:45 PM] Marking "does not refresh tests" as failed with 1 errors
[6:15:45 PM] Marking "does not see new tests" as passed
[6:15:45 PM] Ending test run src/default.test.ts
[6:15:45 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:15:54 PM] [VSCODE] File changed: src\sum.ts
[6:15:54 PM] [VSCODE] File changed: src\sum.ts
[6:15:56 PM] Initiating deferred test run
[INFO 6:15:56 PM] Running 1 file(s): src
[6:15:56 PM] The runner is starting because tests src/default.test.ts were started due to a file change
[6:15:56 PM] Enqueuing "does not refresh tests"
[6:15:56 PM] Enqueuing "does not see new tests"
[6:15:57 PM] No task result for "default.test.ts", ignoring
[6:15:57 PM] Enqueuing "does not refresh tests" because it was just collected
[6:15:57 PM] Enqueuing "does not see new tests" because it was just collected
[6:15:57 PM] No errors found for "default.test.ts"
[6:15:57 PM] Marking "does not refresh tests" as failed with 1 errors
[6:15:57 PM] Marking "does not see new tests" as passed
[6:15:57 PM] Ending test run src/default.test.ts
[6:15:57 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:16:02 PM] Initiating deferred test run
[INFO 6:16:02 PM] Running 1 file(s) with name pattern: ^\s?does not refresh tests$
[6:16:02 PM] The runner is starting because tests src/default.test.ts were started due to a file change
[6:16:02 PM] Enqueuing "does not refresh tests"
[6:16:03 PM] No task result for "default.test.ts", ignoring
[6:16:03 PM] Enqueuing "does not refresh tests" because it was just collected
[6:16:03 PM] Marking "does not see new tests" as skipped during collection
[6:16:03 PM] No errors found for "default.test.ts"
[6:16:03 PM] Marking "does not refresh tests" as failed with 1 errors
[6:16:03 PM] Ending test run src/default.test.ts
[6:16:03 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:16:04 PM] Disposing test runner
[6:16:04 PM] Ending test run <none>
[6:16:04 PM] Disposing test runner
[6:16:04 PM] Ending test run <none>
[INFO 6:16:04 PM] [API] Vitest process 9668 closed successfully
[INFO 6:16:04 PM] [API] Vitest process 23468 closed successfully
[6:16:04 PM] [API] Vitest child_process connection closed, cannot call RPC anymore.
[6:16:04 PM] [API] Vitest child_process connection closed, cannot call RPC anymore.
[INFO 6:16:04 PM] [API] Running Vitest v2.1.5 (VitestConfig/vitest_sometimes.config.ts) with Node.js@v20.11.0: C:\Program Files\nodejs\node.EXE 
[INFO 6:16:04 PM] [API] Running Vitest v2.1.5 (VitestConfig/vitest.config.ts) with Node.js@v20.11.0: C:\Program Files\nodejs\node.EXE 
[INFO 6:16:04 PM] [API] Vitest v2.1.5 (VitestConfig/vitest_sometimes.config.ts) child process 21748 created
[INFO 6:16:04 PM] [API] Vitest v2.1.5 (VitestConfig/vitest.config.ts) child process 22688 created
[INFO 6:16:04 PM] [VSCODE] Watching VitestConfig with pattern **/*
[INFO 6:16:04 PM] [API] Collecting tests: src/default.test.ts
[6:16:04 PM] Not starting the runner because tests are being collected for src/default.test.ts
[INFO 6:16:04 PM] [API] Collecting tests: src/sometimes.test.ts
[6:16:04 PM] Not starting the runner because tests are being collected for src/sometimes.test.ts
[6:16:05 PM] There is no test run for "default.test.ts"
[6:16:05 PM] No test run to finish for src/default.test.ts
[6:16:05 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:16:05 PM] There is no test run for "sometimes.test.ts"
[6:16:05 PM] No test run to finish for src/sometimes.test.ts
[6:16:05 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:16:06 PM] Initiating deferred test run
[INFO 6:16:06 PM] Running 1 file(s) with name pattern: ^\s?does not refresh tests$
[6:16:06 PM] The runner is starting because tests src/default.test.ts were started due to a file change
[6:16:06 PM] Enqueuing "does not refresh tests"
[6:16:06 PM] No task result for "default.test.ts", ignoring
[6:16:06 PM] Enqueuing "does not refresh tests" because it was just collected
[6:16:06 PM] Marking "does not see new tests" as skipped during collection
[6:16:06 PM] No errors found for "default.test.ts"
[6:16:06 PM] Marking "does not refresh tests" as passed
[6:16:06 PM] Ending test run src/default.test.ts
[6:16:06 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:16:07 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:16:10 PM] Initiating deferred test run
[INFO 6:16:10 PM] Running 1 file(s) with name pattern: ^\s?refreshes tests$
[6:16:10 PM] The runner is starting because tests src/sometimes.test.ts were started due to a file change
[6:16:10 PM] Enqueuing "refreshes tests"
[6:16:10 PM] No task result for "sometimes.test.ts", ignoring
[6:16:10 PM] Enqueuing "refreshes tests" because it was just collected
[6:16:10 PM] Marking "sees new tests" as skipped during collection
[6:16:10 PM] No errors found for "sometimes.test.ts"
[6:16:10 PM] Marking "refreshes tests" as passed
[6:16:10 PM] Ending test run src/sometimes.test.ts
[6:16:10 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json
[6:16:31 PM] Initiating deferred test run
[INFO 6:16:31 PM] Running 1 file(s): src
[6:16:31 PM] The runner is starting because tests src/default.test.ts were started due to a file change
[6:16:31 PM] Enqueuing "does not refresh tests"
[6:16:31 PM] Enqueuing "does not see new tests"
[6:16:31 PM] No task result for "default.test.ts", ignoring
[6:16:31 PM] Enqueuing "does not refresh tests" because it was just collected
[6:16:31 PM] Enqueuing "does not see new tests" because it was just collected
[6:16:31 PM] No errors found for "default.test.ts"
[6:16:31 PM] Marking "does not refresh tests" as passed
[6:16:31 PM] Marking "does not see new tests" as passed
[6:16:31 PM] Ending test run src/default.test.ts
[6:16:31 PM] [VSCODE] Ignoring file: node_modules\.vite\vitest\results.json

Version

1.6.9

Validations

• Check that you are using the latest version of the extension
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[sheremet-va]

Thank you! This is a very good reproduction 🙏🏻