How to filter tests in CLI (without actually running everything else before)

[heysokam]

I've been working with mocka, where marking a test as it.only means that ONLY that test is run at any time.
I was expecting that same behavior in vitest, but I cannot get it to behave like that no matter what I try.

I was also using the -g Pattern option in mocka (equivalent to -t in vitest), and with that I was also ONLY running the tests that matched the pattern.

The behavior I'm getting in vitest, instead, is that first all hooks/tests that would happen before the filtered test are checked and listed as skipped (which takes a while) and only then the test is run.

That increases test iteration time enormously, moving it

• from: some amount of miliseconds to run a single test
• to: tens of seconds to wait for the other tests to be skipped and reach the desired test, and only then run the test

I'm aware of watch mode, and the ability to stop tests inside it with p, and then filtering inside it with t.
But it feels like a complicated hack around a simple problem, taking multiple steps to achieve what a single command is meant to achieve just by itself.

Is there any way around this problem?
How can I make it so that actually only the filtered tests are touched by the runner when ordering the runner to filter by some condition?

[hi-ogawa]

Probably --standalone is a current alternative for your workflow. Starting vitest --standalone, then modifying a file with test.only should re-run a single test.

Or in the future, we'll have a filtering by line number like vitest basic.test.ts:123, which might fit your usage

• feat(cli): Support specifying a line number when filtering tests #6411

[heysokam]

I worded my message ambiguously.
This issue is specifically when using vitest run, not when using watch mode.
The explanation about watch mode in my question was pointing to the hack around the problem, not to the desired workflow.

If my understanding is correct, --standalone is a watch-mode feature that only improves the messy hack-around I proposed that doesn't solve the problem.

One of the reasons we use it.only (with the equivalent to run-mode) is also to debug single tests from the vscode debugger/inspector, in order to step through the code to track issues.
Watch mode does not solve that usecase, which is also a primary part of the workflow.

For the vscode debugger usecase (which it.only used to solve), running tests without concurrency also triggers the slowness problem I mentioned above, since all tests/hooks are checked until the test with only is reached.

For the CLI usecase, watch mode overrides CLI output automatically, which creates problems when sharing screen with a coworker, explaining them the error, touching a file (even if only a character), saving by habit... and coming back to cli only for the error to be gone because watch mode reruns the tests based on file-changes, not on explicit request.
Plus the terminal window that holds the runner is rendered unusable for any other cli tasks.
Using vitest run for this usecase avoids the watch-mode issues, but triggers the workflow issues mentioned in the original question.

Hopefully the clarification can shed some light into the issue.

In case I made the explanation worse, the expected behavior is:

When using vitest run --allowOnly=true
as long one test/describe is marked with .only
completely ignore everything not marked with .only
(including hooks and skip checks that are out of scope of any .only marks)

[sheremet-va]

.only affects only the test file where it was defined in. This is intentional. Vitest runs test files eagerly (collect->run), there is no global collection phase. It is not possible to achieve due to how parallelisation works (we can't just collect a test inside a worker to then destroy the worker).

The hooks inside the file where there is an .only test will run only in those suites where the test is declared - I just confirmed this behaviour. If all tests are skipped in a suite, no hook is running.

We do not plan to change the behaviour.

[heysokam]

--no-file-parallelism and --max-concurrency=1 exist
There could also be a collect->filter->run process (even if its behind a --sequentialFiltering config flag or similar) that could make the code behave as per the expectation.

This has turned our test debugging workflow into a "waiting for the runner to understand what I asked for" game,
Tests used to run immediately when filtered and now the runners interpret every file in alphabetical order...
and good luck debugging a failing test in a packages/{U,V,W,X,Y,Z}/** folder.
Takes minutes to get to the breakpoint :(

[sheremet-va]

You would also need a —no-isolate flag for this to work. But even then the changes to the test lifecycle would be too big to support this use case. There is a lot of expectation in the code about how it works and introducing this inconsistency would make it hard to maintain and even harder to document and explain the different models.

I would recommend specifying the test file when debugging or using the vscode extension (or your IDE extension directly)

[heysokam]

I will try that 👍

[sheremet-va]

It won’t work, we force it to be 1 by 1 even without isolation. I am just explaining why we don’t support it even then

[heysokam]

You: I would recommend specifying the test file when debugging

Me: I will try that

[sheremet-va]

Ah, sorry. I thought you meant —no-isolate 😁