Cannot Debug Test Files with --inspect-brk Flag

[alexlafroscia]

Describe the bug

With the new --inspect and --inspect-brk flags, I expected to have a way for Vitest to debug test files without a tool like VSCode or IntelliJ. Unfortunately, that was not the case!

With a debugger in my test file, the following command does not pause on the debugger statement

npx vitest --inspect-brk --threads false run whatever-filter

A debugger in my config file is paused on just fine, but ones in tests (which are more likely what folks want to debug) are not respected.

The commit that introduces the flags doesn't specifically say that the flag does or doesn't allow for debugging tests, so I suppose this expectation is an assumption on my part, but I believe that this flag not allowing child processes to be debugged would surprise users.

Adding the --threads false configuration was an attempt to avoid spawning child processes, like Jest's --runInBand, but that doesn't resolve the issue.

The only way I have found to debug a test file is with a tool that automatically allows child processes to be debugged, such as ndb or the VSCode debugging tools (through their "launch" config or integrated JavaScript debugging terminal). While this is fine for some users (myself included!) I am not sure how to advise team members that do not use these tools on how to debug test files.

Reproduction

This repo contains a reproduction environment, as well as instructions

https://github.com/alexlafroscia/__vitest-inspect-brk

System Info

System:
    OS: macOS 12.5
    CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
    Memory: 29.96 GB / 64.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 16.13.2 - ~/.volta/tools/image/node/16.13.2/bin/node
    Yarn: 1.22.17 - ~/.volta/tools/image/yarn/1.22.17/bin/yarn
    npm: 8.1.2 - ~/.volta/tools/image/node/16.13.2/bin/npm
    Watchman: 4.9.0 - /opt/brew/bin/watchman
  Browsers:
    Chrome: 104.0.5112.79
    Safari: 15.6
    Safari Technology Preview: 16.0
  npmPackages:
    vitest: ^0.22.0 => 0.22.0

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[antfu]

Yes, I am aware of this. The support is currently only forwarding the flag to the Node process, where the main issue is that Node does not support inspecting workers yet. --threads=false for now is still running in a single worker for context isolation but I agree it's might be good to have the capability of running the test in the main process for debugging purposes.

[tomholford]

Same issue here. Also not able to attach to the debug process in VSCode using the recommended launch config.

Node 14.19.0 / npm 8.6.0 / Vitest 0.22.1
Chromium: 104.0.5112.102 (Official Build) (64-bit)

@alexlafroscia can you share the launch config that worked for you? will see if it resolves the issue

[alexlafroscia]

As far as I know I haven't used a launch config to debug tests; the Vitest plugin for VSCode is excellent, and I use that a bunch. I also use the JavaScript Debug Terminal in VSCode,, which does some magic under-the-hood to ensure that all Node processes are launched in --inspect mode with VSCode attached to it.

[alexlafroscia]

Does #2772 get us any closer to being able to support a more "typical" Node debugging story?

I'm imagining a situation where, if --no-threads is used, the underlying child process is also spawned with --inspect/--inspect-brk so that debuggers in the tests themselves will be respected.

My thought is that if --inspect and --inspect-brk are added to the allowed execArgv members that are forwarded along here, this might "just work"

vitest/packages/vitest/src/node/pool.ts

Lines 28 to 30 in 0280825

	const execArgv = process.execArgv.filter(execArg =>
	execArg.startsWith('--cpu-prof') || execArg.startsWith('--heap-prof'),
	)

[AriPerkkio]

I've been able to step into Vitest test cases with debugger even when threads are enabled. Not sure if vitest --inspect-brk works but pass it directly to node and it works.

.vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "inspect vitest",
      "type": "node",
      "request": "launch",
      "autoAttachChildProcesses": true,
      "runtimeArgs": [
        "--inspect-brk",
        "${workspaceRoot}/node_modules/vitest/vitest.mjs"
      ],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen"
    }
  ]
}

Debugger on Chrome Devtools works just as well. Simply run node --inspect-brk ./node_modules/vitest/vitest.mjs and open chrome://inspect.

[alexlafroscia]

I think it's the "autoAttachChildProcesses": true, bit in there that does something to the Node process to allow inspectors to be attached, automatically, to child processes as they are spawned. ndb works in a similar way, but I have found that Vitest becomes unusably slow when I try to use ndb.

What I'm looking for is a command to document for my team so they can debug tests without using VSCode. Many of the developers do use it, but we also are not making VSCode a requirement for working on our project, so I'm hoping something can be done within Vitest to allow for debugging in an editor-agnostic way.

[AriPerkkio]

The VSCode's autoAttachChildProcesses seems to do some magic indeed. With latest Vitest release I'm also unable to attach to test cases without using autoAttachChildProcesses.

But by modifying the sources so that --inspect-brk is passed to child_process when threads: false is used, I'm able to pause on debugger using Chrome dev tools. The flags introduced by ea80f2bf are not used by Vitest (unless retries is being triggered), so I think those can be removed from the CLI-wrapper. I'll work on fixing this next.

[alexlafroscia]

I upgraded to 0.29.3 today and tested out the new debugging features, and it worked just as expected! Thank you @AriPerkkio for taking this on!