Make it possible to run extension tests while VS Code is running

[hbenl]

I would like to be able to run extension tests (that use vscode-test) in the Mocha Test Explorer (using a custom "launcher script").
However, when I try to run the tests from this extension, I get the error message "Running extension tests from the command line is currently only supported if no other instance of Code is running.".
Please make this possible, it would be much more convenient to be able to run extension tests in the Test Explorer.

[octref]

It's not possible. I suggest you either use https://code.visualstudio.com/insiders/ or use the command from CLI.
Alternatively you can use launch.json to launch the E2E test.

[hbenl]

So it is possible if I use the insiders edition to run the tests while the regular edition is running, that's great!
Unfortunately I am now stuck with another problem: when I try to run the runTest script (which uses vscode-test/runTests to start the tests) in a child process started from the Mocha Test Explorer extension, I get the following errors:

[2020-03-08 16:52:02.400] [ERROR] /home/marvin/.vscode-test/vscode-insiders/VSCode-linux-x64/code-insiders: bad option: --no-sandbox
/home/marvin/.vscode-test/vscode-insiders/VSCode-linux-x64/code-insiders: bad option: --extensionDevelopmentPath=/home/marvin/Misc/projects/test-explorer/vscode-extension-samples/helloworld-test-sample
/home/marvin/.vscode-test/vscode-insiders/VSCode-linux-x64/code-insiders: bad option: --extensionTestsPath=/home/marvin/Misc/projects/test-explorer/vscode-extension-samples/helloworld-test-sample/out/test/suite/index

Any idea why this could be happening? If I run the runTest script in a child process started from a regular Node process, everything works fine, the error only seems to happen when the child process is started from the extension.

[octref]

So it is possible if I use the insiders edition to run the tests while the regular edition is running, that's great!

Exactly, you can use Insiders version and run tests against stable as well (that's my setup).

When I install Mocha Test Explorer I see nothing on the sidebar for helloworld-test-sample. Can you push that to a repo so I can repro?

Maybe it's an issue for the extension instead of for VS Code.

[hbenl]

You need to add the following settings to the project:

  "mochaExplorer.files": "out/test/**/*.js",
  "mochaExplorer.launcherScript": "out/test/runTest.js",
  "mochaExplorer.logpanel": true

Then you can open the output panel called "Mocha Explorer Log", click Reload in the Test Explorer and you will see the error messages in the output panel.
(Note that this setup wouldn't work even if I didn't get the "bad option" error messages - I still need to create a version of src/test/suite/index.ts that uses the worker script from the extension instead of calling Mocha directly. But before I can do that, I need to get past the "bad option" errors.)

With the settings above, the extension will run out/test/runTest.js in a child process, which is started here.

Maybe it's an issue for the extension instead of for VS Code.

I hope so, then I could fix this myself. But I have no idea why I get this error and why I only get it if the child process is started from the extension - when I try to run out/test/runTest.js in a child process started from a regular Node process, it works as expected.

Thanks for looking into this!

[octref]

I can't repro on macOS. The issue seems to apply only to Linux.

@bpasero @deepak1556 Seems to be caused by Electron 6 change:
https://github.com/microsoft/vscode/blame/1876b51870e30e9ba8f35dc0dcfaf49a3799a8ef/src/vs/code/node/cli.ts#L328-L330

[bpasero]

@octref --no-sandbox is necessary since we updated to Electron 6, otherwise Electron will refuse to start on Linux. If you think there is a bug, please clarify.

[hbenl]

This happens not just on Linux: on Windows I get the exact same errors, including the error "bad option: --no-sandbox".
I also tried it on MacOS and there I get a different error: "Error: EROFS: read-only file system, mkdir '/.vscode-test'" (so it fails to download VS Code). Like the other errors, this one also only occurs when I am starting vscode-test from the Mocha Explorer extension, but not if I start it from a regular Node process.

[hbenl]

It looks like the error on MacOS occurs because process.cwd() returns an empty string!? See here and here.
I tried fixing this by putting process.chdir('/Users/me') before the call to runTests, but that doesn't seem to change anything!?
Any idea why this could be happening? Is there anything special about child processes started from the extension host process?

[hbenl]

I've created a little sample with which it is easy to reproduce the issue: https://github.com/hbenl/vscode-test-issue-57

[hbenl]

FYI: After playing with that sample for a while I found out how I can make it work: I need to use spawn() instead of fork() to create the child process and remove ELECTRON_RUN_AS_NODE from the child process' environment:

const cp = spawn(
	'/usr/bin/node',
	[ path.resolve(workspacePath, 'out/test/runTest.js') ],
	{
		env: { ...process.env, "ELECTRON_RUN_AS_NODE": undefined },
		stdio: [ 'pipe', 'pipe', 'pipe', 'ipc' ]
	}
);

[tamuratak]

I have found that, with --user-data-dir, we can run VS Code from the command line. See #58.