Unable to run extension test from command line when vscode-insider is open

[ankitbko]

Version: 1.53.0-insider (user setup)
Commit: c927a80
Date: 2020-12-18T05:27:48.567Z
Electron: 11.0.5
Chrome: 87.0.4280.88
Node.js: 12.18.3
V8: 8.7.220.29-electron.0
OS: Windows_NT x64 10.0.19042

Unable to run test through CLI if vscode-insider is open. This was working just yesterday and then I updated the insider and it stopped working.

Steps to Reproduce:

1. Checkout https://github.com/microsoft/vscode-pull-request-github
2. Open the repo in vscode-insider and run yarn install and yarn test.
   Get below error -

❯ yarn test
yarn run v1.21.1
warning vscode-pull-request-github@0.22.0: The engine "vscode" appears to be invalid.
$ yarn run preprocess && node ./out/test/runTests.js
warning vscode-pull-request-github@0.22.0: The engine "vscode" appears to be invalid.
$ yarn run compile && yarn run preprocess-gql && yarn run preprocess-svg
warning vscode-pull-request-github@0.22.0: The engine "vscode" appears to be invalid.
$ tsc -p ./
warning vscode-pull-request-github@0.22.0: The engine "vscode" appears to be invalid.
$ node scripts/preprocess-gql --in src/github/queries.gql --out out/github/queries.gql
warning vscode-pull-request-github@0.22.0: The engine "vscode" appears to be invalid.
$ node scripts/preprocess-svg --in resources/ --out preview-src/dist/resources
Found .vscode-test/vscode-insiders matching latest Insiders release. Skipping download.


(electron) Sending uncompressed crash reports is deprecated and will be removed in a future version of Electron. Set { compress: true } to opt-in to the new behavior. Crash reports will be uploaded 
gzipped, which most crash reporting servers support.

Warning: 'sandbox' is not in the list of known options, but still passed to Electron/Chromium.

[main 2020-12-18T07:37:16.271Z] Running extension tests from the command line is currently only supported if no other instance of Code is running.

[main 2020-12-18T07:37:16.274Z] Error: Running extension tests from the command line is currently only supported if no other instance of Code is running.
    at Object.doStartup (D:\src\github\ankitbko\vscode-pull-request-github\.vscode-test\vscode-insiders\resources\app\out\vs\code\electron-main\main.js:656:314)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async D:\src\github\ankitbko\vscode-pull-request-github\.vscode-test\vscode-insiders\resources\app\out\vs\code\electron-main\main.js:653:836
    at async Object.startup (D:\src\github\ankitbko\vscode-pull-request-github\.vscode-test\vscode-insiders\resources\app\out\vs\code\electron-main\main.js:653:682)

Exit code:   1
Done

Failed
Done in 12.58s.

---

Edit: Can confirm this is also happening in samples present in vscode-test repo. You will need to first comment all the runTests except for the 'insider' one. It seems this only happens on insider version.

[alexr00]

Edit: Can confirm this is also happening in samples present in vscode-test repo. You will need to first comment all the runTests except for the 'insider' one. It seems this only happens on insider version.

@bpasero are you an owner of https://github.com/microsoft/vscode-test ?

[bpasero]

We never supported to run the tests from the command line when VSCode is running. You can simply run tests from the debug viewlet in that case.

I think @connor4312 would be the new owner of extension test running. To my knowledge nothing has changed in this area. An easy way to find out is to use Code stable to test with @ankitbko

[ankitbko]

@bpasero The documentation states running test through CLI from insider version is possible. Is that not the case?

I also found microsoft/vscode-test#57 which also seems to suggest the same.

[bpasero]

@ankitbko yeah it is possible but not from within VSCode.

[ankitbko]

@bpasero I am kinda confused. Can you confirm if it is possible to run the extension tests through command line while vscode-insider is open? If so, can you share how to run the test when vscode insider is open? If not, we may want to update the document. The thing is it used to work till few days ago. So I am not sure if it was incidental that it worked earlier of if this is actually a bug. If it is a but I would be grateful if someone else can also try to reproduce the same.

[bpasero]

@ankitbko you can run extension tests from the command line provided that NO instance of VSCode is running. I am not sure how to improve the message further, it currently says:

"Running extension tests from the command line is currently only supported if no other instance of Code is running"

How would you change that?

And since you indicate it worked before, can you do the same steps using Code stable and not insiders.

[ankitbko]

@bpasero It never worked with Code stable but worked with insiders. I was talking about improving the documentation, not the error message. The current FAQ in documentation implies that running from command line while VSCode stable is open does not work and suggests to download insider to get it working. The wording - "You can either use VS Code Insiders for development or launch the extension test from the debug launch config that bypasses this limitation." gives the impression it is either-or scenario.

[bpasero]

@ankitbko ah! yeah I get it now, running insiders and launching tests from stable is indeed something that should work. but if an extension for example suddenly decides to use proposed API I think we would automatically run the tests against insiders version. So maybe that changed recently in the PR extension?

[ankitbko]

I would start by apologizing for making this confusing. I am new to VSCode extension development so am unaware of the history and I am getting too confused by docs and issues open regarding this. So thank you @bpasero for patiently being with me.

ah! yeah I get it now, running insiders and launching tests from stable is indeed something that should work.

So it is possible to run test from command line while insider is open? I always used to run tests against insider VSCode as PR extension uses proposed API. So I used to open PR extension repo in VSCode-insider and run the tests against VSCode-insider through command line. I also used to have VSCode stable open but that had some different project and nothing to do with the PR extension. Also I found microsoft/vscode-test#58 which allows this scenario for some reason.

[bpasero]

So it is possible to run test from command line while insider is open

The rule is that you cannot run extension tests from the command line if the instance to run the tests against is opened. But it should be possible to run tests against stable when insiders is open. What do you see if you do that?

[ankitbko]

ah.. now I get it. I never ran against stable when insider was open as extension is using proposed api. Will the other way also work - Run tests against insider from stable terminal? But thank for clarification. Weirdly it was working earlier and sudden change surprised me. We should definitely mention this clearly on the doc.. perhaps have some sort of truth table on what scenario works.

[bpasero]

Yeah any way works (stable, insiders) as long as the tests execute in a version that is not running already, I can update the docs end of month to make this clearer.

[bpasero]

I just pushed microsoft/vscode-docs@8f9f070 to our docs which hopefully makes this clearer.