test: setup vscode-e2e

[hi-ogawa]

After further exploration, I ended up creating a package vitest-e2e https://github.com/hi-ogawa/vscode-extension-shell-shortcut/tree/main/packages/e2e, so let me dog-food this here.
EDIT: now all code is inlined to test-e2e/helper.ts

---

You can download playwright trace.zip from the artifact https://github.com/vitest-dev/vscode/actions/runs/8135218393 and you can load it on https://trace.playwright.dev/

Current (failing) trace looks like this now: Finally successful trace is here:

Screenshot

[sheremet-va]

Maybe just use vscodeTest.extend?

const test = vscodeTest.extend({
  extensionPath: '.',
  workspacePath: './samples/e2e'
  trace: 'on'
})

And inside _launch you can just access them:

const vscodeTest = test.extend({
  extensionPath: null, //default
  workspacePath: null, //default
  _launch: async ({ extensionPath, workspacePath }, use) => {
    const _launch = await launch({
      extensionPath,
      workspacePath,
    });
    await use(_launch);
    await _launch.app.close();
  },
})

[hi-ogawa]

I agree that using task.meta doesn't feel convenient, but If API requires users to do one more test.extend themselves, then I would simply expose something like createVscodeTest instead of exposing "partial" fixture vscodeTest.
This is a prototype for that idea dd5dba5

[sheremet-va]

I think it's absolutely fine to call extend several times, this is the recommended way in playwright for example. The basic test function is already called with playwright to define "page" and other fixtures for example

[hi-ogawa]

Yeah, I understand it's a valid API for users to do test.extend, but while iterating on the API, I just started to feel I'm over-engineering for what would be much more direct and simpler.

For now, I'll take back my package and I would be more comfortable to just copy minimal implementation here.
(Also, execute vscode proxy turns out to be not so useful here, so that's another reason my package isn't necessary.)

The one interesting thing Playwright's fixture can do is that, since they have fixture options https://playwright.dev/docs/next/test-fixtures#fixtures-options, they can pick up certain configuration from their config file to be used as their default fixture customization.

If Vitest would have this feature, then that could be another potential API to configure extensionPath.

[hi-ogawa]

I just found one more idea of providing launch function fixture a0a709d.

This is inspired by playwright's internal fixture _contextFactory https://github.com/microsoft/playwright/blob/d5d4f591f35970f48ffe82bfc5c9e117660d7f0f/packages/playwright/src/index.ts#L304

[hi-ogawa]

As I've seen test-e2e failing a few times on ecosystem-ci PR vitest-dev/vitest-ecosystem-ci#12, I just found it's also failing in this repo too:
https://github.com/vitest-dev/vscode/actions/runs/8139113562/job/22241536646

This failing trace is showing that "Test result not found" for all-pass.test.ts, but I'm not sure what race condition would cause this:

Trace screenshot

Probably just adding --retry should save us from CI failure for now.

[sheremet-va]

Can this be merged? I would love to have some new tests in #253

[hi-ogawa]

Yeah, I think this was is ready. Please feel free to merge!