Improve selfhost testing experience

[connor4312]

• Make it possible to run extension tests using our running
  • This probably entails making a generic VS Code extension test runner (neat!)
• @ulugbekna is passionate about Jest!
  • We probably cannot readily copy-paste jest into core tests, but we should see if there's features we miss in Jest that we can bring over
  • Any inspiration we can take from the Jest extension to improve UX in core?

[gjsjohnmurray]

Personally I am hoping greater use by the team of the native testing UI will lead to the coverage API getting some attention. In particular, until its results get surfaced in the UI there's not a lot of incentive for us extension developers to invest in it. See #123713

[ulugbekna]

I’m a big fan of snapshot testing that Jest supports and think we could benefit greatly from adopting it.

Snapshot testing

Snapshot tests (aka "expect tests) are tests that capture the string representation of a given object and compare to the expected string. The important aspect of these tests is that those snapshots

• can be hand-written or "captured" on the first run. Note how I write toMatchInlineSnapshot() and the test runner runs the test, get the expected output and updates the test with the expected snapshot. This allows writing tests a lot faster

Screen.Recording.2023-08-10.at.11.56.36.mov

• if the actual snapshot doesn't match the expected snapshot, one can update the latter automatically with various granularity - test, suite, file. Henning contributed a code action that allows updating a single assertion failure in our tests, but it is limited to a single assertion at a time

Snapshots also allow for more visual tests because one doesn't need to write those by hand:

		test('returns the positions of all functions in the source code', async () => {
			const source = outdent`
				function add(a, b) {
					return a + b;
				}

				function subtract(a, b) {
					return a - b;
				}
				`;
			expect(await jsSourceWithFunPos(source)).toMatchInlineSnapshot(`
			"<start-0>function add(a, b) {
				return a + b;
			}<end-0>

			<start-1>function subtract(a, b) {
				return a - b;
			}<end-1>"
		`);
		});

Some things that could be improved with Jest & Jest extension:

1. Updating the snapshot requires a re-run of tests (possibly reuses the cached test runs, but still not instantaneous update as is the case with the OCaml language test runner), while we could store the expected outputs and update the file as required (this's the problem with Jest rather than the extension)
2. It uses diagnostics to show the diff between test mismatches instead of a diff viewzone

3. Each diff viewzone could have a button to accept the proposed snapshot

---

I like this blog post describing how snapshot tests can be nice - https://blog.janestreet.com/the-joy-of-expect-tests/

Jane Street is big on snapshot testing and have their own framework. Their way of creating snapshots is a bit different because they don't compare the snapshot against an object but they capture stdout from a test as a snapshot.

---

Happy to jump on a call to discuss this more and, in general, help with implementation some of the things, if time permits.

[connor4312]

Followup items from discussion:

• A way to view/peek the snapshot expected results
  • Syntax highlighting for peek expectations would be nice
• Test Results should reflect retired state so we can see what snapshots have been applied
• isPeekVisible context key does not seem to behave correctly 🤔
• Make electron messages in the output terminal dimmed out / less prominent

[connor4312]

I finished the big themes I wanted to implement for this issues with the test CLI and its adoption in core. Copying the notice I sent for posterity

---

This week I've been working on a new way to run extension tests. It includes a configuration-driven command-line runner, as well as an extension that runs extensions tests in the VS Code UI.

👉 For people who own built-in extensions: You can now use this for built-in extensions in VS Code! To onboard your extension: (example)

1. Add your extension to the extensions array in .vscode-test.js
2. Update test-integration.sh and test-integration.bat to run using the runner
3. If the tests don't run in the web, you can remove your old test .ts runner script. The runner does not yet support web tests (but it will soon) so this is still needed.
4. You can now run your extension tests in two ways:
   a. Via yarn test-extension -l <label> on the command line. Do this to validate your config is set up correctly!
   b. With the ms-vscode.extension-test-runner extension installed, you can run and debug tests using the VS Code UI, as in the image below

👉 For people who write extensions: pretty similar to the above! You can use the above as guidelines as well as the documentation and an example.