🧪 Testing: Establish a preference for small, self-contained mutation tests

[JoshuaKGoldberg]

Overview

This is something I'd wanted to start on ages ago, but never had the time and never got around to writing down...

Today, there are quite a few mutation tests that have a lot of stuff in them. Example: https://github.com/JoshuaKGoldberg/TypeStat/blob/16f76cf106033d5ca45da3eed82de3120aae4c08/test/cases/fixes/noImplicitAny/variableDeclarations/original.ts is just over 200 lines. Large mutation tests were handy when I was getting started on TypeStat many years ago and wanted to quickly iterate on many things at once.

But, the library is a bit more settled now, and these huge tests are unwieldy. It's hard to precisely check one thing at a time with them or search for where one thing is tested for. Example: to test noImplicitAny with variable declarations and Promise resolves, I'd probably temporarily comment out all but the last ~10 lines of the aforementioned variableDeclarations/original.ts file.

Proposal: let's...

1. Quick change: mention in .github/DEVELOPMENT.md a preference for small, self-contained mutation tests
2. Larger change: split up each of these larger original.ts files into families of more granular ones

WDYT @rubiesonthesky?

[rubiesonthesky]

I think it would be good to experiment with mutation test that does not actually need typestat.json and tsconfig.json in the disk. What I mean is that, you could write test somehow like this:

const { actualContent, expectedFilePath } = await runMutationTest(
	// code
	`
		const returnsStringAndNumber = (): number => {
			return BigInt(123n);
		};
	`,
	// typestat
	`
	{
		"fixes": {
			"incompleteTypes": true
		},
		"types": {
			"strictNullChecks": true
		}
	}
	`,
	// tsconfig
	`{
		"files": ["actual.ts"]
	}
	`

);
await expect(actualContent).toMatchFileSnapshot(expectedFilePath);

This would allow creating small tests for specific thing.

So I'm favor of this idea and there are some ways it could be done. :)

[JoshuaKGoldberg]

+1, I would like that very much. #1491 (comment) mentions long-term hopes around not being tied into a file system.

[rubiesonthesky]

This seems interesting! https://sdorra.dev/posts/2024-02-12-vitest-tmpdir

With writing temp file, we could save the mutations to temp file and then read that. This would work for smaller tests.