x/tools/gopls: friction working with internal/lsp/tests

[findleyr]

This issue catalogs friction experienced with the "old" LSP "marker tests" in internal/lsp/testdata (run by internal/lsp/tests).

These tests have long been tricky to work with, and recently have caused a significant amount of friction while making changes to error messages in go/parser and go/types.

Notable problems:

• test output include noisy LSP logs for the entire test session. It would be better if these logs were scoped to the actual failing package, or completely excluded.
• test output includes red-herring "errors" that are not actually related to the test failure
• tests are all run in the same session / workspace, so changes in far-away files can affect e.g. completion results
• auto-generated test names (which include the annotation position) are not stable, and confusing
• failure messages can be hard to read, because they do a poor job of highlighting differences between expected and actual output.
• tests often match error messages too precisely, resulting in churn when error messages change across Go versions
• test annotations are not documented; it is not clear how to add new annotations
• tests run in multiple contexts (as tests for the internal/lsp/source, internal/lsp/cmd and gopls packages), and this is not clearly documented, nor are the differences between these contexts made clear. (and the necessity for all three contexts is not clear)
• tests use summary*.txt.golden files (depending on go version) as checksums to ensure that the expected number of tests ran. These are (by construction!) change detectors.

CC @adonovan @griesemer

[gopherbot]

Change https://go.dev/cl/431843 mentions this issue: gopls/internal/lsp: suppress noisy log output in tests

[gopherbot]

Change https://go.dev/cl/431844 mentions this issue: gopls/internal/lsp/tests: pass in a *testing.T to Data.Golden

[gopherbot]

Change https://go.dev/cl/405254 mentions this issue: internal/lsp/source: derive document symbols from syntax alone

[gopherbot]

Change https://go.dev/cl/432138 mentions this issue: gopls/internal/lsp: simplify prepareRename tests

[gopherbot]

Change https://go.dev/cl/432336 mentions this issue: gopls/internal/lsp/tests: use the mustRange helper in more places

[gopherbot]

Change https://go.dev/cl/432337 mentions this issue: gopls/internal/lsp/tests: simplify collectCompletionItems, remove Data.t

[gopherbot]

Change https://go.dev/cl/432955 mentions this issue: gopls/test: disable stderr output for command line tests as well

[gopherbot]

Change https://go.dev/cl/434635 mentions this issue: gopls: fix the reset_golden.sh script and regenerate golden files

[gopherbot]

Change https://go.dev/cl/434636 mentions this issue: gopls: update to handle 'undefined:' versus 'undeclared' in type errors

[gopherbot]

Change https://go.dev/cl/435355 mentions this issue: gopls/internal/lsp: tolerate new 'imported and not used' error message

[findleyr]

@adonovan and I just discussed where to go with this next, and there are a few major problems we'd like to solve:

1. the global namespace of marker tests: adding a new test can cause action at a distance
2. the lack of discoverability for new markers, and generally poor documentation for marker tests
3. the discrepancy with the regtests (and regtests could be marker tests, but aren't because of 1 and 2)

We just brainstormed a design for solving all of these problems:

• represent the marker, collection method, and application method as an instance of a new type
• run this new type through the regtest framework

To illustrate what I mean, here is a hypothetical implementation of an isolated marker test:

func TestMarkerTest(t *testing.T) {
	const mod = `
-- go.mod --
module mod.com

go 1.12
-- main.go --
package main

var C = 3

func main() {
	println(C) //@rename("C", "D")
}
-- main.go.rename_C_D_unified --
- var C = 3
+ var D = 3
-	println(D) //@rename("C", "D")
+	println(D) //@rename("C", "D")
`
	RunMarkerTest(t, mod, tests.Rename)
	RunMarkerTest(t, mod, tests.Rename, tests.References)
	RunMarkerTest(t, mod, tests.All...)
}

[gopherbot]

Change https://go.dev/cl/540915 mentions this issue: gopls/internal/regtest/marker: porting remaining suggestedfix tests

[gopherbot]

Change https://go.dev/cl/540916 mentions this issue: gopls/internal/regtest/marker: use diffs for rename tests

[gopherbot]

Change https://go.dev/cl/539664 mentions this issue: gopls/internal/regtest/marker: trim redundant header from golden diffs

[gopherbot]

Change https://go.dev/cl/539665 mentions this issue: gopls/internal/regtest/marker: port remaining rename tests

[gopherbot]

Change https://go.dev/cl/540917 mentions this issue: gopls/internal/regtest/marker: port the inlay hint tests

[gopherbot]

Change https://go.dev/cl/541196 mentions this issue: gopls/internal/regtest/marker: port the semantic tokens tests

[gopherbot]

Change https://go.dev/cl/541126 mentions this issue: gopls/internal/lsp/tests: remove AddImport support

[gopherbot]

Change https://go.dev/cl/541127 mentions this issue: gopls/internal/regtest/marker: clean up some random packages

[gopherbot]

Change https://go.dev/cl/541129 mentions this issue: gopls/internal/regtest/marker: port the selectionrange markers

[gopherbot]

Change https://go.dev/cl/541236 mentions this issue: gopls/internal/regtest/marker: port remaining marker tests

[findleyr]

We ported all these tests to a new marker framework (x/tools/gopls/internal/regtest/marker), which addresses all of the problems we identified with the old tests.

[gopherbot]

Change https://go.dev/cl/542636 mentions this issue: gopls: remove the LiteralCompletions option