x/tools/go/analysis/analysistest: add module-aware analyzer test runners

[hyangah]

What I want to achieve

I want to write tests for analyzers that utilize packages.Module information.

x/tools/go/analysis/analysistest provides helper functions for testing go/analysis analyzers.

// It loads the packages from the specified GOPATH-style project
// directory using golang.org/x/tools/go/packages, ...
func Run(t Testing, dir string, a *analysis.Analyzer, patterns ...string) []*Result

// RunWithSuggestedFixes behaves like Run, ...
func RunWithSuggestedFixes(t Testing, dir string, a *analysis.Analyzer, patterns ...string) []*Result

• Run and RunWithSuggestedFixes internally call go/packages.Load but explicitly in GOPATH mode.
• The dir is supposed to be GOPATH, which doesn't apply in module mode.
• packages.Config is not accepted.

These limit the utility of the package when authoring module-aware analyzers.

What I propose

I propose to add module-friendly test helpers. Something like:

func TestAnalyzer(t Testing, a *analysis.Analyzer, pkgs *[]packages.Package) []*Result

or

func TestAnalyzer(t Testing, a *analysis.Analyzer, cfg *packages.Config, patterns ...string) []*Result

Alternatives to consider

GOPATH style project layout is easy and convenient. For the last half decade, nobody
complained about this issue, which indicates supporting module mode in this package is not
important. My analyzer is not a typical one.

So I considered -

1. Refactor and move the core of x/tools/go/analysis/analysistest to x/tools/internal/analysisinternal/.... Unfortunately, I see this will cause move of x/tools/go/analysis/internal/checker and analysisflags. I like containing analysis specific internal logic under x/tools/go/analysis/internal and am not sure about this big move.

2. Make part of analysistest.Run available through a var in x/tools/internal/analysisinternal/... that's set by analysistest init. See CL408375 This is a smaller change than 2, but documentation for the type aliased analysistest.Testing and checker.TestAnalyzerResult isn't great.

What about WriteFiles

It also supports only GOPATH mode. WriteFiles can be potentially replaced with go/packages/packagestest.

// WriteFiles is a helper function that creates a temporary directory
// and populates it with a GOPATH-style project using filemap...
func WriteFiles(filemap map[string]string) (dir string, cleanup func(), err error)

But for a simple analyzer test, this is more convenient than the helpers in go/packages/packagestest.

cc @golang/tools-team

[heschi]

See also #37054. People have been complaining for a while, that was the first issue I found.

[timothy-king]

I think it is fine to extend analysistest for modules mode.

I want to understand what a "module-aware analyzer" is though. *analysis.Pass does not know what a packages.Module is ATM. Does this need to happen first? I am trying to make sure analysistest is really the bottleneck here.

[hyangah]

What I mean by module-aware analyzer is simply an analyzer that can operate on the package layout in Go Modules mode (not GOPATH mode).

I think including packages.Module in *analysis.Pass should be a separate, independent proposal.

But I think users can always build a package-to-packages.Module mapping outside. With #53215 or other tweak, it's also possible that users can inject the mapping build logic even when using the singlechecker/multichecker framework.

[romaindoumenc]

If that helps make things more concrete, I think the simple line change below is enough (while still allowing GOPATH to be set by an environment variable).

[hyangah]

@romaindoumenc This discussion is not only about making analysistest load packages with module mode (the code path was already mentioned in the original report), but also making the loaded packages and their modules accessible.

[gopherbot]

Change https://go.dev/cl/577995 mentions this issue: go/analysis/analysistest: add a TestDir helper for modules