cmd/go: make test caching aware of testdata directories

[crawshaw]

I have a test that processes test files from a testdata directory (that it finds using a filepath.Glob). Adding a new file to the directory and running "go test mypkg" ignores the new file, and returns the cached test results.

The go tool already has special handling for testdata directories: they are ignored when "go test" searches for packages to run. Could we extend this and include a hash of the contents of the testdatadirectory in the test cache key?

[mvdan]

What would be included - filenames, modes and bodies?

I like this proposal better than the proposals to tell go test about extra file dependencies. Would anyone depend on an extra file outside of testdata for good reason?

[cespare]

This is a great idea. It would cover almost all the tests at my company which depend on data files.

In a few cases we have test fixtures that are shared between a few different projects (not all of which are Go). In these cases the fixtures are symlinked into testdata.

[rsc]

This is a possible approach but it is imprecise, meaning it will both rerun tests too often (when a testdata file not relevant to a particular test run changes, for example a cached -run=SpecificTest that doesn't read any testdata at all) and not enough (some tests depend on other things). If we can do a bit better, it would be nice to.

[bradfitz]

We could hook the os and/or syscall package and watch which filesystem operations the test binary executes, and cache that as well, keyed by the digest of the test binary.

So running a cached test involves looking up the FS operations from cache, then checking the state of those (including the result of globs), and then using that in the computation of the test result cache key.

[dlsniper]

This issue does not address the cases where people do not use testdata as folder for their test data or in which the test data is part of folders like assets or fixtures. A cursory look on Github for fixtures indicates that there are plenty of people using this name for their test data folder: https://github.com/search?l=Go&q=fixtures&type=Code&utf8=✓

[rsc]

Off topic but if people use some other directory name besides testdata to mean testdata, they should probably be reminded that the Go name is testdata. That name is known to the go command and not scanned by directory walks and so on, so there are computational benefits to using the standard name, in addition to the benefits everyone enjoys when everyone follows a single convention.

[rsc]

Closing this bug in favor of #22593.