std,cmd: add test to ensure that go.mod and vendor are in sync

[dmitshur]

We should make it so that trybots catch and report when the src/go.mod and src/vendor directories are not in sync, to prevent them from going out of sync without anyone noticing.

See #36851 for more context.

/cc @golang/osp-team

[dmitshur]

@bcmills I think this issue is the same as issue #36907 (which was created slightly later), and can be considered resolved via CL 217218. Do you agree?

[dmitshur]

Oh, I misread, this is about divergence between go.mod and vendor. That issue was about divergence between two different go.mods.

[dmitshur]

That said, the test in CL 217218 seems to check for this issue too, does it not?

Edit: It only checks for missing vendored packages, not that the vendored packages are identical.

[bcmills]

Correct. The new test only checks that vendor/modules.txt is consistent with go.mod: it does not verify that the actual vendored source code is unchanged.

[dmitshur]

@bcmills I have some questions about this, since you have more context and might be able to answer more easily.

Do you think this can be implemented by adding a test that does go mod vendor in the two modules and does the equivalent of git diff? Does go mod vendor need internet beyond accessing the module mirror? Do you know of any precedent for doing the "equivalent of git diff" in tests in the standard library? Final question, would it be sufficient to do this check on one or two configurations (e.g., linux/amd64), or can you think of a reason to do it on more/all builder types?

[dmitshur]

I've confirmed there isn't an existing test (on master) that catches this issue by making a trivial edit in src/cmd/vendor and running src/all.bash; it reported "ALL TESTS PASSED".

[bcmills]

Do you think this can be implemented by adding a test that does go mod vendor in the two modules and does the equivalent of git diff?

Yes.

Does go mod vendor need internet beyond accessing the module mirror?

It should need only the module mirror.

Do you know of any precedent for doing the "equivalent of git diff" in tests in the standard library?

There are a few tests that check that generated source files are clean (such as cmd/go.TestDocsUpToDate), but to my knowledge none that rely on git (or on being able to write to GOROOT).

Another approach might be to set up an overlay of the std module in a temporary directory and run go mod vendor within that overlay, then diff the resulting trees (using diff -r or an equivalent Go function, perhaps implemented in terms of filepath.Walk).

The misc/reboot test illustrates how to make an overlay of GOROOT/src in order to test commands that mutate GOROOT. (Unfortunately, due to the fact that the misc repo does not have a fully-qualified import path in GOPATH mode, that part of the repo contains many duplicate copies of the file overlaydir_test.go that constructs the actual overlay.)

Final question, would it be sufficient to do this check on one or two configurations (e.g., linux/amd64), or can you think of a reason to do it on more/all builder types?

go mod vendor is intended to be platform-independent, so running the test on one or two configurations should suffice.

[gopherbot]

Change https://golang.org/cl/255859 mentions this issue: misc/update: add program to update golang.org/x dependencies

[gopherbot]

Change https://golang.org/cl/256357 mentions this issue: cmd/updatestd: add program to maintain standard library modules

[dmitshur]

Thanks for answering @bcmills. Based on those answers (and our other discussion of this topic), I think we want a normal test that is skipped in -short test mode (so it only runs on longtest builders or when executed locally manually).

As long as it's possible to implement a reasonably fast/efficient test without needing to rely on git, I don't see a reason for the test to be implemented out of tree; that would make it difficult to reproduce the result of builders locally, and it would contribute negatively to #39054. The cmd/internal/moddeps, misc/reboot, and test/winbatch.go tests may be good sources of inspiration. (To avoid needing git, a temporary directory can be used to compute the expected result, and check the tree against that.)

So, I'll retitle this to be less about x/build and builders and more about investigating adding a test in the main repo. Once a test is added, it will be automatically executed by longtest builders (and via SlowBots). (If it turns out to be infeasible to add as a normal test in-tree, we can reconsider.)

[dmitshur]

Managing backports can become much more expensive if skew is introduced in a new major Go release. We've been lucky that go.mod and vendor skew hasn't made it into any major Go release so far (although it did happen on tip, e.g., #36851), but I think it's important we have an automated measure so we can be confident it can't happen due to human error.

I'm going to make this a release-blocker for Go 1.16: we should either resolve this issue by adding a test, or if there isn't enough time, we must manually verify that there is no skew introduced in Go 1.16 release candidates and the final release.

CC @golang/release.

[gopherbot]

Change https://golang.org/cl/283643 mentions this issue: cmd/internal/moddeps: check content of all modules in GOROOT