cmd/go: don't error out due to inconsistent requirements in -mod=readonly mode if the 'go' version is higher than the toolchain in use

[bcmills]

There is an interesting wrinkle in the upgrade path to lazy module loading in Go 1.17 (#36460), illustrated in this example:

package main

import (
	"encoding/json"
	"fmt"
	"runtime/debug"

	_ "golang.org/x/text/number"
	_ "golang.org/x/tools/txtar"
)

func main() {
	info, _ := debug.ReadBuildInfo()
	b, _ := json.MarshalIndent(info, "", "\t")
	fmt.Printf("%s\n", b)
}
-- go.mod --
module example.com/m

go 1.17

require (
	golang.org/x/text v0.3.0
	golang.org/x/tools v0.1.1
)

---

gotip run . shows that the program is successfully built against the declared version of golang.org/x/text (v0.3.0):

$ gotip run .

{
        "Path": "example.com/m",
        "Main": {
                "Path": "example.com/m",
                "Version": "(devel)",
                "Sum": "",
                "Replace": null
        },
        "Deps": [
                {
                        "Path": "golang.org/x/text",
                        "Version": "v0.3.0",
                        "Sum": "h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=",
                        "Replace": null
                },
                {
                        "Path": "golang.org/x/tools",
                        "Version": "v0.1.1",
                        "Sum": "h1:wGiQel/hW0NnEkJUk8lbzkX2gFJU6PFxf1v5OlCfuOs=",
                        "Replace": null
                }
        ]
}

go1.16.4 run . with Go 1.16.4, on the other hand, errors out:

$ go1.16.4 run .

go: example.com/m: package golang.org/x/text/number imported from implicitly required module; to add missing requirements, run:
        go get golang.org/x/text/number@v0.3.3

And go1.15.12 run . successfully builds, but against a different version of x/text (v0.3.3 instead of v0.3.0):

$ go1.15.12 run .

{
        "Path": "example.com/m",
        "Main": {
                "Path": "example.com/m",
                "Version": "(devel)",
                "Sum": "",
                "Replace": null
        },
        "Deps": [
                {
                        "Path": "golang.org/x/text",
                        "Version": "v0.3.3",
                        "Sum": "h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=",
                        "Replace": null
                },
                {
                        "Path": "golang.org/x/tools",
                        "Version": "v0.1.1",
                        "Sum": "h1:wGiQel/hW0NnEkJUk8lbzkX2gFJU6PFxf1v5OlCfuOs=",
                        "Replace": null
                }
        ]
}

---

These differences are because:

• Go 1.17 prunes out the dependencies of golang.org/x/net, which is required indirectly by golang.org/x/tools, because no package relevant to the main module imports anything from golang.org/x/net.

• Go 1.16.4 includes the dependencies of golang.org/x/net, and notices that the selected version of golang.org/x/text is different from what is listed in the go.mod file. Since Go 1.16 defaults to -mod=readonly (cmd/go: default to & improve -mod=readonly #40728), it cannot change the go.mod file to restore consistency, and so it errors out.

• Go 1.15.12 includes the dependencies of golang.org/x/net, but defaults to -mod=mod, so it happily overwrites the go.mod file to restore what it sees as “consistent”, and then builds using the updated requirements.

$ gotip mod graph

example.com/m golang.org/x/text@v0.3.0
example.com/m golang.org/x/tools@v0.1.1
golang.org/x/tools@v0.1.1 github.com/yuin/goldmark@v1.3.5
golang.org/x/tools@v0.1.1 golang.org/x/mod@v0.4.2
golang.org/x/tools@v0.1.1 golang.org/x/net@v0.0.0-20210405180319-a5a99cb37ef4
golang.org/x/tools@v0.1.1 golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c
golang.org/x/tools@v0.1.1 golang.org/x/sys@v0.0.0-20210510120138-977fb7262007
golang.org/x/tools@v0.1.1 golang.org/x/xerrors@v0.0.0-20200804184101-5ec99f83aff1

$ go1.16.4 mod graph

example.com/m golang.org/x/text@v0.3.3
example.com/m golang.org/x/tools@v0.1.1
golang.org/x/text@v0.3.3 golang.org/x/tools@v0.0.0-20180917221912-90fa682c2a6e
golang.org/x/tools@v0.1.1 github.com/yuin/goldmark@v1.3.5
golang.org/x/tools@v0.1.1 golang.org/x/mod@v0.4.2
golang.org/x/tools@v0.1.1 golang.org/x/net@v0.0.0-20210405180319-a5a99cb37ef4
golang.org/x/tools@v0.1.1 golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c
golang.org/x/tools@v0.1.1 golang.org/x/sys@v0.0.0-20210510120138-977fb7262007
golang.org/x/tools@v0.1.1 golang.org/x/xerrors@v0.0.0-20200804184101-5ec99f83aff1
golang.org/x/mod@v0.4.2 golang.org/x/crypto@v0.0.0-20191011191535-87dc89f01550
golang.org/x/mod@v0.4.2 golang.org/x/tools@v0.0.0-20191119224855-298f0cb1881e
golang.org/x/mod@v0.4.2 golang.org/x/xerrors@v0.0.0-20191011141410-1b5146add898
golang.org/x/net@v0.0.0-20210405180319-a5a99cb37ef4 golang.org/x/sys@v0.0.0-20210330210617-4fbd30eecc44
golang.org/x/net@v0.0.0-20210405180319-a5a99cb37ef4 golang.org/x/term@v0.0.0-20201126162022-7de9c90e9dd1
golang.org/x/net@v0.0.0-20210405180319-a5a99cb37ef4 golang.org/x/text@v0.3.3
golang.org/x/crypto@v0.0.0-20191011191535-87dc89f01550 golang.org/x/net@v0.0.0-20190404232315-eb5bcb51f2a3
golang.org/x/crypto@v0.0.0-20191011191535-87dc89f01550 golang.org/x/sys@v0.0.0-20190412213103-97732733099d
golang.org/x/tools@v0.0.0-20191119224855-298f0cb1881e golang.org/x/net@v0.0.0-20190620200207-3b0461eec859
golang.org/x/tools@v0.0.0-20191119224855-298f0cb1881e golang.org/x/sync@v0.0.0-20190423024810-112230192c58
golang.org/x/tools@v0.0.0-20191119224855-298f0cb1881e golang.org/x/xerrors@v0.0.0-20190717185122-a985d3407aa7
golang.org/x/term@v0.0.0-20201126162022-7de9c90e9dd1 golang.org/x/sys@v0.0.0-20201119102817-f84b799fce68
golang.org/x/net@v0.0.0-20190404232315-eb5bcb51f2a3 golang.org/x/crypto@v0.0.0-20190308221718-c2843e01d9a2
golang.org/x/net@v0.0.0-20190404232315-eb5bcb51f2a3 golang.org/x/text@v0.3.0
golang.org/x/net@v0.0.0-20190620200207-3b0461eec859 golang.org/x/crypto@v0.0.0-20190308221718-c2843e01d9a2
golang.org/x/net@v0.0.0-20190620200207-3b0461eec859 golang.org/x/sys@v0.0.0-20190215142949-d0b11bdaac8a
golang.org/x/net@v0.0.0-20190620200207-3b0461eec859 golang.org/x/text@v0.3.0
golang.org/x/crypto@v0.0.0-20190308221718-c2843e01d9a2 golang.org/x/sys@v0.0.0-20190215142949-d0b11bdaac8a

To some extent this is working as intended: part of the point of lazy loading is to allow users to more easily upgrade away from transitive dependencies on cyclic, retracted, irretrievable, or otherwise broken module versions that are not otherwise relevant to the main module, so it should not be surprising that the transitive dependencies change between go1.16.4 and gotip.

---

We would like Go users who are actively developing with a mix of Go 1.16 and Go 1.17 to be able to continue to do so, as long as they only run go mod tidy using the version listed in the module's go directive or higher.

To make that work, we plan to change the go command so that inconsistent requirements in the main module's go.mod file are not treated as an error in -mod=readonly mode if the go version in that go.mod file is higher than the version of the go command in use.

We can then backport (really, reimplement) that change in upcoming minor releases of Go 1.15 and Go 1.16, so that users still on those versions can continue to use them in go 1.17 modules in readonly mode.

[bcmills]

@gopherbot, please backport to Go 1.15 and 1.16.

This change should be relatively uninvasive, and will allow users on those releases to more smoothly interoperate with users on Go 1.17.

[gopherbot]

Backport issue(s) opened: #46138 (for 1.15), #46139 (for 1.16), #46140 (for 1.17).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[bcmills]

In the meantime, note that users on gotip can ensure than their go.mod dependencies remain interoperable by running:

go mod tidy -go=1.16
go mod tidy -go=1.17

The first step will convert the module back to Go 1.16, incorporating any otherwise-pruned-out dependencies, and the second step will add back the indirect dependencies needed to maintain Go 1.17 completeness invariants.

[bcmills]

I'm not 100% certain that we should actually do this. #46141 will make it so that the sequence

go mod tidy -go=1.16
go mod tidy -go=1.17 -compat=1.16

produces a set of module requirements that are consistent with Go 1.16, retains checksums needed by Go 1.16, and also maintains the invariants needed by Go 1.17 (enabling module graph pruning and lazy loading).

Given that, maybe it's better for Go 1.16 to explicit error out if it would use requirements that are different from those explicitly listed in the 1.17 go.mod file.

[bcmills]

On further consideration, I think we should not do this. The choice to use a different set of versions from what is clearly intended by the go.mod file should be intentional and explicit, but suppressing errors here would be implicit and could be unintended.

Instead, we should provide some command in go 1.17 that can report inconsistencies between the 1.17 and previous-version views of a module — perhaps the same go mod why flag as in #46141.