proposal: cmd/go: ignore +incompatible versions as of Go 1.14

[bcmills]

Abstract

This is a proposal to ignore +incompatible versions found in the module graph starting with Go version 1.14.

Background

The go command requires that the import path of a module (or package within a module) match its semantically-versioned API. In particular, starting with major version 2 — the first breaking change from the API stabilized in major version 1 — the module path must end with a /vN suffix indicating the major version of its API.

However, prior to the introduction of modules, many Go package maintainers had already reused existing import paths (such as github.com/google/go-github) across multiple major versions. To accommodate the migration to modules for users of those packages, the vgo prototype — and the initial module support released in the go command in Go 1.11 — allowed those existing major versions to be used directly, and even preferred them over compatible versions (#26238) under two conditions:

1. The module with the incompatible version must not contain an explicit go.mod file.

2. The version must be annotated in the user's go.mod and go.sum files with the suffix +incompatible, indicating that the selected version is not compatible with the original API for that path.

Unfortunately, those exceptions introduce a number of problems:

• proxy.golang.org: accidentally publishing a major version interplays poorly with Go modules #34189: If a user adopts modules (by adding an explicit go.mod file) and then accidentally tags a pre-module commit with an inappropriate major version, the erroneous pre-module tag will always be semantically higher than the highest valid module-enabled release tag (1.99[…].99[…]).

• cmd/go: go modules ignores go.mod in semver repos not using semantic import versioning #27009: If a user has already tagged a major version above 1 and adopts modules (by adding an explicit go.mod file), then they must either adopt the major-subdirectory layout for their project (disrupting developers on the project), or break upgrades for non-module users (who will be expecting the unversioned import path).

• cmd/go: creating v2+ modules has lower success rate than it could #31543: Since the damage of introducing breaking changes is already done, users often expect that their v2-or-higher repository can be migrated to modules without changing its import path, but for various reasons that does not work in the general case.

• cmd/go: tries to download v2.2.2 (no suffix) when checking a replaced v2.2.2+incompatible for a missing import #33795, cmd/go: foo@v2.0.0+incompatible currently allows Semantic Import Versioning to be optional #32695: Since +metadata tags in general are not semantically meaningful, the fact that +incompatible is semantically meaningful requires numerous special cases that are difficult to test and maintain, and creates confusion about when incompatible versions are or are not allowed.

• cmd/go: 'go get' with semantic-version prefix doesn't fall back to matching tag or branch #29731: The possibility of using a +incompatible version led us to support major-version wildcard queries, which are only useful for legacy repos and interfere with more useful branch queries.

• cmd/go: 'go mod download' fails with xyz+incompatible replacement targets while 'go mod verifies' works #34254: The constraints on +incompatible versions derive from the module path, but that introduces even more complexity (and makes things more difficult to debug) with replace directives, which involve two module paths that may or may not impose the same constraints on the corresponding versions.

---

In contrast, for another interesting case of legacy tagging — semantic versions with metadata (#31713) — we came up with what I believe is a simpler solution: instead of accepting the non-canonical version tags as-is, we instead rewrite them to canonical pseudo-versions with an appropriate major version.

In light of the problems we have encountered with incompatible major versions, I believe that we should have applied a similar strategy for incompatible versions: perhaps using them for “latest” version resolution, but rewriting them to canonical pseudo-versions.

Unfortunately, the decision was made, and cannot be unmade in light of our subsequent experience without breaking compatibility.

---

...or can it?

Observation

Since a +incompatible version cannot have an explicit go.mod file, it cannot impose any transitive requirements on module selection. Therefore, a +incompatible version selected as the minimal version of a module cannot impact the version selected for any other module.

This implies that if we ignore the +incompatible versions in the module graph entirely, we will not accidentally drop requirements that pertain to other modules.

This leads to the following proposal (see the comment below; updates will be linked from here).

CC @jayconrod @thepudds @hyangah @katiehockman @heschik

[bcmills]

Proposal

I propose that, if the main module's go.mod file specifies go 1.14 or higher:

• If a module path passed to go get does not have a major-version suffix, and its major version is incompatible with the path, and the corresponding version of the repository does not contain an explicit go.mod file, then the version should be resolved to a pseudo-version with a compatible major version.

  • I note that https://proxy.golang.org/k8s.io/client-go/@v/v2.0.0-alpha.1.info currently resolves the non-canonical version to the corresponding +incompatible version. Under this proposal, it would be updated to instead resolve to the compatible pseudo-version.

• If an existing +incompatible version is found in the main module's go.mod file, it should be similarly rewritten to a canonical pseudo-version.

  • Proxies may continue to serve +incompatible versions, and should not redirect a version requested with an explicit +incompatible suffix to the corresponding pseudo-version.

• If a +incompatible version is found in the requirements of some other module in the build list, that version constraint should be ignored entirely, as if the other module had not required it at all.

  • If the +incompatible module is needed to satisfy an import from some module during a build operation, go mod tidy, or go mod vendor, the version of that module should be re-resolved to latest and the dependency added to the main module's go.mod file.

• When resolving the latest or upgrade version of a module, the go command should first check the latest version of that module tagged with a compatible version.

  • If that version includes a go.mod file, then that version should be used and no further search performed.

  • Otherwise, the highest release at each major version should be checked for a go.mod file. The last such release that lacks a go.mod file should be resolved to a pseudo-version, and that pseudo-version should be used as the latest or upgrade version of the module.

[thepudds]

• If a +incompatible version is found in the requirements of some other module in the build list, that version constraint should be ignored entirely, as if the other module had not required it at all.
  • If the +incompatible module is needed to satisfy an import from some module during a build operation, go mod tidy, or go mod vendor, the version of that module should be re-resolved to latest and the dependency added to the main module's go.mod file.

Is this saying that if you are building module foo, which requires module bar, and bar has a go.mod containing require baz v2.3.4+incompatible, then that would be interpreted as require baz/v2 latest?

If so, does that mean the version of baz selected for the build could change (that is, ending up with some later commit than the commit tagged v2.3.4 )?

And if so, what is the brief rationale for that?

[bcmills]

Is this saying that if you are building module foo, which requires module bar, and bar has a go.mod containing require baz v2.3.4+incompatible, then that would be interpreted as require baz/v2 latest?

It would be treated as if module bar's go.mod file did not mention baz at all.

In that case, your module (foo) would resolve baz@latest (not baz/v2@latest), and add that dependency to your (foo's) go.mod file, all as usual for a package not found in the existing module graph.

That means that you could indeed end up with a version of baz more recent than v2.3.4, which is exactly what we want if (say) the tags occur in the sequence v1.0.0, v2.0.0 (incompatible), […], v2.3.4 (incompatible), v1.5.0 (with the go.mod file added in between v2.3.4 and v1.5.0).

[bcmills]

The rationale is that I don't want to make the MVS step use any notion of “version precedence” that isn't ordinary semantic versioning.

(I especially don't want to have to walk back from v1.5.0 looking for the discontinuity in order to figure out the relative ordering of v1.5.0 and v2.3.4+incompatible, and I don't want to have to re-resolve the pseudo-version corresponding to v2.3.4+incompatible every time I reload the build list.)

[thepudds]

In that case, your module (foo) would resolve baz@latest (not baz/v2@latest), and add that dependency to your (foo's) go.mod file, all as usual for a package not found in the existing module graph.

That means that you could indeed end up with a version of baz more recent than v2.3.4, [...]

But would that then mean go.mod is not providing reproducible builds if the versions selected in the build change upon upgrading to Go 1.14? If so, would that be reasonable to do given prior statements around future releases being able to properly consume go.mod files defined by older releases? There might be some shades of grey there, including for example Go 1.13 invalidated some go.mod files that built under Go 1.12, but there is at least an argument to be made that reporting a fatal error for a previously uncaught problem is different than shifting the versions used.

(Sorry for the multiple questions; mainly still trying to see if I understand the proposal).

[bcmills]

But would that then mean go.mod is not providing reproducible builds if the versions selected in the build change upon upgrading to Go 1.14?

It would mean that the transition from Go 1.13 to Go 1.14 would not reproduce the same set of dependencies, but since the entire toolchain and standard library changes at each major release, that boundary is not exactly “reproducible” to begin with.

Once you run a go build or go mod tidy using 1.14, then each successive build using 1.14 would produce the same result, just not necessarily the same result as when using 1.13.

The more interesting problem occurs when going back from 1.14 to 1.13, since the version selected (and written to the go.mod file) by 1.14 may be “lower” (in the strict-semver sense) than the version selected by 1.13.

[gopherbot]

Change https://golang.org/cl/194600 mentions this issue: cmd/go: strip trailing slash from versioned arguments

[jayconrod]

It would mean that the transition from Go 1.13 to Go 1.14 would not reproduce the same set of dependencies, but since the entire toolchain and standard library changes at each major release, that boundary is not exactly “reproducible” to begin with.

I haven't fully considered this yet, but it seems like a significant, breaking change in MVS. This would change builds in a more significant way than toolchain and standard library changes do (those tend to be in Hyrum's law territory).

An obvious consequence: suppose you depend on module A (which has migrated to modules), which depends on module B (which has not) at v2.0.0+incompatible. With this change, the B requirement is ignored, and we get a pseudo-version based on v3.0.0, the newest major version of B without a go.mod file. The B API has been rewritten between major versions, and the upgrade breaks the build. The author of the main module now needs to either replace B or go get B@v2.0.0, despite not being familiar with B or having any explanation of why the build broke.

(of course this can already happen if someone else requires B v3.0.0+incompatible)

[bcmills]

@jayconrod, that's an interesting consideration (and somewhat ties in to #34165).

One alternative would be to only apply this proposal if the main module's go.mod file specifies go 1.14 or higher, so that the need to re-resolve versions would be opt-in at that point.

That has the advantage of retaining compatibility with as much of the +incompatible support as actually works today, but the disadvantage of not reducing the complexity (and bugs) induced by that support.

It also has the disadvantage of changing the meaning of a module's requirements depending on which other module is importing it, which makes fixes and patches more difficult to test — but perhaps the advantage of clearly attributing the change outweighs that disadvantage, and at any rate the main module can already change the meaning of a module's requirements using replace directives.

[bcmills]

Thinking about it some more, gating on go 1.14 in the main module's go.mod file seems like the clear way to go. Edited the proposal accordingly.