proposal: cmd/go: list available major versions in 'go list -u -m all'

[thepudds]

In #44550 (comment), Russ wrote:

However, we absolutely do expose the existence of v2 in other people-friendly places. There is a banner advertising v2 at the top of https://pkg.go.dev/github.com/russross/blackfriday, for example, and I think the go.mod view in VSCode Go shows a tooltip (powered by gopls) to notify about the existence of v2 as well (if not, it will soon).

In addition to the locations Russ mentioned, it would be helpful to expose the existence of a major version upgrade in 'go list -u -m all'.

I think this would help reduce toe-stubbing around major versions.

I believe this has been mentioned as a possibility in the past (e.g., #44550 (comment)), and for example discussed as part of a much broader & complex #40323, but I'm filing this short proposal because I'm not sure there is an existing issue that narrowly targets 'go list -u -m all'.

Spelling

There would need to be a decision on how it would be spelled. Bikeshedding can come later, but it could possibly use words like 'breaking' or 'major' in the output. ('incompat' or 'incompatible' I think would probably confuse people given the '+incompatible' in go.mod. 'latest' might also be best to avoid. It might be nice pick a word that hints you might need to do some work to upgrade).

So perhaps something along the lines of:

github.com/rogpeppe/go-internal v1.8.0 [v1.8.1] [v2.0.1 breaking]

But the exact spelling could come later if there is interest in the core capability.

Concerns

• One concern expressed in the past I think was people might not like it if they see a major version listed in 'go list -u -m all' that does not get picked up by a subsequent invocation of 'go get -u'. That is a fair concern, but I think it is still a net benefit to be aware of an available major version even if the cmd/go does not automatically upgrade major versions (yet 🤞).

• One other concern could be the stability of 'go list' output for programs that parse its output, but it is already somewhat complex and has already changed in the past. Parsing 'go list -u -m all' without using the '-json' flag is probably already a bad idea.

• Another concern has been performance for cmd/go. Perhaps parallel requests would make it fast enough for the common case of using a GOPROXY? Also, if it helps, I think it would be reasonable to assume no vN means no need to look for a vN+1 -- in other words, no need to handle holes in the major version numbering if that helps performance at all.

Existing output

For reference, here is some sample output of go list -u -m all today:

$ go list -u -m all

github.com/thepudds/fzgen
github.com/google/go-cmp v0.5.6 [v0.5.7]
github.com/kr/pretty v0.1.0 [v0.3.0]
github.com/kr/pty v1.1.1 [v1.1.8]
github.com/kr/text v0.1.0 [v0.2.0]
github.com/rogpeppe/go-internal v1.8.1
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e
golang.org/x/net v0.0.0-20211015210444-4f30a5c0130f [v0.0.0-20220225172249-27dd8689420f]

Implementation

If this proposal is accepted, I would be interested in sending a CL.

[mvdan]

Should the output perhaps clarify that new major versions may require a new /vN module suffix? For example, with your example, a naive user might try go get github.com/rogpeppe/go-internal@v2.0.1 to upgrade, but that would almost certainly fail. Perhaps you could print something like [v1.5.7] [v2.0.1 /v2] as a hint, which could already imply "breaking".

[ianlancetaylor]

CC @bcmills @matloob

[seankhliao]

Given that major upgrades will always require more work both to query and to do the actual upgrade, maybe it would make sense to put this behind a flag? Or maybe #29666

How would this information be represented in -json (and -f)?

Does this interact with the -versions flag? Right now the output matches closely.

[seankhliao]

cc @icholy for input/experience based on https://github.com/icholy/gomajor

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[mpx]

go list -u -m shows module versions available -- I think it's the best place to highlight new major versions as well.

AFAIK, doing this properly is currently expensive. I currently use tooling that tracks & caches the Index to detect new major versions. Other tools try querying version N+1, but this fails when major versions are skipped.

In any case, I think the user experience is more important. Any query costs should be reduced.

The module proxy protocol probably needs to be updated for this use case. Perhaps by adding a @v/related or similar endpoint that lists related modules (major versions). The recommended upgrade path(s) could be included if captured by go.mod in a structured way. Eg:

• github.com/golang/protobuf could recommend google.golang.org/protobuf
• github.com/boltdb/bolt could recommend github.com/etcd-io/bbolt

(just examples -- those decisions should be left to the module authors)

[rsc]

/cc @bcmills @matloob

[bcmills]

Currently go list -u has the property that if an upgrade is listed, the module can be upgraded to the listed version using go get. This proposal does not have that property: a major-version upgrade could be listed, but the user would not be able to upgrade to it without unavoidable code changes. I am concerned that it would add unactionable noise to an otherwise precise command.

Moreover, we have only had support for module deprecation notices since Go 1.17. I don't think we have yet had enough time to see how well that addresses the problem space, but it does at least afford the opportunity to list related or recommended substitute modules (as described by @mpx).

[bcmills]

I also want to note that go list -u -m all is not usually an appropriate upgrade command. (all potentially — and frequently — includes additional modules that are not relevant to the packages imported by the main module.)

We probably ought to have some kind of substitute for that — maybe allowing go list -u to work on package patterns? — but with the status quo it makes me question the value of other UX improvements to go list -u.

[icholy]

@bcmills go list -u -m direct #40364

[rsc]

It sounds like we shouldn't do this specifically, and that we should encourage people who want to deprecate their major version to use module deprecation notices, as @bcmills pointed out.