cmd/go: default to & improve -mod=readonly

[rsc]

For Go 1.16, I propose to default to & improve -mod=readonly, making -mod=mod completely equivalent to -mod=readonly.
This would mean that go commands other than go mod ... and go get no longer update versions in go.mod.
If go.mod needs to be fixed, these commands would report that error and stop.
And the errors in go list should be reported only for the packages that can't be resolved - the overall go list operation should succeed.

Discussed previously on golang-tools@: https://groups.google.com/g/golang-tools/c/BRCgqwWLwoY.

This is what I sent there:

As you probably know, we're trying to make Go 1.16 the release when there are no blockers remaining for anyone to adopt modules, including things like "go get -b" which Jay has been working on.

I'm wondering, as part of this polishing of modules, if we should change the default behavior and stop doing so much updating of go.mod.

Obviously I can justify the current behavior at length, and have in the past, but I think the past thinking falls apart a bit when users make mistakes.

For example, I was working in a module recently (let's call it m) and had a package (let's call it p), and I decided to rename the package (to q). I caught most of the m/p import paths and updated them to m/q. But not all, and the next time I ran "go build", the go command went off trying to find some other module to provide the no-longer-existent m/p. When m/p is a real thing that exists, automatically updating go.mod to use it makes some sense. When m/p is a mistake that needs updating, doing network work just slows down the real fix.

Another example is adding an import you didn't actually want. You might copy a file into a module m from somewhere else, and when you "go build" it adds something to go.mod silently, perhaps even changing the versions of existing require statements. In this case - since you don't want the import - it would be better for the tool to stop and tell you about it instead of doing additional damage that you will have to undo.

I suspect there are many more of these. In general, while the automatic additions to go.mod seemed good on paper, my impression is that they haven't delivered the vision we wanted: they do the wrong thing too often, they slow down the compile-edit-debug cycle for typos, and they are confusing.

We already have a mode that reports errors instead of making changes, namely -mod=readonly. That flag does not apply to "go get" or "go mod" - those commands' job is to update go.mod and they would still do that - but it stops other commands like "go list", "go build", "go test" from changing go.mod. If we made this mode the default and also cleaned up some of its rough edges (go list can probably do a better job reporting partial results, for example), that might go a long way to fixing the problems I listed above.

If we do change the default, it would probably make sense to change -mod=mod (the name for the current automatic updating) to something like -mod=autoupdate, leaving -mod=mod as an alias for that of course.

The command "go get" (with no arguments) has always done a build of the package in the current directory, including downloading any necessary dependencies. With a default -mod=readonly, "go get" would become the way to do a one-time fix-up of go.mod. If "go build" or "go list" fails because of a missing requirement or other problem with go.mod, "go get" would be the way to fix it. If "go test" fails due to an import specific to the test, we'd need to provide a way to deal with that, probably adding back "go get -t", which was the pre-modules way to download the dependencies for a test.

Along with this I think you'd want to change goimports to have a way to say what new requirements it thinks go along with the import paths. Then tooling invoking goimports could present the new imports and the new go.mod requirements together. Users opting in to that kind of automation would still have the option of both. It just wouldn't happen automatically in commands like "go build". (This listing of new go.mod requirements only matters when goimports has exhausted your module's current requirements and is grubbing around in your module cache. When it finds a match, especially a v0, you probably do want to get the version it found and not whatever the go command decides is latest from the internet. So letting goimports specify the new go.mod requirements would correct a sort of race in the current behavior anyway.)

I haven't heard many people suggesting we should default to -mod=readonly before, so maybe others haven't seen the current behavior as too much of a problem. But maybe people just figure it's not worth bringing up because it can't change at this point. I don't believe that's true - if the behavior is wrong, then Go 1.16 would be the time to get it right (and soon).

[DisposaBoy]

I haven't heard many people suggesting we should default to -mod=readonly before, so maybe others haven't seen the current behavior as too much of a problem.

FWIW, I personally started complaining about this whole magic behaviour as early as #29452, but I didn't feel like anyone was taking it seriously so from that point on I essentially stopped participating in the discussion (IIRC I was even invited to comment on a proposal moving things forward which I naturally ignored).

Not only do I think the default behaviour is C@#0%, I'm 99% sure it's a security issue (which I believe I brought up somewhere else). If you make a simple typo, it goes out and pulls the package off the internet with no input from the user. I know this happens in RL because we previously ran an experiment by registering a few common typos of golang.org and observed people pulling package from it. Those requests returned 404 and was before the checksum database, so I don't know the real security impact but I feel like the fact that it was/is possible to some extent is bad enough.

TL;DR

I'm still salty, but IMO this is the best thing to happen to Go modules since their introduction and I support it fully 💯

[rsc]

Another compelling example:

% go run main.g
go: finding module for package main.g
^C

[rasky]

Actually, running go run MODULE is a very good way to have people running CLI tools written in Go. Would that be broken by this proposal for people running in this command in a folder where go.mod doesn't even exist?

[jayconrod]

@rasky In module mode, when invoked outside a module, go run can't build packages outside the standard library that require a module lookup. You'll get an error like this:

$ GO111MODULE=on go run use.go
use.go:3:8: cannot find module providing package golang.org/x/mod/semver: working directory is not part of a module

That was #32027. Before that, pretty much everything required several network fetches, so builds were slow and non-reproducible. We felt it wasn't a good user experience, especially for new users.

[myitcv]

@rsc just thinking of side effects of this change, would this change make a go mod download a necessary prestep in CI?

Consider a module which we know to have a complete go.{mod,sum}, that is being tested with an empty module cache (a regular situation on CI). Running go test will necessitate a download of modules, in order to determine the go.{mod,sum} are indeed complete. Are we saying this download will no longer automatically happen? And therefore that a go mod download will be a necessary prestep, that itself will potentially modify the go.{mod,sum} files?

[bcmills]

@myitcv, -mod=readonly prevents resolving new modules, but it does not prevent the go command from fetching modules that are already both selected by the requirements in the go.mod file and listed in the go.sum file.

So as long as the module is already tidy (or at least complete) when it gets sent to CI, no explicit go mod download should be needed.

[rasky]

That was #32027. Before that, pretty much everything required several network fetches, so builds were slow and non-reproducible. We felt it wasn't a good user experience, especially for new users.

Thanks for the pointer, I had missed that discussion. I'm not really happy about the outcome but such is life.

[myitcv]

@bcmills - thanks, I was forgetting we are actually defaulting to the semantics of -mod=readonly here, despite that being in the title. Minor brain fade. Apologies for the noise.

[gopherbot]

Change https://golang.org/cl/251881 mentions this issue: cmd/go: default to -mod=readonly in most commands

[gopherbot]

Change https://golang.org/cl/251879 mentions this issue: cmd/go: let 'go list -mod=readonly' succeed if go.mod is inconsistent

[gopherbot]

Change https://golang.org/cl/253837 mentions this issue: cmd/go: update tests to work with -mod=readonly on by default

[gopherbot]

Change https://golang.org/cl/253745 mentions this issue: cmd/go: refactor modload.Import for better -mod=readonly errors

[gopherbot]

Change https://golang.org/cl/253744 mentions this issue: cmd/go: refactor -mod flag parsing

[mvdan]

@rsc @bcmills this is missing from the release notes still, right? It seems like a big change that should be there before beta1 is tagged.

[gopherbot]

Change https://golang.org/cl/268859 mentions this issue: cmd/go: release note for -mod=readonly by default

[gopherbot]

Change https://golang.org/cl/285113 mentions this issue: content/static/doc: document -mod=readonly enabled by default

[jdog8541]

Man you guys are kicking ass!!! Keep up the good work!