cmd/go: go1.16 default go build behavior changes seem detrimental to developer flow

[andrewstuart]

What version of Go are you using (go version)?

$ go version
go version go1.16rc1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY="*"
GONOSUMDB="*"
GOOS="linux"
GOPRIVATE="*"
GOPROXY="direct"
GOSUMDB="off"
GOTMPDIR=""
GOVCS=""
GOVERSION="go1.16rc1"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4000834506=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Tried to build a package in a development cycle from within vim while adding dependencies the way I have for 6+ years.

What did you expect to see?

Compiler errors relating to the code in my repository.

What did you see instead?

go build errors out telling me that a package is not provided by anything in go.mod and that I need to add it with go get

After running go get to add the package, I attempt another build and am met with another error telling me that there is a "missing go.sum entry for module providing package ... imported by ..." and prints out helpful commands, but does not keep me in the flow of development as Go has historically been great at by default.

I understand the reasons behind the desire to not constantly change go.mod and go.sum invisibly to the user, but I think it's worth rethinking this default behavior as it seems very much contrary to the ethos of go tooling up to this point. I've always appreciated when things just work while also trying to help me keep myself safe.

[dmitshur]

This is related to the intentional change in proposal #40728.

After running go get to add the package, I attempt another build and am met with another error telling me that there is a "missing go.sum entry for module providing package ... imported by ..."

Rather than running go get on a missing package, have you tried running go get . to get all missing dependencies of your own package? That should avoid having to repeat the step many times.

@jayconrod will know if there were improvements to the suggestions printed by cmd/go that landed since Go 1.16 RC 1. We'll have some updates to the suggested practice on the website when issue #43672 is resolved.

[dmitshur]

Can you please include what was the exact output printed from cmd/go that you found confusing, then we can tell if further improvements are needed. Thanks.

[seankhliao]

note the old behaviour is still available with -mod=mod, which can be provided by default with GOFLAGS

[andrewstuart]

Rather than running go get on a missing package, have you tried running go get . to get all missing dependencies of your own package? That should avoid having to repeat the step many times.

Yeah, that's what I've been doing today, though just go get (which works the same way). There's also apparently go get -d which may remove the need to run go get the second time to modify go.sum. Full disclosure, I'm importing kubernetes ecosystem packages, and so I am probably feeling this pain more than most, but it's definitely interrupting my loop and been a source of frustration.

I guess the main issue is that I think this is a significant departure, and feels like a regression, from historical Go tooling experience. While I think I understand the reasoning (prefer explicit), it's been a source of frustration. I'm also quite a bit more familiar with Go than many of the Go developers that I work with, so I'd imagine the output being shown, to go get each individually, will be assumed by many to be "the right way" to do it (and understandably so -- it's coming straight from the tool itself).

I see that -mod=mod is available, but even just typing that feels kind of goofy, like I'm reminding the tool that 1=1. I'd consider that an artifact induced by what's currently planned to be an unnatural-feeling default.

Finally, I'm trying to read more into the change proposal and link chains above, so I'll try to respond with some hopefully more educated opinions later.

[jayconrod]

As Dmitri mentioned, this change in default behavior was discussed and approved in #40728. It's too late to change that for 1.16, and we'd need significant new information to reopen that discussion.

If there's something specific we can do to make the experience better (better diagnostic message; simpler command to accomplish a common task), we'd be happy to consider that. I expect there will be a few rough edges, since -mod=readonly wasn't frequently used before. I've noticed and fixed a few problems with go.sum in particular since we flipped the switch.

Just for background, the feedback we've gotten on the previous behavior (-mod=mod) was almost uniformly negative. Personally, I got used to it after a while, but it was very surprising for new module users that commands that didn't seem to be related to dependency management (especially go list; see #29452) would modify go.mod or go.sum in any way.

[andrewstuart]

Yeah, that's understandable. With the reading provided it seems it was a wider reaching change than I'd initially hoped.

I think my thoughts on a better experience would be around providing error feedback, or go get behavior that helps users accomplish the goal in as few steps as possible. It's really jarring to do what the tool suggested then get another error with more suggestions.

Is there any way go get can either add transitive dependencies to go.sum automatically (I think this improves security posture anyway even if slightly; a change in code served over the network between the two commands could sneak something in), or immediately provide the feedback that currently comes back from the next go.build?

I'd also be glad to sling some code on this too, though I've still only prepped for the the process of contributing to Go, and haven't actually done it yet.

[jayconrod]

I think my thoughts on a better experience would be around providing error feedback, or go get behavior that helps users accomplish the goal in as few steps as possible. It's really jarring to do what the tool suggested then get another error with more suggestions

That's entirely fair. I've filed a couple issues about related errors (like #43653). More issues with specific examples would be helpful, too.

go mod tidy will generally fix all these problems, but it's a pretty blunt instrument that can have other side effects. I initially recommended it in most of the hint messages, but we decided it was safer for the go command to recommend narrower commands. But that's still what I'd personally recommend.

Is there any way go get can either add transitive dependencies to go.sum automatically (I think this improves security posture anyway even if slightly; a change in code served over the network between the two commands could sneak something in), or immediately provide the feedback that currently comes back from the next go.build?

go get pkg@version should make the changes to go.mod and go.sum that are needed to build pkg at version, including all packages it transitively imports. That may include adding or updating other requirements in go.mod, and it will add any missing sums for transitive dependencies.

I think I've fixed most cases where that wasn't happening, though it's possible there are still edge cases we haven't seen yet. #44129 is probably the biggest issue I know of. If it comes up often, we could backport the fix in 1.16.1. Please file issues if you see anything else like that.

I'd also be glad to sling some code on this too, though I've still only prepped for the the process of contributing to Go, and haven't actually done it yet.

Help is welcome! :) Feel free to @ me on any open issue, and I can help you get started.

[andrewstuart]

Sounds good, and I may take you up on that. Appreciate the responsiveness and as always, the Kubernetes community spirit. :D