cmd/go: give library/tool maintainers advice on how to word READMEs during modules transition

[wsc1]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.11

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

⎣ ⇨ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/scott/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/scott/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/wv/qs6x44tj2bs8rm7fqw8p79yw0000gn/T/go-build789971487=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

⎣ ⇨ GO111MODULE=on go get zikichombo.org/sound/...
go: cannot find main module; see 'go help modules'

What did you expect to see?

not sure, most things haven't really been working for me with modules yet.

What did you see instead?

⎣ ⇨ GO111MODULE=on go get zikichombo.org/sound/...
go: cannot find main module; see 'go help modules'

A standard workflow would be to "go get whatever...." and then work on the resulting code,
without having the module set up at first.

It seems to me it would be easier if go get with modules on would
go get the module requested on the command line if there is no go.mod. Now I have to turn modules off, go get, and then turn them back on.

[thepudds]

@wsc1 I might not be following, but the normal workflow when you are starting a fresh module is to first create a go.mod (usually via go mod init or go mod init example.com/my/module). Among other things, this gives the go tooling a place to record dependencies as you then start doing go get or adding imports to your code and doing things like go build. In particular, those dependencies get recorded in that go.mod file.

Are you asking to flip that sequence, and/or make go mod init optional?

In general, when running a command like go get in module-aware mode, the resulting behavior depends on which module you are currently "in" as the current module, which means it depends on where you are in your filesystem hierarchy when you execute the go get command (because it walks up the filesystem hierarchy looking for a go.mod file to know what module you are "in").

Possibly semi-related: one aspect that might catch people by surprise is if the GO111MODULE environment variable is explicitly set to GO111MODULE=on, then getting a tool via something like go get example.com/cmd requires a go.mod file and hence will fail if run outside of a module (that is, if run outside of a file tree rooted by a go.mod). The current error message is:

go: cannot find main module; see 'go help modules'

Part of the overall issue is that modules are currently opt-in, and a full solution for operating outside of all modules will likely wait until GO111MODULE=on becomes the default behavior. See #24250 and #24250 (comment) for more related discussion, including this comment from there:

This isn't blocking Go 1.11. The main reason being that if GO111MODULE=auto (the default) then we can't reasonably do anything with modules when outside a go.mod tree. That's the opt-in. And inside a go.mod tree this already works.

Sorry if this is not directly on target...

[thepudds]

@gopherbot please add label modules

[wsc1]

@thepudds
you wrote
"""
@wsc1 I might not be following, but the normal workflow when you are starting a fresh module is to first create a go.mod (usually via go mod init or go mod init example.com/my/module). Among other things, this gives the go tooling a place to record dependencies as you then start doing go get or adding imports to your code and doing things like go build. In particular, those dependencies get recorded in that go.mod file.
"""

I have been playing them and following the discussion and started using them in some projects, so I'm not a modules noob (eg I know what go.mod and go.sum store), but there are certainly details I'm not aware of.

Since operating "outside" a module is the entry point on very many project pages that I've seen: "use: go get a/b/....". (no module specified). I would consider it unfortunate to exclude browsing/touring of go projects from the normal work flow with GO111MODULE=on. I would encourage it also to be made as simple/untricky as possible with modules as soon as possible, so people aren't turned away by such problems. That would produce more uses and thus give modules functionality more of a solid basis on which to move out of "preliminary" status.

I also do not think anyone can really define what "the normal workflow" is, I thought the issue tracker was to inform those manning it what the workflows are, not the other way around, and modules are moreover preliminary.

Thanks for your pointers to the other issues operating outside a module for go1.12. It seems to me to be essentially the same problem conceptually, In view of those issues and the timeline, we may update the project pages documentation relying on modules so as not to give users impression "go get" works out of the box, which is unfortunate. If y'all considered changing the behaviour of go get with GO111MODULE=on so it does something reasonable, it would smooth things out for us in this respect.

[thepudds]

Hi @wsc1, at the start of my comment, I was trying to describe the normal workflow "when starting a fresh module".

I 100% agree it is a shock when go get doesn't work, and one of the main ways go get fails right now is when GO111MODULE=on is set and you are outside of a module (and not starting a fresh module).

It is a significant way people are "stubbing their toes" on modules.

That said, I believe the plan is for go get to work when modules are enabled even when go get is run outside of a module (including so all the repos that say "run go get foo to install" won't need to be updated), but as far as I'm aware the reason it is not in Go 1.11 is because modules rely a go.mod right now being in the filesystem hierarchy and the opt-in nature of modules also present some challenges and hence there is a good chance the "full" solution might wait until modules are no longer opt-in.

Here is one sample statement about "clearly this must work eventually" from #24250 (comment):

$ vgo get github.com/golang/dep/cmd/dep@v0.4.1
cannot determine module root; please create a go.mod file there

This clearly must work eventually. The thing I'm not sure about is exactly what this does as far as the version is concerned: does it create a temporary module root and go.mod, do the install, and then throw it away? Probably. But I'm not completely sure, and for now I didn't want to confuse people by making vgo do things outside go.mod trees. Certainly the eventual go command integration has to support this.

And of course, things work more as expected if you temporarily do GO111MODULE=off or leave GO111MODULE unset (because then things work more like they used to work in Go 1.10 regarding go get behavior if you are outside of a module or inside GOPATH).

In short, I agree with what you are saying that go get outside of a module with module-mode enabled should work. I was just trying to provide some context around the way it works now and why as of this moment Go 1.11 does not yet have the "full" solution that everyone will want regarding go get behavior.

(As far as I understand things, anyway... and I'm happy to learn more if my understanding is off).

[wsc1]

Sounds like we're understanding each other better!

One thing particular to my case is I'm involved in some projects which have decided to use modules collectively (and non-nested). As a result, the development process can be much facilitated in terms of allowing independence between the projects, because modules will find the right dependencies between them and the heads can be out of sync. All that is great, as in this scenario the independence between heads seems very much necessary.

But old style go-get won't work for us reliably or even often because this independence/desynchronisation at the heads will be too common. This puts us in a bit of an awkward position w.r.t. go get and modules, perhaps more so than projects which have one module. It would be nice to say to those involved "we use modules." and not "we use modules, so you have to do blah=auto and GOPATH=.... and cwd=...."

So in this kind of setup, a collection of inter-related, non-nested modules is perhaps a new use case to consider. But it may also help provide insight into how interdependent projects using single modules will interact -- there may be a similar need for them as well to say to their users that go get needs modules, especially if using them to interact with other projects, and they might also prefer just saying "use modules" as compared to listing all the requirements under GO111MODULE=auto.

I'll keep an eye out in case y'all decide and have resources to address go get with GO111MODULE=on before 1.12.

Thanks

[wsc1]

As an update,

It seems GO111MODULE=auto is just as difficult to get working with module aware go get as GO111MODULE=on.

Below is what we put in our dev README. Any other ideas/comments?

======
Due to the very limited nature of current go get support with modules we recommend

1. Turning off modules and setting GOPATH for go get.
2. If go get fails to build, run go get -d for download only
3. After go get succeeds via one of the above, turn modules on while working with ZikiChombo modules.
4. For example, if you want to work with the code in an isolated environment under some directory D, then

cd D
mkdir src
GOPATH=`pwd` GO111MODULE=off go get -d foo.org/P/...  # with P a project

should work whether or not heads are desynchronised.

Then set GO111MODULE=on for go build, go test, etc.

[dolmen]

@wsc1 My own experience: If you work in module-aware mode (GO111MODULE=on) with a go.mod where your main package is defined, just stop using go get. Because in module-aware mode, fetching and installing dependencies becomes an automatic operation launched by go build/go test/go list when needed based on go.mod content. If you keep go.mod up to date, everything just works without requiring to manually fetch dependencies.

1. $EDITOR main.go
2. GO111MODULE=on go mod init main
3. GO111MODULE=on go mod edit -require=zikichombo.org/sound
4. GO111MODULE=on go build

[wsc1]

@dolmen Thanks, sounds good if you have a main.go or other entry point in mind before working with the module.

At least in my experience, when I want to look at and play with some project, I usually don't have enough information to have a main.go or other module in mind.

But your idea may serve as a useful template for some examples...

[myitcv]

@wsc1 I think everything here is working as intended as you and @thepudds effectively concluded.

#24250 covers the case of a "global" install of a command/tool with GO111MODULE=on outside of a module context, i.e. no go.mod.

If you are in a module context then go (get|install) will work as expected (with perhaps one exception, where you actually want a go (get|install) to be "global" despite being in a module context, but again that's covered by #24250).

Otherwise if you are in GOPATH mode the go command should just work as expected with pre Go 1.11 behaviour.

I'm going to retitle this issue to be more about what advice we should give package/tool maintainers on how to word their READMEs during the transition (@thepudds and I discussed such an issue, I can't find one for now), because this point has come up on multiple occasions.

[myitcv]

/cc @rsc @bcmills

[bcmills]

A fix for #24250 should make it into the 1.12 release. If the latest version of your project (latest tagged, or latest untagged if you have no tags) has a go.mod file that specifies a complete, consistent set of module requirements, then go get example.com/path/to/your/binary should work consistently starting with 1.12.

Given that, I don't think most maintainers should need to update their READMEs at all, but please do let me know if I've missed something.