cmd/go: module mode removes concept of global docs

[rhysh]

The ergonomics of go doc have regressed in go1.13beta1.

Background on why and how this affects me: When I download code, I usually arrange it in my GOPATH (to organize it—even if it's not Go code). I usually have a shell (or ten) open, and the working directory is often one of the projects (under GOPATH) that I've worked on recently. Sometimes I want to look up docs for popular packages without opening a browser, especially so I can pipe the output through grep or head to focus on relevant bits when I want to answer a question in chat/email.

As the projects in my GOPATH convert to being modules, they stop working for this kind of "go doc" usage. When my shell's working directory is in a module, "go doc" can only find packages that are in modules referenced by the current module. If I use a package's fully-qualified name to look up its docs, I end up with unintended changes to the nearby go.mod file.

This seems like an unfortunate implication of a bunch of other rules that make sense in isolation. But is this the right behavior for the tool?

CC @rsc (closing the loop from GopherCon), @bcmills (modules), and @dmitshur (go doc)

(Maybe I need to write/use a shell alias that runs go doc in the current directory, and on failure tries a second time in a different directory. I wonder if there are other cases where a fallback go.mod file would be helpful—here I'd use it for docs, but it could also set a default version in other cases where Go tools have to guess or decide to use "today's latest".)

What version of Go are you using (go version)?

$ go version
go version devel +0212f0410f Thu Aug 15 17:49:11 2019 +0000 darwin/amd64
$ go1.13 version
go version devel +60f14fddfe Wed Jun 26 16:35:14 2019 +0000 darwin/amd64
$ go1.12 version
go version go1.12.9 darwin/amd64

Does this issue reproduce with the latest release?

This behavior is new since Go 1.12. It exists in go1.13beta1 and current tip, where module mode is enabled by default.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rhys/Library/Caches/go-build"
GOENV="/Users/rhys/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/rhys/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/local/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/49/zmds5zsn75z1283vtzxyfr5hj7yjq4/T/go-build125238824=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

$ go doc errgroup.WithContext
$ go mod init example.com/repro
$ go doc errgroup.WithContext

$ go1.13 doc errgroup.WithContext
$ go1.12 doc errgroup.WithContext

$ GO111MODULE=on go1.12 doc errgroup.WithContext
$ GO111MODULE=off go doc errgroup.WithContext

$ cat go.mod
$ go doc golang.org/x/sync/errgroup.WithContext
$ cat go.mod
$ go doc errgroup.WithContext

What did you expect to see?

I expected looking up docs from the command line for packages I've downloaded to my laptop to be really convenient. (I have many packages with docs of interest stored in my GOPATH.) I expected my ability to read those docs to not be strongly affected by my shell's current working directory. I expected that looking up docs for a package using its fully-qualified name would not affect the dependency manifest file in my shell's current working directory.

Put another way, I expected go doc to be a way of reading docs that's more convenient than godoc.org (particularly when I need to paste the result into a chat window, maybe after piping to grep to point out a part of interest).

What did you see instead?

When my shell's working directory is in GOPATH outside of a module (no go.mod file nearby), I can use go doc to look up documentation for any package in my GOPATH.

When my shell's working directory is in GOPATH but inside a module, go doc only works for packages in modules that are referenced by the current module.

$ go version
go version devel +0212f0410f Thu Aug 15 17:49:11 2019 +0000 darwin/amd64
$ go1.13 version
go version devel +60f14fddfe Wed Jun 26 16:35:14 2019 +0000 darwin/amd64
$ go1.12 version
go version go1.12.9 darwin/amd64
$ go env GOPATH
/Users/rhys/go
$ mkdir $(go env GOPATH)/src/example.com{,/repro}
$ cd $(go env GOPATH)/src/example.com/repro

$ go doc errgroup.WithContext
package errgroup // import "golang.org/x/sync/errgroup"

func WithContext(ctx context.Context) (*Group, context.Context)
    WithContext returns a new Group and an associated Context derived from ctx.

    The derived Context is canceled the first time a function passed to Go
    returns a non-nil error or the first time Wait returns, whichever occurs
    first.

$ go mod init example.com/repro
go: creating new go.mod: module example.com/repro
$ go doc errgroup.WithContext
doc: no buildable Go source files in /Users/rhys/go/src/example.com/repro
exit status 1
$ go1.13 doc errgroup.WithContext
doc: no buildable Go source files in /Users/rhys/go/src/example.com/repro
exit status 1

The working directory is "in" a module now, and go1.13beta1 and tip can't load docs for the errgroup package. But go1.12.9 can load the docs just fine (below).

$ time go1.12 doc errgroup.WithContext
package errgroup // import "golang.org/x/sync/errgroup"

func WithContext(ctx context.Context) (*Group, context.Context)
    WithContext returns a new Group and an associated Context derived from ctx.

    The derived Context is canceled the first time a function passed to Go
    returns a non-nil error or the first time Wait returns, whichever occurs
    first.


real	0m0.257s
user	0m0.085s
sys	0m0.203s
$ GO111MODULE=on go1.12 doc errgroup.WithContext
doc: no buildable Go source files in /Users/rhys/go/src/example.com/repro
exit status 1
$ GO111MODULE=off go doc errgroup.WithContext
package errgroup // import "golang.org/x/sync/errgroup"

func WithContext(ctx context.Context) (*Group, context.Context)
    WithContext returns a new Group and an associated Context derived from ctx.

    The derived Context is canceled the first time a function passed to Go
    returns a non-nil error or the first time Wait returns, whichever occurs
    first.

$ cat go.mod
module example.com/repro

go 1.13
$ time go doc golang.org/x/sync/errgroup.WithContext
package errgroup // import "golang.org/x/sync/errgroup"

func WithContext(ctx context.Context) (*Group, context.Context)
    WithContext returns a new Group and an associated Context derived from ctx.

    The derived Context is canceled the first time a function passed to Go
    returns a non-nil error or the first time Wait returns, whichever occurs
    first.


real	0m2.896s
user	0m0.284s
sys	0m0.281s
$ cat go.mod
module example.com/repro

go 1.13

require golang.org/x/sync v0.0.0-20190423024810-112230192c58 // indirect
$ time go doc errgroup.WithContext
package errgroup // import "golang.org/x/sync/errgroup"

func WithContext(ctx context.Context) (*Group, context.Context)
    WithContext returns a new Group and an associated Context derived from ctx.

    The derived Context is canceled the first time a function passed to Go
    returns a non-nil error or the first time Wait returns, whichever occurs
    first.


real	0m0.130s
user	0m0.055s
sys	0m0.079s

[bcmills]

When my shell's working directory is in a module, "go doc" can only find packages that are in modules referenced by the current module. If I use a package's fully-qualified name to look up its docs, I end up with unintended changes to the nearby go.mod file.

That seems like the correct behavior most of the time. In particular:

1. The documentation for a given package often changes from one release to the next. To show accurate documentation for the version in use, go doc should consult the dependencies of the main module.

2. Once go doc shows you the documentation for some package, go build and go test should use a version consistent with that documentation.

3. Resolving an unrecognized package path to its containing module can be a fairly expensive operation: first we have to identify the set of possible module paths and the latest version of each of those paths, then we have to actually download the source code to check which module contains the requested package. Having done that work already, we should not then throw that information away (see cmd/go: rethink "missing go.mod" mode #32027).

CC @jayconrod

[bcmills]

I wonder if there are other cases where a fallback go.mod file would be helpful—here I'd use it for docs, but it could also set a default version in other cases where Go tools have to guess or decide to use "today's latest".)

That's the general theme of #32027. Use cases I'm aware of include:

• Resolution of new package names in goimports. (cmd/go: add a facility to peek into the module cache #26717, CC @heschik, @matloob, @ianthehat)
  • goimports should prefer packages already in the dependencies of the current module, but if it can't find such a package, it should arguably add new a dependency if it can identify an “obvious” best candidate.
• go run of one-off source files (cmd/go: rethink "missing go.mod" mode #32027).
• go get of a tool should work outside of a module (cmd/go: offer a consistent "global install" command #30515), but it's not obvious to me that it needs to avoid the overhead of the usual module resolution.

[rhysh]

For (1), in the case that the main module requires the package whose docs I'm trying to read, then I agree it should control which version of those docs get displayed.

For (2), I see go build and go test as commands that have side-effects as some of their core features. From that perspective, it seems odd for go doc and go list to be anything but read-only.

Using the module cache for (3) seems ok.

Maybe go doc isn't the right command for me to use for casual doc-reading. It used to be (up through Go 1.12) and it was great. This might be a necessary part of the cost of modules—the rules mostly make sense individually—but I'd like to flag it as a specific unfortunate implication.

[bcmills]

This is gated on figuring out the design for “global” operations throughout the go command (#30515). Once we have a coherent overall design, we should apply that to the doc command.

[rhcarvalho]

This is gated on figuring out the design for “global” operations throughout the go command (#30515). Once we have a coherent overall design, we should apply that to the doc command.

Considering the changes that are planned for Go 1.16, is there already some obvious way forward for go doc?

[bcmills]

We've been adding version support to commands like go install (in 1.16) and go run (in 1.17). Perhaps we could do the same for go doc: maybe outside of a module you would run golang.org/x/sync/errgroup.WithContext@latest to get the documentation for the WithContext function in the latest version of errgroup?

But that would be very slow for @latest in particular, because we'd be constantly checking which version is the latest, so that seems like a non-starter.

[bcmills]

#43646 is related, in the sense of wanting to scope a go command to only what is already downloaded.

[mvdan]

But that would be very slow for @latest in particular, because we'd be constantly checking which version is the latest, so that seems like a non-starter.

I'm thinking aloud here, so this might be a terrible idea, but... We can live with pkg.go.dev potentially lagging behind the "true" latest branch by up to thirty minutes, given its caching and version discovery logic. Could we not make go doc have a similar default behavior? For example, it would only check for a newer latest version at most once every half an hour, so a dozen consecutive go doc invocations on the same unversioned module would only do one network roundtrip.

I definitely think that behavior would be wrong for commands which are designed around fetching the latest version, like go get, or for cases where one would explicitly ask for @latest. I guess what I mean here is like a @cached-latest default.