cmd/go: accept main packages as dependencies in go.mod files

[marwan-at-work]

If this issue is a duplicate (other than #25922 and #30515): apologies ahead of time, and please feel free to close it.

Summary

Many Go programs today not only depend on other packages, but they depend on other programs themselves. In other words, a module (whether a program or a library), may depend on other main programs for various reasons, such as code generation, peer connections, rpc connections and more.

When Go Modules first came out, the same exact question came up here: #25922.
The question was revolving around programs that depended on tools while this issue is talking more abstractly about programs that depend on any main program regardless of whether it's a tool or not.

The answer from @rsc at the time was that it was appropriate to use a tools.go file with // +build tools to force go.mod to record main package requirements.

The answer was appropriate for Go 1.11 when Modules was still highly experimental, but it might be worth reconsidering when Go Modules becomes the official dependency manager for all Go code out there.

The reason being, the tools.go is more of a workaround than first-class support. Having first-class support for main package dependencies would be more developer friendly than ignored-import-paths with an ignore build tag.

Furthermore, a module should be able to depend on other Go code regardless of whether it's package main or package <lib>.

It's also worth mentioning that other tools (such as https://github.com/myitcv/gobin) exist to make this workaround a little bit easier. But it's worth drawing a comparison of how other languages have an external binary for dependency management such as node/npm, ruby/bundle, rust/cargo while Go only has go. It would be odd to have a whole new program just to manage main dependencies.

Proposal

I propose that Go provides first class support for modules that depend on main packages

The proposal has two goals:

1. To have a user-friendly way to manage main package dependencies inside of a module.
2. Be able to execute a main program at that recorded version from 1.

How to achieve that is left out of the issue description, but can definitely be discussed in comments. I'll start with one suggestion just as a thought experiment.

Ultimately, what I would love to see is that if my coworker git-cloned my project that depended on other main packages such as github.com/golang/protobuf/protoc-gen-go, they would be able to get the precise version that I intended to use protoc-gen-go with.

Thanks

What version of Go are you using (go version)?

$ go version
go version go1.12 darwin/amd64

Does this issue reproduce with the latest release?

Yes

cc: @ianthehat

[marwan-at-work]

Suggestion

I suggest that the Go Modules system provides first class support for such programs by providing the following two functionalities:

1. Recording main-package requirements in the go.mod file, without having to resort to the build-ignored tools.go file.
2. Provide a way to execute a main program that was specified in the go.mod file through the Go command, so that it knows to execute the correct versioned binary.

The above two features would mean two changes to the Go toolchain:

1

Create a new block statment to separate main packages from non-main ones. Something like this:

module my.module

go 1.12

require (
  github.com/golang/protobuf v1.3.1
  github.com/pkg/errors v0.8.1
)

execute (
  github.com/golang/protobuf/protoc-gen-go v1.3.1
)

The reason for that is that when go mod tidy gets run, it needs to know not to remove any import paths that fall under the execute clause since they would never be present in any .go files within the module.

Note that the module root for protoc-gen-go is inside the require clause. Although that's just an implementation detail, I think it'd be nice if the execute block actually showed the full path to the main package and not just the module root, so that the programmer knows exactly which main programs are used by this go.mod file.

2

If each go.mod file has its own version of a binary, then that must mean the Go workspace ($GOPATH/pkg or $GOPATH/bin) must be able to host multiple versions of the same binary.

More importantly, we should be able to have an explicit way to run a versioned binary such as:

go exec github.com/golang/protobuf/protoc-gen-go # or `go exec protoc-gen-go` if possible. 

A quick hack would be to just have go exec run go run on $GOPATH/pkg/<path-to-versioned-main> under the hood, but that's another implementation detail.

However, sometimes we cannot execute a binary directly. For example, protoc-gen-go gets called from another binary which is protoc and therefore we can either leave that up to the user to figure out or we can have Go be able to copy versioned binaries to the main GOBIN folder so that it becomes the default binary in PATH.

Perhaps a go install <path> from within a go.mod file should do the trick. In other words, go install github.com/golang/protobuf/protoc-gen-go would look at the current directory's go.mod file and installs the spcified version there.

This changes the behavior of go install from always looking up the latest version to looking at the go.mod file and installing the specifide version instead.

On the other hand, we should be able to distinguish a regular go-get, and go-install that is independent of the current directory's go.mod file. Maybe a user just wanted to install a binary that had nothing to do with the project they were in the CWD of. Therefore, we could include a new -g flag so you can express a global install as such: go install -g github.com/golang/protobuf/protoc-gen-go which would instruct the Go command to not look/change the go.mod file. This is discussed a lot more in depth at #30515

One other solution for running a versioned binary is by simply allowing the -o flag in the go build command to be specified by the go install command.

What this means is that if we run go install -o=protoc_gen_go_v1.2.3 github.com/golang/protobuf/protoc-gen-go, we can then run the protoc compiler like this:

protoc --go_v1.2.3_out=. *.proto

The -o option in go install could mean that there's no reason for go exec to exist.

[bcmills]

CC @myitcv @thepudds @jayconrod @rogpeppe

[bcmills]

Ultimately, what I would love to see is that if my coworker git-cloned my project that depended on other main packages such as github.com/golang/protobuf/protoc-gen-go, they would be able to get the precise version that I intended to use protoc-gen-go with.

Could you give some more detail on that? Generally we expect the output of protoc-gen-go to itself be checked it to the source code, and if it's the input to a go generate stage the //go:generate comment can list the intended version explicitly (no need for an entry in the go.mod file).

Similarly, putting the dependencies of tools in the require block imposes minimum versions from the (transitive) dependency graph of those tools, which for some tools could be a lot more restrictive than what users actually need in order to go build the module.

[marwan-at-work]

@bcmills here's an example: I start a new gRPC project and use protoc-gen-go v1.3.0 to generate my Go files (assume a simple service.proto file with one RPC)

I make a gen.go file for convenience:

// please use v1.3.0! 
//go:generate protoc --go_out=. service.proto

Now, I give this project to my coworker, and my coworker git clones the project and now he/she wants to add a new RPC. They update the service.proto file to add a second RPC and run go generate to regenerate all the Go files.

In this process, there's no way to enforce what version of protoc-gen-go is being used by my coworker...I can put it in the comments as above, or I can put it in tools.go, but I believe those are not the friendliest solutions IMHO.

What I'd love to see is that a Module can depend on a main package the same way it depends on a nonmain package. It's all part of the development process.

How to enforce those "versioned binaries", is really up to everyone...but I'd love for a friendly and official way to do it because it's quite a common use case. My suggestion above is one of a few that I can think of.

Similarly, putting the dependencies of tools in the require block imposes minimum versions from the (transitive) dependency graph of those tools, which for some tools could be a lot more restrictive than what users actually need in order to go build the module.

That's a good point, but that's also the case with importing many nonmain libraries, there could always be a conflicting transitive dependency. However, if that's a concern then it's definitely valid to separate "main"'s transitive deps from the nonmain ones. Emphasis on "main" which is not necessarily a tool. It's just an executable dependency of a Module.

[myitcv]

@marwan-at-work and I were exchanging some comments offline. This has come up on the golang-tools calls many times before, and @ianthehat has strong feelings on the direction in which all of this should head. My comments stem mainly from learnings from gobin.

But first:

This changes the behavior of go install from always looking up the latest version to looking at the go.mod file and installing the specifide version instead.

go install resolves dependencies via the main module, i.e. it does not always install the latest version. go get has different semantics, but go install is stable in this respect.

One other solution for running a versioned binary is by simply allowing the -o flag in the go build command to be specified by the go install command.

I think you're after the GOBIN environment variable here; see go help environment.

---

This topic has come up before in many different guises (incomplete list):

• cmd/go: allow "go get" when outside a module in module mode #24250 - cmd/go: allow "go get" when outside a module in module mode
• cmd/go: go run pkg is significantly slower than running built binary #25416 - cmd/go: go run pkg is significantly slower than running built binary
• cmd/go: clarify best practice for tool dependencies #25922 - cmd/go: clarify best practice for tool dependencies
• cmd/go: go get should not add a dependency to go.mod #27643 - cmd/go: go get should not add a dependency to go.mod
• cmd/go: support module-local install/run of tool dependencies #27653 - cmd/go: support module-local install/run of tool dependencies
• cmd/go: offer a consistent "global install" command #30515 - cmd/go: offer a consistent "global install" command

as well as being a regular-ish feature on the golang-tools calls/Slack.

@ianthehat's comments in #30515 (comment) help us I think to step back one step further and consider the higher level goals from the end-user's perspective:

The concrete use cases people have suggested so far are:

• go code generators (often from go generate lines) which must be versioned at possibly per invocation granularity
• code analysis tools (the kinds of things we are trying to merge into gopls, an also gopls itself of course) which may need to be versioned per project
• System wide installs of binaries, for the kinds of instructions people put on their wiki

noting @josharian's comment that follows too, #30515 (comment)

So a key point is that the problem space also includes the "global"/system-wide tools. There need not be a single solution, rather just clarifying the wider scope.

My position (in the loosest sense) is heavily influenced by learnings from gobin. gobin essentially builds on top of and combines the behaviour/semantics of go get, go build and go install.

With those high level goals in mind, is a rough list of more detailed requirements from my perspective. (Note "tool" is just the generic term I use for main package; could equally have used command, binary... not looking to be opinionated on the term.)

• need the ability to "install" a tool globally
  • to a specified directory (a la explicitly set GOBIN)
  • to a discoverable directory that allows for multiple versions of the same tool (with possibly varying build constraints, see below) to co-exist on disk
• need the ability to "install" a tool locally, where "local" means "with respect to the main module"
  • to a specified directory (a la explicitly set GOBIN)
  • to a discoverable directory, which needs to be specific/local to the main module (so that two main modules do not share the same target directory), but also unique to build constraints
• need the ability to globally/locally run a main package, a la go run
  • i.e. no requirement on installing/setting PATH
  • we also want to avoid the link cost of go run
  • would (typically) be used in //go:generate directives within the main module, or similar
• need the ability to globally/locally specify build tags for install/run
• need the ability to globally/locally cross-compile
• for global install/run, should not need to specify version with package path; i.e. "latest" should have meaning within the local module cache
• for local install/run, specifying version is in effect an error, because all versions should be resolved via go.mod (or similar)
• for both global/local operation, it should be possible to specify replace-ments (support for replace directives and dependency overrides myitcv/gobin#50)

The edit to refer to "discoverable" directories above is critical to allowing multiple versions of a tool to be installed on disk in the global setting, and to allow for variations in GOOS/GOARCH and build constraints for both global/local installs.

Implementation details

@rsc has, in the past, indicated that cmd/go should not be overloaded with every command/sub command/option under the sun. I have to say, I've very much come around to that point of view, but like @marwan-at-work I see the tension between that position and the position of otherwise needing a separate program as part of the Go distribution: users/docs now need to reference go and (for example) gobin.

For local tools, there is the question on whether to use the main module's go.mod or not. The tools.go solution for recording main package dependencies means they get included along with, and can be influenced by/influence, all other dependencies.

• the same point/question exists for (transitive) test dependencies
• it's related to the point of whether replace directives for a tool should be honoured or not (To apply replace/exclude directives or not to apply replace/exclude directives? myitcv/gobin#51)

Alternative solutions to recording requirements in tools.go include (but are not limited to):

• @marwan-at-work's suggestion of a separate section in the main module's go.mod
• a separate file altogether, e.g. go.tools
• sub module(s) underneath the main module
• ... likely others...

The only point being this is an implementation detail to my mind, albeit it a significant/contentious one

Note, with the go.tools and submodule options, there is the scope for each tool effectively being isolated from other tools; i.e. in effect each has its own go.mod. This also raises the possibility of being able to honour replace directives in a tool's respective go.mod if required/desired

Also to state the perhaps obvious point that the implementation detail behind the recording of main dependencies is orthogonal to the cmd/go command/subcommand, or separate tool that improves the UX of working with tools, globally or locally.

Next steps

Whilst, selfishly, I'm happily using gobin to meet all the needs above (it's not perfect, but good enough), I agree with @marwan-at-work, @mvdan and others that an official solution/official solutions would be very much welcomed. Whether that discussion should continue here, in #30515, or elsewhere, I'm not fussed.

On a recent golang-tools call, @ianthehat made noises that gobin could be used as a the basis for a single solution, separate from cmd/go.

@ianthehat - is there any update on this point, or the discussion more generally?

[seankhliao]

Is there anything else in this proposal now that #48429 (cmd/go: track tool dependencies in go.mod) is accepted?