cmd/go: flags to control changes to go.mod, go.sum

[jayconrod]

Proposed changes

I propose we add three new flags to go subcommands that deal with modules.

• -modfile=go.mod - instead of reading and writing go.mod from the current directory or a parent directory, the go command would read and write the specified file. The original go.mod would still determine the module root directory, so this file could be in any directory (perhaps /tmp). It would be an error to use this flag when no actual go.mod file is present.
• -sumfile=go.sum - instead of reading and writing go.sum from the directory containing go.mod, the go command would read and write the specified file. It would be an error to use this flag when no original go.mod file is present.
• -g - "global mode" - the go command would behave as if no go.mod file were present.

These flags would be allowed when GO111MODULE is set to on or auto (the current default) and rejected when GO111MODULE is set to off. -modfile and -sumfile both require an actual go.mod to be present, so modules must be enabled when they're used. -g does not require an actual go.mod to be present, and in auto mode, it implies that modules are enabled.

Background

The go command updates go.mod and go.sum after any command that needs to find a module for a package not provided by any module currently in the build list. This ensures reproducibility: if you run the same command twice, it should build (or list or test) the same packages at the same versions, even if new versions have been published since the first invocation.

For example, if you run go build ., and a .go file in the current directory imports example.com/m/pkg which is not provided by any known module, the go command will add a requirement on the latest version of the module example.com/m to go.mod. Future runs of go build . will produce the same result.

While these updates are usually helpful, there are many situations where they're not desirable.

Selected issues

gopls (various issues)

gopls loads information about source files in a workspace using golang.org/x/tools/go/packages, which invokes go list. gopls may also run go list directly. In either case, gopls may trigger changes to go.mod and go.sum. This may be caused by user actions that seem unrelated to building anything, for example, opening a file. go.mod appears to change mysteriously on its own, and users don't realize gopls is triggering it.

It's not usually important that the information gopls loads is reproducible; files it operates on are frequently changing. However, it is important that when it resolves an unknown import path to a module, it doesn't need to do so repeatedly since this can add a lot of latency, especially on slow connections.

gopls could set -modfile and -sumfile to temporary copies of the original go.mod and go.sum. The original go.mod and go.sum would not be modified (until the user explicitly runs a command like go build). Resolved module requirements would stay in the temporary files so they would not need to be resolved again.

#25922 - clarify best practice for tool dependencies

Developers need a way to express module requirements that aren't implied by package imports. This is especially useful for tools invoked by go generate. Authors can add tool requirements to go.mod manually or with go get, but these requirements are erased by go mod tidy.

The current recommendation is to create a tools.go file, tag it with // +build tools, then import main packages of needed tools. tools.go will never be built because of the tag, but go mod tidy will read the imports and preserve the requirements. This feels like a hacky workaround rather than a best practice. It also pushes requirements which may not otherwise be needed on downstream modules.

A better solution would be to keep a separate go.tools.mod file with tool requirements, then point to that with -modfile=go.tools.mod when running commands that require tools.

#26640 - allow go.mod.local to contain replace/exclude lines

This is a feature request to keep some go.mod statements out of source control. It's frequently useful for module authors to check out dependencies and point to them with replace statements for local development and debugging. These statements shouldn't necessarily be exposed to users or other developers on the same project though.

Setting -modfile=go.local.mod and -sumfile=go.local.sum would solve this problem, at least partially. The two files could be copied from the regular go.mod and go.sum files and added to .gitignore. Note however, that these local files are used instead of the regular files, not in addition to, so some synchronization might be required.

#30515 - offer a consistent "global install" command

Developers want to be able to install tools from any directory, regardless of the requirements of the current module. go get tool@version may update the current go.mod, so users need to change to a temporary directory without a go.mod file to run commands like this. Tool authors need to be careful when writing installation instructions because of this.

The -g flag would address this issue. It would tell the go command to run as if it were outside any module. Tool authors could write go get -g tool@latest in their installation instructions: this would install the latest version of the tool, regardless of the current directory.

Note that "missing go.mod" is being reconsidered (#32027), so the actual semantics of -g may change: this issue is just about ignoring the current module.

#33710 - module mode removes concept of global docs

In module mode, go doc example.com/pkg prints documentation for the packages named on the command line at the same version they would be built with. Like go build, go doc may add or update requirements in go.mod. This may be undesirable, especially if you're using the documentation to decide whether you want to depend on a package that is not currently imported.

The -g flag would partially solve this. The current module would be ignored, and "global" documentation would be shown.

Note that go doc does not currently work in "missing go.mod" for packages outside std. #33710 would need to be fixed, but -g would provide a useful way to access that mode.

Other related issues

There are a large number of open issues about unexpected and unwanted go.mod changes. The flags suggested here won't solve all these problems, but they provide useful context.

• cmd/go: do not resolve missing imports in 'go mod why' #26977 - go mod why adds a go.mod line
• cmd/go: go list has too many (more than zero) side effects #29452 - go list has too many (more than zero) side effects
• cmd/go: 'go list' should not resolve or record modules that are not relevant to the requested output fields #29869 - 'go list' should not resolve or record modules that are not relevant to the requested output fields
• cmd/go: 'mod verify' should not modify the go.mod file #31372 - 'mod verify' does not respect -mod=readonly
• x/tools/gopls: support go.mod files #31999 - x/tools/gopls: support go.mod files
• cmd/go: rethink "missing go.mod" mode #32027 - rethink "missing go.mod" mode
• cmd/go: go install: don’t fail when go.mod can’t be updated on a read-only system #33326 - go install: don’t fail when go.mod can’t be updated on a read-only system
• cmd/go: 'go mod download' should not modify the go.mod file #34450 - go mod download support -readonly

[jayconrod]

cc @bcmills @rsc @stamblerre @ianthehat

[bcmills]

-g - "global mode" - the go command would behave as if no go.mod file were present.

Per #32027 (comment), we'll probably want to make the go command less willing to resolve dependencies when no go.mod file is present.

If we do that, presumably the -g flag will allow the go command to resolve the transitive dependencies of the modules containing the packages listed on the command line. Should -g also allow the go command to resolve missing dependencies found in the import statements of those packages?

[bcmills]

The original go.mod would still determine the module root directory,

but

These flags would be accepted when GO111MODULE is set to on and rejected when GO111MODULE is set to off.

If GO111MODULE is on but there is no go.mod file above the current working directory, what would we use as the module root?

[jayconrod]

Per #32027 (comment), we'll probably want to make the go command less willing to resolve dependencies when no go.mod file is present.

If we do that, presumably the -g flag will allow the go command to resolve the transitive dependencies of the modules containing the packages listed on the command line. Should -g also allow the go command to resolve missing dependencies found in the import statements of those packages?

I think -g should just make the go command do whatever it would do without a go.mod file. So if go run x.go resolves transitively imports and succeeds, go run -g x.go would do the same within a module. But if we change go run x.go to fail, then go run -g x.go would also fail.

[jayconrod]

If GO111MODULE is on but there is no go.mod file above the current working directory, what would we use as the module root?

I contradicted myself a bit here. Updated the text. -modfile and -sumfile require an actual go.mod to be present in order to set the module root directory.

[heschi]

Regarding the file flags: I think having to specify both -modfile and -sumfile is cumbersome. I can't think of any compelling reason to want to share go.sum with any other module, since go mod tidy will throw away the unnecessary lines anyway. So I would suggest that at a minimum, -sumfile be derived from -modfile if it's not set. A more extreme option would be to specify a single prefix, say -modprefix, and add .mod and .sum to it as needed. That may be too strange, though.

This is also a good precedent to set in case there's ever a third file, since nobody will know to override its location before it exists.

[myitcv]

@jayconrod regarding the "gopls (various issues)" motivation. I'm not entirely clear on what the issue here is and hence why -g/-modfile/other is a solution.

Is the problem that people are not used to cmd/go having side effects in module mode?

Because for me, I would expect that anything I do inside my editor, e.g. adding an import for a package whose module is not in my go.mod, could have side effects on the main module.

Could you give a bit more background?

[mvdan]

#30515 - offer a consistent "global install" command

While not part of the original issue, the consensus for a while seemed to be that a "global" install should also obey replace directives. At least, this is what @ianthehat said in #30515 (comment). Has the team's position generally changed to not obeying replace directives at all?

I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go <args>.

[myitcv]

Possibly worth reiterating that replace directives come in two forms:

• directory-based
• non-directory-based

I think the consensus that @mvdan refers to above was on the latter being applied.

See @mvdan's comment following this one.

[mvdan]

Sorry, I should have clarified. There were initially some comments about applying non-directory replace directives, but the consensus seemed that we either apply all of them or none of them - to not add a third build mode. @ianthehat's comment that I linked seemed to lean towards applying all replace directives.

[rogpeppe]

Personally, I'm in favour of applying directory replacements only when the source is in a user-controlled directory (as opposed to in global mode where the source is in the module cache). But I feel strongly that installs in global mode should respect other replacements, FWIW.

[bcmills]

@mvdan, @rogpeppe: see #30515 (comment) and #30515 (comment).

Furthermore, given that the proposed semantics of the -g flag are to do whatever we would do if outside of a module, the question of whether or how to apply replace directives seems orthogonal (and a bit off-topic). #31173 is probably a more appropriate venue for that discussion.

[bcmills]

It would be an error to use this flag when no actual go.mod file is present.

We should probably make that even stronger: the module directive in the replacement go.mod file should specify the same module path as the actual go.mod file. (Otherwise, if we're building packages within the module, we will end up resolving what should be module-local imports by looking for an external module.)

[jayconrod]

regarding the "gopls (various issues)" motivation. I'm not entirely clear on what the issue here is and hence why -g/-modfile/other is a solution.

Is the problem that people are not used to cmd/go having side effects in module mode?

Because for me, I would expect that anything I do inside my editor, e.g. adding an import for a package whose module is not in my go.mod, could have side effects on the main module.

@myitcv, @ianthehat and @stamblerre have told me there's no canonical issue for this, but they've had to close a lot of issues as "working as intended", pointing to #29452, and they explain this frequently on Slack.

They mentioned one egregious example where someone in a clean workspace switched branches, then tried to switch back but were unable to because go.mod had uncommitted changes. Their editor (and gopls) was open in the background. It had detected changes in open files, run go list indirectly, and modified go.mod as a result.

-modfile would have made these changes to a temporary go.mod file instead of the go.mod in the main repo. #31999 is about supporting go.mod files in gopls, and while there aren't any details there yet, part of the plan is to provide easy ways to make changes and upgrades. It would be difficult to do that without -modfile.

[jayconrod]

Regarding the file flags: I think having to specify both -modfile and -sumfile is cumbersome. I can't think of any compelling reason to want to share go.sum with any other module, since go mod tidy will throw away the unnecessary lines anyway. So I would suggest that at a minimum, -sumfile be derived from -modfile if it's not set. A more extreme option would be to specify a single prefix, say -modprefix, and add .mod and .sum to it as needed. That may be too strange, though.

This is also a good precedent to set in case there's ever a third file, since nobody will know to override its location before it exists.

I like the simplicity of just saying what both files should be, but I couldn't actually come up with a scenario where you'd want to set -modfile without -sumfile.

How about this: there's no -sumfile flag. If -modfile is set to M, then the sum file is strings.TrimPrefix(M, ".mod") + ".sum".

[jayconrod]

I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go .

I don't think that solves the same set of problems. -modfile would still use the location of actual go.mod file to set the module root directory, not its argument. -modfile just redirects reads and writes, providing a way to control changes.

[ianthehat]

While not part of the original issue, the consensus for a while seemed to be that a "global" install should also obey replace directives. At least, this is what @ianthehat said in #30515 (comment). Has the team's position generally changed to not obeying replace directives at all?

After thinking through all the weird edge cases and trying out a bunch of things, I came to the conclusion that applying replace directives is a confusing and dangerous ball of scary, and the only sane thing to do is not to apply them, in fact the only sane thing to do is to never ever check replace directives in to your repository in the first place.
Part of the benefit of this proposal is it allows for a workflow that makes that a much easier goal, by allowing an alternate go.mod that has the replace directives in it but is not used by default.

I'm also a bit confused by the multiple flags, like @heschik. How about a -modroot=path flag? It would roughly be equivalent to cd path && go <args>.

That would not allow most of the useful patterns (having multiple .mod files in the same directory that you can choose between, or a cache directory with modified .mod files for a bunch of different modules etc)

It would also mean that you have to specify what happens in a bunch of interesting edge cases (thinks like relative paths, are they from the original module or the alternate root?)

I think that specifying the .sum file would/should be incredibly rare, leaving it as the original one would be fine for the majority of use cases.

[jayconrod]

@mvdan @rogpeppe

On replacements:

1. We should apply all of them or none of them. A third mode would cause confusion. (cmd/go: offer a consistent "global install" command #30515 (comment)).
2. In order to apply file path replacements, we'd need to check out a whole repo. That's very different from what go get does now (especially when a proxy is in use), and there are many ways it could fail. We also won't get reproducible builds if replacement paths point outside the repository.

Personally, I think the downsides of (2) outweigh the benefits, and it's better to have something very simple like -g.

[bcmills]

How about this: there's no -sumfile flag. If -modfile is set to M, then the sum file is strings.TrimPrefix(M, ".mod") + ".sum".

Maybe split the difference? If the original is go.mod and -modfile is M.mod, first check whether M.sum exists: if so, use (and update) it, and otherwise use (and update) the original go.sum.

As far as I can tell, that handles both the tools.go.mod case and the /tmp/some-gopath-dir/go.{mod,sum} case.

[jayconrod]

Maybe split the difference? If the original is go.mod and -modfile is M.mod, first check whether M.sum exists: if so, use (and update) it, and otherwise use (and update) the original go.sum.

As far as I can tell, that handles both the tools.go.mod case and the /tmp/some-gopath-dir/go.{mod,sum} case.

@bcmils Could work, but it seems a little subtle. If a module only depends on std and has no go.sum file, a naïve tool might copy go.mod to /tmp without creating an empty go.sum file. It would then unexpectedly modify the temp go.mod and the original go.sum.

[myitcv]

@jayconrod re #34506 (comment)

-modfile would have made these changes to a temporary go.mod file instead of the go.mod in the main repo

But what about the situations where you do want gopls to have these side effects? i.e. adding an import for a package whose module is not in my go.mod.

[bcmills]

I think -g should just make the go command do whatever it would do without a go.mod file.

Without a go.mod file, go get example.com/some/really/old/tool (that is, some tool without its own go.mod file) should probably fail, rather than re-resolving the latest transitive imports of that tool and discarding the result.

On the other hand, I think it is probably reasonable to expect go get -g example.com/some/really/old/tool to succeed, even if it is consistently slow.

[jayconrod]

But what about the situations where you do want gopls to have these side effects? i.e. adding an import for a package whose module is not in my go.mod.

@myitcv It would be up to gopls and the editor to provide a sensible way to do that. Perhaps it could show a warning that go.mod is incomplete and provide a quick fix to add the missing requirement. Or it could update go.mod when a file is saved if it adds new imports.

There are situations where modifying go.mod is not wanted, and -modfile provides gopls with a knob that controls when those modifications happen.

[jayconrod]

Without a go.mod file, go get example.com/some/really/old/tool (that is, some tool without its own go.mod file) should probably fail, rather than re-resolving the latest transitive imports of that tool and discarding the result.

On the other hand, I think it is probably reasonable to expect go get -g example.com/some/really/old/tool to succeed, even if it is consistently slow.

@bcmills There's a lot of nuance here. Assuming no go.mod is present, should go get example.com/tool when example.com/tool does have a go.mod file? What if the go.mod file is missing some requirements?

Should -g have this variation in behavior for go get only or for other commands, too?

[jayconrod]

Re-milestoning this for Go1.14 since -modfile is merged and seems likely to ship. I'll close this issue after we've validated -modfile in gopls.

I'd like to direct further discussion on -g or any other "global install" to #30515.

[gopherbot]

Change https://golang.org/cl/208236 mentions this issue: cmd/go: add 'go generate' commands to modfile_flag test

[ianlancetaylor]

@jayconrod What's the status of this issue? Thanks.

[jayconrod]

@ianlancetaylor -modfile is implemented. I'd like some of the gopls folks to evaluate it and post feedback after the beta ships. I expect to either close this issue or re-milestone for 1.15 before the RC.

[gopherbot]

Change https://golang.org/cl/211538 mentions this issue: internal/lsp: Use the -modfile flag to update a different go.mod file

[gopherbot]

Change https://golang.org/cl/212100 mentions this issue: cmd/go: in 'go list -m', print effective go.mod file

[jayconrod]

Closing this issue since 1.14rc1 is imminent. -modfile support was added to gopls in CL 211538, and it seems to be working as intended.

@stamblerre @ridersofrohan @ianthehat Let me know if you have any other feedback on this feature.

[rodrigc]

@jayconrod If I have a Makefile which does something like this:
https://github.com/libopenstorage/cloudops/blob/45bab4b444c4fe23c33eaca4146ce3b10dfd1a69/Makefile#L44

How should I change this Makefile when go 1.14 is released to use the -modfile flag?

Can I specify something like -modfile /dev/null?

[jayconrod]

@rodrigc It looks like this Makefile is installing then running several tools at the latest versions in GOPATH mode.

I think the module-mode equivalent of this (using -modfile) would be to create a tools.mod file requiring the versions of the tools you want to use. You could build the tools with mkdir -p tools && go build -modfile=tools.mod -o tools/ example.com/tool.... Then run using executables in that tools subdirectory.

That would get you predictable versions of tools without interfering with your main go.mod file. It also wouldn't have side effects on the rest of your system since you aren't installing anything.

[rajathagasthya]

@jayconrod Is this the recommended way to install tools now instead of the tools.go file? Will there be an update to https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module?

[winmillwill]

@rodrigc It looks like this Makefile is installing then running several tools at the latest versions in GOPATH mode.

I think the module-mode equivalent of this (using -modfile) would be to create a tools.mod file requiring the versions of the tools you want to use. You could build the tools with mkdir -p tools && go build -modfile=tools.mod -o tools/ example.com/tool.... Then run using executables in that tools subdirectory.

That would get you predictable versions of tools without interfering with your main go.mod file. It also wouldn't have side effects on the rest of your system since you aren't installing anything.

This seems strange to me. I will maintain a file with the names and constraints of executables I need, which makes sense. But then, I want to just point the tool at that file when I use the command for "use this file to give me the things I enumerate". Instead, it seems like the recommendation is to pass each tool enumerated to a different go install invocation. I apologize if I'm misunderstanding what is meant by example.com/..., but I don't see how it can capture the need of downloading tools from several different repositories. So it seems like I only get value out of maintaining this file if I also maintain a script that parses out each dependency and invokes the tool with a different command than I would normally use.

[suntong]

@jayconrod Could you summarize how to use it pls? Is the following still hold 100%?

• Copy go.mod to go.local.mod. Add go.local.mod to .gitignore (or equivalent for your workspace).
• Run go env -w GOFLAGS=-modfile=go.local.mod. This tells the go command to use that file by default.
• Any any replace and exclude directives or other local edits.
• Before submitting and in CI, make sure to test without the local file: go env -u GOFLAGS or just -modfile=. Probably also go mod tidy.

And what is the minimum requirement to use it pls.

[jayconrod]

@suntong That's all still correct. -modfile was added in Go 1.14.

Right now, -modfile is mostly used in tools and editors (gopls), not so much directly by users on the command line. We're in the process of putting together a proposal that will make it easier to work on multiple modules at the same time. Stay tuned.