cmd/go: 'go install' should install executables in module mode outside a module

[jayconrod]

Authors: Jay Conrod, Daniel Martí

Last Updated: 2020-09-29

Design doc: CL 243077
Comments on the CL are preferred over comments on this issue.

Abstract

Authors of executables need a simple, reliable, consistent way for users to
build and install exectuables in module mode without updating module
requirements in the current module's go.mod file.

Background

go get is used to download and install executables, but it's also responsible
for managing dependencies in go.mod files. This causes confusion and
unintended side effects: for example, the command
go get golang.org/x/tools/gopls builds and installs gopls. If there's a
go.mod file in the current directory or any parent, this command also adds a
requirement on the module golang.org/x/tools/gopls, which is usually not
intended. When GO111MODULE is not set, go get will also run in GOPATH mode
when invoked outside a module.

These problems lead authors to write complex installation commands such as:

(cd $(mktemp -d); GO111MODULE=on go get golang.org/x/tools/gopls)

Proposal

We propose augmenting the go install command to build and install packages
at specific versions, regardless of the current module context.

go install golang.org/x/tools/gopls@v0.4.4

To eliminate redundancy and confusion, we also propose deprecating and removing
go get functionality for building and installing packages.

Details

The new go install behavior will be enabled when an argument has a version
suffix like @latest or @v1.5.2. Currently, go install does not allow
version suffixes. When a version suffix is used:

• go install runs in module mode, regardless of whether a go.mod file is
  present. If GO111MODULE=off, go install reports an error, similar to
  what go mod download and other module commands do.
• go install acts as if no go.mod file is present in the current directory
  or parent directory.
• No module will be considered the "main" module.
• Errors are reported in some cases to ensure that consistent versions of
  dependencies are used by users and module authors. See Rationale below.
  • Command line arguments must not be meta-patterns (all, std, cmd)
    or local directories (./foo, /tmp/bar).
  • Command line arguments must refer to main packages (executables). If a
    argument has a wildcard (...), it will only match main packages.
  • Command line arguments must refer to packages in one module at a specific
    version. All version suffixes must be identical. The versions of the
    installed packages' dependencies are determined by that module's go.mod
    file (if it has one).
  • If that module has a go.mod file, it must not contain directives that
    would cause it to be interpreted differently if the module were the main
    module. In particular, it must not contain replace or exclude
    directives.

If go install has arguments without version suffixes, its behavior will not
change. It will operate in the context of the main module. If run in module mode
outside of a module, go install will report an error.

With these restrictions, users can install executables using consistent commands.
Authors can provide simple installation instructions without worrying about
the user's working directory.

With this change, go install would overlap with go get even more, so we also
propose deprecating and removing the ability for go get to install packages.

• In Go 1.16, when go get is invoked outside a module or when go get is
  invoked without the -d flag with arguments matching one or more main
  packages, go get would print a deprecation warning recommending an
  equivalent go install command.
• In a later release (likely Go 1.17), go get would no longer build or install
  packages. The -d flag would be enabled by default. Setting -d=false would
  be an error. If go get is invoked outside a module, it would print an error
  recommending an equivalent go install command.

Examples

# Install a single executable at the latest version
$ go install example.com/cmd/tool@latest

# Install multiple executables at the latest version
$ go install example.com/cmd/...@latest

# Install at a specific version
$ go install example.com/cmd/tool@v1.4.2

Current go install and go get functionality

go install is used for building and installing packages within the context of
the main module. go install reports an error when invoked outside of a module
or when given arguments with version queries like @latest.

go get is used both for updating module dependencies in go.mod and for
building and installing executables. go get also works differently depending
on whether it's invoked inside or outside of a module.

These overlapping responsibilities lead to confusion. Ideally, we would have one
command (go install) for installing executables and one command (go get) for
changing dependencies.

Currently, when go get is invoked outside a module in module mode (with
GO111MODULE=on), its primary purpose is to build and install executables. In
this configuration, there is no main module, even if only one module provides
packages named on the command line. The build list (the set of module versions
used in the build) is calculated from requirements in go.mod files of modules
providing packages named on the command line. replace or exclude directives
from all modules are ignored. Vendor directories are also ignored.

When go get is invoked inside a module, its primary purpose is to update
requirements in go.mod. The -d flag is often used, which instructs go get
not to build or install packages. Explicit go build or go install commands
are often better for installing tools when dependency versions are specified in
go.mod and no update is desired. Like other build commands, go get loads the
build list from the main module's go.mod file, applying any replace or
exclude directives it finds there. replace and exclude directives in other
modules' go.mod files are never applied. Vendor directories in the main module
and in other modules are ignored; the -mod=vendor flag is not allowed.

The motivation for the current go get behavior was to make usage in module
mode similar to usage in GOPATH mode. In GOPATH mode, go get would download
repositories for any missing packages into $GOPATH/src, then build and install
those packages into $GOPATH/bin or $GOPATH/pkg. go get -u would update
repositories to their latest versions. go get -d would download repositories
without building packages. In module mode, go get works with requirements in
go.mod instead of repositories in $GOPATH/src.

Rationale

Why can't go get clone a git repository and build from there?

In module mode, the go command typically fetches dependencies from a
proxy. Modules are distributed as zip files that contain sources for specific
module versions. Even when go connects directly to a repository instead of a
proxy, it still generates zip files so that builds work consistently no matter
how modules are fetched. Those zip files don't contain nested modules or vendor
directories.

If go get cloned repositories, it would work very differently from other build
commands. That causes several problems:

• It adds complication (and bugs!) to the go command to support a new build
  mode.
• It creates work for authors, who would need to ensure their programs can be
  built with both go get and go install.
• It reduces speed and reliability for users. Modules may be available on a
  proxy when the original repository is unavailable. Fetching modules from a
  proxy is roughly 5-7x faster than cloning git repositories.

Why can't vendor directories be used?

Vendor directories are not included in module zip files. Since they're not
present when a module is downloaded, there's no way to build with them.

We don't plan to include vendor directories in zip files in the future
either. Changing the set of files included in module zip files would break
go.sum hashes.

Why can't directory replace directives be used?

For example:

replace example.com/sibling => ../sibling

replace directives with a directory path on the right side can't be used
because the directory must be outside the module. These directories can't be
present when the module is downloaded, so there's no way to build with them.

Why can't module replace directives be used?

For example:

replace example.com/mod v1.0.0 => example.com/fork v1.0.1-bugfix

It is technically possible to apply these directives. If we did this, we would
still want some restrictions. First, an error would be reported if more than one
module provided packages named on the command line: we must be able to identify
a main module. Second, an error would be reported if any directory replace
directives were present: we don't want to introduce a new configuration where
some replace directives are applied but others are silently ignored.

However, there are two reasons to avoid applying replace directives at all.

First, applying replace directives would create inconsistency for users inside
and outside a module. When a package is built within a module with go build or
go install, only replace directives from the main module are applied, not
the module providing the package. When a package is built outside a module with
go get, no replace directives are applied. If go install applied replace
directives from the module providing the package, it would not be consistent
with the current behavior of any other build command. To eliminate confusion
about whether replace directives are applied, we propose that go install
reports errors when encountering them.

Second, if go install applied replace directives, it would take power away
from developers that depend on modules that provide tools. For example, suppose
the author of a popular code generation tool gogen forks a dependency
genutil to add a feature. They add a replace directive pointing to their
fork of genutil while waiting for a PR to merge. A user of gogen wants to
track the version they use in their go.mod file to ensure everyone on their
team uses a consistent version. Unfortunately, they can no longer build gogen
with go install because the replace is ignored. The author of gogen might
instruct their users to build with go install, but then users can't track the
dependency in their go.mod file, and they can't apply their own require and
replace directives to upgrade or fix other transitive dependencies. The author
of gogen could also instruct their users to copy the replace directive, but
this may conflict with other require and replace directives, and it may
cause similar problems for users further downstream.

Why report errors instead of ignoring replace?

If go install ignored replace directives, it would be consistent with the
current behavior of go get when invoked outside a module. However, in
#30515 and related discussions, we found that
many developers are surprised by that behavior.

It seems better to be explicit that replace directives are only applied
locally within a module during development and not when users build packages
from outside the module. We'd like to encourage module authors to release
versions of their modules that don't rely on replace directives so that users
in other modules may depend on them easily.

If this behavior turns out not to be suitable (for example, authors prefer to
keep replace directives in go.mod at release versions and understand that
they won't affect users), then we could start ignoring replace directives in
the future, matching current go get behavior.

Should go.sum files be checked?

Because there is no main module, go install will not use a go.sum file to
authenticate any downloaded module or go.mod file. The go command will still
use the checksum database (sum.golang.org) to
authenticate downloads, subject to privacy settings. This is consistent with the
current behavior of go get: when invoked outside a module, no go.sum file is
used.

The new go install command requires that only one module may provide packages
named on the command line, so it may be logical to use that module's go.sum
file to verify downloads. This avoids a problem in
#28802, a related proposal to verify downloads
against all go.sum files in dependencies: the build can't be broken by one bad
go.sum file in a dependency.

However, using the go.sum from the module named on the command line only
provides a marginal security benefit: it lets us authenticate private module
dependencies (those not available to the checksum database) when the module on
the command line is public. If the module named on the command line is private
or if the checksum database isn't used, then we can't authenticate the download
of its content (including the go.sum file), and we must trust the proxy. If
all dependencies are public, we can authenticate all downloads without go.sum.

Why require a version suffix when outside a module?

If no version suffix were required when go install is invoked outside a
module, then the meaning of the command would depend on whether the user's
working directory is inside a module. For example:

go install golang.org/x/tools/gopls

When invoked outside of a module, this command would run in GOPATH mode,
unless GO111MODULE=on is set. In module mode, it would install the latest
version of the executable.

When invoked inside a module, this command would use the main module's go.mod
file to determine the versions of the modules needed to build the package.

We currently have a similar problem with go get. Requiring the version suffix
makes the meaning of a go install command unambiguous.

Why not a -g flag instead of @latest?

To install the latest version of an executable, the two commands below would be
equivalent:

go install -g golang.org/x/tools/gopls
go install golang.org/x/tools/gopls@latest

The -g flag has the advantage of being shorter for a common use case. However,
it would only be useful when installing the latest version of a package, since
-g would be implied by any version suffix.

The @latest suffix is clearer, and it implies that the command is
time-dependent and not reproducible. We prefer it for those reasons.

Compatibility

The go install part of this proposal only applies to commands with version
suffixes on each argument. go install reports an error for these, and this
proposal does not recommend changing other functionality of go install, so
that part of the proposal is backward compatible.

The go get part of this proposal recommends deprecating and removing
functionality, so it's certainly not backward compatible. go get -d commands
will continue to work without modification though, and eventually, the -d flag
can be dropped.

Parts of this proposal are more strict than is technically necessary (for
example, requiring one module, forbidding replace directives). We could relax
these restrictions without breaking compatibility in the future if it seems
expedient. It would be much harder to add restrictions later.

Implementation

An initial implementation of this feature was merged in
CL 254365. Please try it
out!

Future directions

The behavior with respect to replace directives was discussed extensively
before this proposal was written. There are three potential behaviors:

1. Ignore replace directives in all modules. This would be consistent with
   other module-aware commands, which only apply replace directives from the
   main module (defined in the current directory or a parent directory).
   go install pkg@version ignores the current directory and any go.mod
   file that might be present, so there is no main module.
2. Ensure only one module provides packages named on the command line, and
   treat that module as the main module, applying its module replace
   directives from it. Report errors for directory replace directives. This
   is feasible, but it may have wider ecosystem effects; see "Why can't module
   replace directives be used?" above.
3. Ensure only one module provides packages named on the command line, and
   report errors for any replace directives it contains. This is the behavior
   currently proposed.

Most people involved in this discussion have advocated for either (1) or (2).
The behavior in (3) is a compromise. If we find that the behavior in (1) is
strictly better than (2) or vice versa, we can switch to that behavior from
(3) without an incompatible change. Additionally, (3) eliminates
ambiguity about whether replace directives are applied for users and module
authors.

Note that applying directory replace directives is not considered here for
the reasons in "Why can't directory replace directives be used?".

Appendix: FAQ

Why not apply replace directives from all modules?

In short, replace directives from different modules would conflict, and
that would make dependency management harder for most users.

For example, consider a case where two dependencies replace the same module
with different forks.

// in example.com/mod/a
replace example.com/mod/c => example.com/fork-a/c v1.0.0

// in example.com/mod/b
replace example.com/mod/c => example.com/fork-b/c v1.0.0

Another conflict would occur where two dependencies pin different versions
of the same module.

// in example.com/mod/a
replace example.com/mod/c => example.com/mod/c v1.1.0

// in example.com/mod/b
replace example.com/mod/c => example.com/mod/c v1.2.0

To avoid the possibility of conflict, the go command ignores replace
directives in modules other than the main module.

Modules are intended to scale to a large ecosystem, and in order for upgrades
to be safe, fast, and predictable, some rules must be followed, like semantic
versioning and import compatibility.
Not relying on replace is one of these rules.

How can module authors avoid replace?

replace is useful in several situations for local or short-term development,
for example:

• Changing multiple modules concurrently.
• Using a short-term fork of a dependency until a change is merged upstream.
• Using an old version of a dependency because a new version is broken.
• Working around migration problems, like golang.org/x/lint imported as
  github.com/golang/lint. Many of these problems should be fixed by lazy
  module loading (#36460).

replace is safe to use in a module that is not depended on by other modules.
It's also safe to use in revisions that aren't depended on by other modules.

• If a replace directive is just meant for temporary local development by one
  person, avoid checking it in. The -modfile flag may be used to build with
  an alternative go.mod file. See also
  #26640 a feature request for a
  go.mod.local file containing replacements and other local modifications.
• If a replace directive must be checked in to fix a short-term problem,
  ensure at least one release or pre-release version is tagged before checking
  it in. Don't tag a new release version with replace checked in (pre-release
  versions may be okay, depending on how they're used). When the go command
  looks for a new version of a module (for example, when running go get with
  no version specified), it will prefer release versions. Tagging versions lets
  you continue development on the main branch without worrying about users
  fetching arbitrary commits.
• If a replace directive must be checked in to solve a long-term problem,
  consider solutions that won't cause issues for dependent modules. If possible,
  tag versions on a release branch with replace directives removed.

When would go install be reproducible?

The new go install command will build an executable with the same set of
module versions on every invocation if both the following conditions are true:

• A specific version is requested in the command line argument, for example,
  go install example.com/cmd/foo@v1.0.0.
• Every package needed to build the executable is provided by a module required
  directly or indirectly by the go.mod file of the module providing the
  executable. If the executable only imports standard library packages or
  packages from its own module, no go.mod file is necessary.

An executable may not be bit-for-bit reproducible for other reasons. Debugging
information will include system paths (unless -trimpath is used). A package
may import different packages on different platforms (or may not build at all).
The installed Go version and the C toolchain may also affect binary
reproducibility.

What happens if a module depends on a newer version of itself?

go install will report an error, as go get already does.

This sometimes happens when two modules depend on each other, and releases
are not tagged on the main branch. A command like go get example.com/m@master
will resolve @master to a pseudo-version lower than any release version.
The go.mod file at that pseudo-version may transitively depend on a newer
release version.

go get reports an error in this situation. In general, go get reports
an error when command line arguments different versions of the same module,
directly or indirectly. go install doesn't support this yet, but this should
be one of the conditions checked when running with version suffix arguments.

Appendix: usage of replace directives

In this proposal, go install would report errors for replace directives in
the module providing packages named on the command line. go get ignores these,
but the behavior may still surprise module authors and users. I've tried to
estimate the impact on the existing set of open source modules.

• I started with a list of 359,040 main packages that Russ Cox built during an
  earlier study.
• I excluded packages with paths that indicate they were homework, examples,
  tests, or experiments. 187,805 packages remained.
• Of these, I took a random sample of 19,000 packages (about 10%).
• These belonged to 13,874 modules. For each module, I downloaded the "latest"
  version go get would fetch.
• I discarded repositories that were forks or couldn't be retrieved. 10,618
  modules were left.
• I discarded modules that didn't have a go.mod file. 4,519 were left.
• Of these:
  • 3982 (88%) don't use replace at all.
  • 71 (2%) use directory replace only.
  • 439 (9%) use module replace only.
  • 27 (1%) use both.
  • In the set of 439 go.mod files using module replace only, I tried to
    classify why replace was used. A module may have multiple replace
    directives and multiple classifications, so the percentages below don't add
    to 100%.
  • 165 used replace as a soft fork, for example, to point to a bug fix PR
    instead of the original module.
  • 242 used replace to pin a specific version of a dependency (the module
    path is the same on both sides).
  • 77 used replace to rename a dependency that was imported with another
    name, for example, replacing github.com/golang/lint with the correct path,
    golang.org/x/lint.
  • 30 used replace to rename golang.org/x repos with their
    github.com/golang mirrors.
  • 11 used replace to bypass semantic import versioning.
  • 167 used replace with k8s.io modules. Kubernetes has used replace to
    bypass MVS, and dependent modules have been forced to do the same.
  • 111 modules contained replace directives I couldn't automatically
    classify. The ones I looked at seemed to mostly be forks or pins.

The modules I'm most concerned about are those that use replace as a soft fork
while submitting a bug fix to an upstream module; other problems have other
solutions that I don't think we need to design for here. Modules using soft fork
replacements are about 4% of the the modules with go.mod files I sampled (165
/ 4519). This is a small enough set that I think we should move forward with the
proposal above.

[gopherbot]

Change https://golang.org/cl/243077 mentions this issue: design: add 40276-go-get-b.md

[peebs]

Thanks for the proposal!

• go get acts as if no go.mod file is present in the current directory or
  parent directory.
• No module will be considered the "main" module.

Would it also be accurate to say that the go.mod being used as the main module is that of the module being built rather than no "main" module? Erroring on replaces is then simply a consequence of the implementation preventing replaces from being respected because we consider the module the "main". If the intention was not to have a main module, one might expect replaces to be ignored just like go get does today outside of a module. Regardless on how go get -b and go get outside a module end up unifying, it seems that the intention of go get -b today is to build reproducible binaries using the packages module as the main module or fail if unable to do so. I know the outcome is the same here, but I think the stated intention helps a user know what to expect of this mode.

• The -u flag may be used together with -b. As usual, -u upgrades modules
  providing packages imported directly or indirectly by packages named on the
  command line

go get is used to download and install executables, but it's also responsible
for managing dependencies in go.mod files. This causes confusion and
unintended side effects

Since go get -b is a distinct mode of go get that is explicitly is not for managing dependencies of a local go.mod, my instinct here is that -u shouldn't be an option as it further confuses the distinction between using go get for building binaries vs managing dependencies. I at least don't know a scenario where I would want to build a project with updated dependencies and not view and work with the go.mod afterword. For something like that I would always have the repo cloned already as I am now building previously unused combination of dependencies with that project and would probably also want to run tests and maybe downgrade certain dependencies if necessary.

[jayconrod]

Would it also be accurate to say that the go.mod being used as the main module is that of the module being built rather than no "main" module? Erroring on replaces is then simply a consequence of the implementation preventing replaces from being respected because we consider the module the "main". If the intention was not to have a main module, one might expect replaces to be ignored just like go get does today outside of a module. Regardless on how go get -b and go get outside a module end up unifying, it seems that the intention of go get -b today is to build reproducible binaries using the packages module as the main module or fail if unable to do so. I know the outcome is the same here, but I think the stated intention helps a user know what to expect of this mode.

That's not quite the intent: there will actually be no main module.

In an earlier iteration on #30515, I suggested that go get -b would act the same as go get, but it would ignore the go.mod in the current directory: it would ignore replace. Many people found that unintuitive though. It seemed like we eventually had consensus that go get -b should report errors if there's more than one module on the command line or if there are replace directives. That way, you'll build the same binary whether it's: using go build inside the module providing the executable, using go get outside any module, or using go get -b anywhere.

Since go get -b is a distinct mode of go get that is explicitly is not for managing dependencies of a local go.mod, my instinct here is that -u shouldn't be an option as it further confuses the distinction between using go get for building binaries vs managing dependencies. I at least don't know a scenario where I would want to build a project with updated dependencies and not view and work with the go.mod afterword. For something like that I would always have the repo cloned already as I am now building previously unused combination of dependencies with that project and would probably also want to run tests and maybe downgrade certain dependencies if necessary.

There is some use for -u here: it lets you build and install an executable with updated dependencies. go get -u can already do that when run outside a module. I don't expect it's very common, but we get it for free, so I don't think there's a good reason to break it.

[peebs]

had consensus that go get -b should report errors if there's more than one module on the command line or if there are replace directives. That way, you'll build the same binary whether it's: using go build inside the module providing the executable, using go get outside any module, or using go get -b anywhere.

I think we are on the same page here then regardless!

[jayconrod]

Based on discussion in https://groups.google.com/g/golang-tools/c/BRCgqwWLwoY/m/pKuttL9cAwAJ and on Slack, I'd like to take this in a different direction, so I've updated CL 243077 and the copy above.

Instead of adding go get -b, go install would have the proposed functionality when invoked with arguments with a version suffix. For example the command below would install gopls v0.4.4. It could be run from any directory and would ignore the module in the current directory if there is one.

go install golang.org/x/tools/gopls@v0.4.4

go get would no longer build or install packages. It would only be used for changing dependencies.

PTAL and comment on CL 243077 if you have any thoughts. This would be a significant change, but it would be good to have a clear separation of responsibility for go get and go install.

[gopherbot]

Change https://golang.org/cl/254365 mentions this issue: cmd/go: implement 'go install pkg@version'

[abursavich]

@jayconrod

Moving conversation over from #40728, my only sticking point is that I agree with you here: #30515 (comment) 💯

When called from inside a module, the current behavior in tip is great when referring to a module-local package path, but when referring to a non-local package path go install package should behave the same as go install package@latest, so as to be consistent with other module-related commands.

[jayconrod]

@abursavich There are a couple significant problems with that approach. It would be a break from current go install behavior, and it would be inconsistent with other build commands (go build, go run, and so on).

It is confusing that there will be two variants of go install, but instead of changing the semantics of go install pkg, I think the best way to resolve that would be to remove go install pkg and only allow go install pkg@version. That seems like too disruptive of a change to me, but I'm curious to hear other opinions.

[abursavich]

I think it's important to keep go install pkg for module-local packages, particularly since replace directives are going to be ignored with @version... If you really want to drop the consistency hammer, go get should require an explicit @version to add or update a specific package in module-aware mode.

[abursavich]

I'm just gonna withdraw my comments and step out of the way of progress. Leave it like it is in tip :)

[aarzilli]

Is it too late to roll back the whole "global install command" thing completely? Because I feel like the way it's being impelmented I'd rather not have it in the ecosystem at all. I think the original impetus behind it was that approximately everyone had a README file that said "use go get to install this" and in modules mode it just did a bunch of weird things.

But at this point we've lived without it for 2 years, READMEs have been changed, the community has adapted, do we still need it? And if it isn't going to do ~100% the same thing as git clone && go build is it a net good or a net negative?

A side effect of the way it's being implemented is that it discourages applications from using replace directives, heavily, if it gets adopted by the community as the default way to install applications. It's strange that replace directives are in the language but the tooling discourages from using them. If this is the design we settle in then replace directives should get a deprecation warning in the documentation. But IMHO replace directives are in fact useful, more useful than a global install command, so if they can't be accomodated maybe it's the global install command that should go away.

[jayconrod]

Is it too late to roll back the whole "global install command" thing completely?

This proposal hasn't been formally accepted yet. However, I think there's a pretty firm consensus to move forward with this in 1.16. The lack of a global install command has been a significant usability problem, and many people have asked for something that solves this problem.

But at this point we've lived without it for 2 years, READMEs have been changed, the community has adapted, do we still need it?

I don't think the community has adapted. Many instructions I've seen recommend changing to a directory outside a module, then running go get in either GOPATH mode or with an explicit GO111MODULE=on. I think we need a more convenient, consistent way of installing tools.

And if it isn't going to do ~100% the same thing as git clone && go build is it a net good or a net negative?

In module mode, this would be unexpected behavior. The go command usually fetches dependencies from a proxy. Even in direct mode, it doesn't fetch a whole repository. This is discussed in the design doc.

A side effect of the way it's being implemented is that it discourages applications from using replace directives, heavily, if it gets adopted by the community as the default way to install applications. It's strange that replace directives are in the language but the tooling discourages from using them. If this is the design we settle in then replace directives should get a deprecation warning in the documentation.

replace is not deprecated, and there's no issue with using it during development.

replace is discouraged in release versions that other modules may depend on. replace directives are only applied in the main module; they're ignored by go install and go get in another module or outside modules. This leads to subtle differences in selected versions depending on what command was issued where. We felt that reporting an error was preferable to that ambiguity.

This is also discussed in the design doc and at length in the previous discussion in #30515.

[peebs]

But at this point we've lived without it for 2 years, READMEs have been changed, the community has adapted, do we still need it?

I agree with jayconrad that people haven't really adapted and switching to modules has made things more confusing between go install and go get so I think this is a big win in tooling clarity.

And if it isn't going to do ~100% the same thing as git clone && go build is it a net good or a net negative?

I also think that anything that doesn't build the same binary as git clone && go build is not whats expected. However a really important part of this change is that now go install will refuse to build a module with replace directives and error out rather then ignoring it. Building with replace directives is intuitively what I want with go install but erroring on encountering them is the right thing to do instead. Related to this limitation, go install will be much faster then git clone && go build. Before this change go get would ignore replaces building external binaries so I think this change brings you closer to what you want.

[mvdan]

I want to give extra emphasis to a disadvantage of git clone && [...] that @jayconrod already mentioned. Using the VCS repository directly is:

• slower, since you have to download a lot more (module zips are generally much smaller)
• less reliable, since you can't make use of module proxies
• not future-proof, since a module might change its VCS repo at any point without changing its module path

[aarzilli]

Is it too late to roll back the whole "global install command" thing completely?

This proposal hasn't been formally accepted yet. However, I think there's a pretty firm consensus to move forward with this in 1.16. The lack of a global install command has been a significant usability problem, and many people have asked for something that solves this problem.

I don't disagree with this, in fact I would also like a global install command. The problem that I have is that I dislike the way this is being implemented more than I would like having a global install.

And if it isn't going to do ~100% the same thing as git clone && go build is it a net good or a net negative?

In module mode, this would be unexpected behavior. The go command usually fetches dependencies from a proxy. Even in direct mode, it doesn't fetch a whole repository. This is discussed in the design doc.

I disagree, I think that this would be the expected behavior and that the current behavior of the go command is actually a regression. go get used to do a repository clone.

Address the "Why can't go get clone a git repository and build from there?" list of problems:

It adds complication (and bugs!) to the go command to support a new build mode.

It wouldn't be a new build mode, it would be the same build mode as executing a go build in the root directory of the project. It would add an alternate way to download projects (which used to exist previously).

It creates work for authors, who would need to ensure their programs can be built with both go get and go install.

In reality most code is developed by running go build on a cloned is the normal, inevitable, way of doing development. What adds work for authors is introducing a new way of downloading the code that ignores replace directives and vendor directories. The solution is to also fix go get to build executables as if git clone && go build.

It reduces speed and reliability for users. Modules may be available on a proxy when the original repository is unavailable. Fetching modules from a proxy is roughly 5-7x faster than cloning git repositories.

I concede that this is a valid point, however correctness is more important than performance.

Speaking of unexpected behavior, I think not respecting replace directives is a far more unexpected behavior than the changing details of how a module is downloaded from the internet.

A side effect of the way it's being implemented is that it discourages applications from using replace directives, heavily, if it gets adopted by the community as the default way to install applications. It's strange that replace directives are in the language but the tooling discourages from using them. If this is the design we settle in then replace directives should get a deprecation warning in the documentation.

replace is not deprecated, and there's no issue with using it during development.

replace is discouraged in release versions that other modules may depend on. replace directives are only applied in the main module; they're ignored by go install and go get in another module or outside modules. This leads to subtle differences in selected versions depending on what command was issued where. We felt that reporting an error was preferable to that ambiguity.

When was it decided that replace directives are discouraged from release versions? I've scanned through the vgo blog series and the current Go Modules reference and I can't find anything suggesting this.

This is also discussed in the design doc and at length in the previous discussion in #30515.

It seems to me that the design doc and the previous discussion is simply arguing consistency with pathological behaviors of go get when used to build executables, but I think the point of adding a global install command would be to fix those problems not to further entrench them.

[philipwhiuk]

Why wasn't the removal of go get a major version change. What's the point of 1. if Go maintainers are happy to make breaking changes in a point release. It's fundamentally broken builds that don't use modules.

[mvdan]

@philipwhiuk please read https://go.dev/doc/go1compat:

Finally, the Go toolchain (compilers, linkers, build tools, and so on) is under active development and may change behavior. This means, for instance, that scripts that depend on the location and properties of the tools may be broken by a point release.

Of course they are still careful when breaking the tooling, but it's allowed by the compatibility guarantee.

[philipwhiuk]

That "compatibility guarantee" is a big "yeah we don't care about semver for tools, good luck" isn't it.

[clausecker]

@philipwhiuk Again, the compatibility promise is about the language and libraries, not the tooling. It's more about being able to write code that will compile in the future. The toolchain is disposable and can change, the language is stable.

[Xe]

I know this is probably the wrong place to put this and that this feedback is likely going to be ignored, but this whole "the tooling is exempt from the compatibility promise" thing kind of feels like a cop-out. I understand the intent of this, as you use Go in production places then you will learn more things and realize what mistakes were made; however the community starts to standardize and expect the behaviour of commands to stay consistent unless there is a major version change. This violation of expectations is why you see people make more heated or angry comments. It feels like the tool changed out from under them and now they have to re-learn the tool.

I'd be willing to argue that the semantics of how commands like go get work is probably more important than the standard library being stable. Something like go fix can correct breaking changes in the standard library, but there is no go fix to correct the changes in how people install commands or the documentation people have written about how to install various tools written in Go.

Again, I get the point of why the tooling isn't behind the compat promise. It allows you to make the tooling better and fix behaviours that were outright bugs, but overall I'm really not sure if continuing to keep Go at 1.x post-modules was really a good idea. I know that politically making a "Go 2" is a difficult thing, but the kinds of changes that modules have given developers at nearly every level of the stack really seems like it should have been a candidate for being released as "Go 2". It's sad that this is politically difficult though, because "Go 2" has been lauded as a magical fairy unicorn thing where the standard library is going to be fixed forever or whatever. If anything what we have now may as well be "Go 2" because the Go I started learning near a decade ago is almost a completely different language than the Go I use today.

[philipwhiuk]

@philipwhiuk Again, the compatibility promise is about the language and libraries, not the tooling. It's more about being able to write code that will compile in the future. The toolchain is disposable and can change, the language is stable.

It barely matters if the code still "compiles" if I can't actually get it to compile by running the same commands. Code that works on 1.13.4 will have to be rewritten to use modules most likely because there's no explanation of how to do what I was doing with go get in modern Golang. The compile promise is worthless on it's own.

[mvdan]

I think we're getting off topic for this thread, especially as it's been closed for a while. I would suggest to give #37755 a read, which was a proposal to actually keep the pre-modules workflow working forever. For any other meta discussion about the backwards compatibility guarantee, I would personally suggest https://groups.google.com/g/golang-dev or perhaps a new issue if you have a specific proposal in mind.