cmd/go: `go get -u` on main module bloats go.mod with unnecessary dependencies

[inliquid]

What did you do?

Consider this simple test app:

package main

import (
	"fmt"
	"time"

	"github.com/jinzhu/gorm"
	_ "github.com/lib/pq"
)

var db *gorm.DB

func init() {
	var err error

	db, err = gorm.Open("postgres", "postgres://gorm:gorm@127.0.0.1/gorm?sslmode=disable")
	if err != nil {
		panic(err)
	}
}
func main() {
	t := time.Time{}
	db.Debug().Raw("SELECT NOW()").Row().Scan(&t)
	fmt.Println(t)
}

go.mod is also pretty simple:

module SomeDBTest

go 1.12

require (
	github.com/jinzhu/gorm v1.9.9
	github.com/lib/pq v1.1.1
)

Lets try to update this app dependencies by running go get -u -d, resulting go.mod will look as follows:

module SomeDBTest

go 1.12

require (
	cloud.google.com/go v0.41.0 // indirect
	github.com/DataDog/zstd v1.4.0 // indirect
	github.com/Shopify/sarama v1.23.0 // indirect
	github.com/denisenkom/go-mssqldb v0.0.0-20190710001350-29e7b2419f38 // indirect
	github.com/eapache/go-resiliency v1.2.0 // indirect
	github.com/go-kit/kit v0.9.0 // indirect
	github.com/gogo/protobuf v1.2.1 // indirect
	github.com/golang/protobuf v1.3.2 // indirect
	github.com/gorilla/mux v1.7.3 // indirect
	github.com/jcmturner/gofork v1.0.0 // indirect
	github.com/jinzhu/gorm v1.9.10
	github.com/kisielk/errcheck v1.2.0 // indirect
	github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect
	github.com/lib/pq v1.1.1
	github.com/onsi/ginkgo v1.8.0 // indirect
	github.com/onsi/gomega v1.5.0 // indirect
	github.com/openzipkin/zipkin-go v0.2.0 // indirect
	github.com/pkg/errors v0.8.1 // indirect
	github.com/prometheus/common v0.6.0 // indirect
	github.com/prometheus/procfs v0.0.3 // indirect
	github.com/rcrowley/go-metrics v0.0.0-20190706150252-9beb055b7962 // indirect
	github.com/sirupsen/logrus v1.4.2 // indirect
	github.com/stretchr/objx v0.2.0 // indirect
	golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4 // indirect
	golang.org/x/exp v0.0.0-20190627132806-fd42eb6b336f // indirect
	golang.org/x/image v0.0.0-20190703141733-d6a02ce849c9 // indirect
	golang.org/x/mobile v0.0.0-20190711165009-e47acb2ca7f9 // indirect
	golang.org/x/net v0.0.0-20190628185345-da137c7871d7 // indirect
	golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7 // indirect
	golang.org/x/tools v0.0.0-20190712213246-8b927904ee0d // indirect
	google.golang.org/genproto v0.0.0-20190708153700-3bdd9d9f5532 // indirect
	google.golang.org/grpc v1.22.0 // indirect
	gopkg.in/jcmturner/gokrb5.v7 v7.3.0 // indirect
)

Now let's try go mod tidy if it's possible to remove some redundant dependencies:

module SomeDBTest

go 1.12

require (
	cloud.google.com/go v0.41.0 // indirect
	github.com/denisenkom/go-mssqldb v0.0.0-20190710001350-29e7b2419f38 // indirect
	github.com/jinzhu/gorm v1.9.10
	github.com/lib/pq v1.1.1
	golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4 // indirect
)

What did you expect to see?

Result of running go get -u -d should match corresponding go mod tidy on updated dependencies in go.mod, or match those lines in go.mod being updated according to documented behavior of go get -u without adding any additional indirect dependencies.

What did you see instead?

go.mod is bloated with indirect dependencies, forcing user to run go mod tidy additionally, or edit it manually.

System details

go version go1.12.7 windows/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="C:\Users\***\AppData\Local\go-build"
GOEXE=".exe"
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="windows"
GOOS="windows"
GOPATH="C:\Users\***\go"
GOPROXY=""
GORACE=""
GOROOT="c:\go"
GOTMPDIR=""
GOTOOLDIR="c:\go\pkg\tool\windows_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="NUL"
GOROOT/bin/go version: go version go1.12.7 windows/amd64
GOROOT/bin/go tool compile -V: compile version go1.12.7

[thepudds]

@inliquid would you be able to try this with the Go 1.13 beta?

There were some bug fixes in this area (e.g., #29702), as well as the overall upgrade behavior changed in a way that often fairly dramatically decreases how many indirect dependencies get updated by go get -u, which also affected the meaning of several go get flags. This is summarized in the go get section of the draft Go 1.13 release notes:
https://tip.golang.org/doc/go1.13#go-get

[inliquid]

@thepudds
Tried with this version:

$ gotip version
go version devel +89d300b Sun Jul 14 20:15:47 2019 +0000 linux/amd64

After running go get -u -d there is clean go.mod file with updated versions of dependencies and no indirect entries:

module SomeDBTest

go 1.13

require (
        github.com/jinzhu/gorm v1.9.10
        github.com/lib/pq v1.1.1
)

Then I tried go mod tidy with same result. Very good!

[jayconrod]

Glad to hear this is working. This was fixed in CL 174099 and a few follow-up CLs. Unfortunately, it's too big of a change to backport.

Previously, go get -u meant "upgrade the named modules and everything they transitively require". Without arguments, it would start at the main module, so it would upgrade everything in the module graph (way more than what anyone wants).

In go1.13, go get -u means "upgrade the modules providing the named packages and everything they transitively import". This is much narrower, and more similar to what go get -u did in GOPATH mode.