x/build: reconsider the large number of third-party dependencies #29935

mvdan · 2019-01-25T11:25:53Z

Today, a coworker and myself noticed how doing a go get -u downloaded a large number of modules which were completely uninteresting to us, mostly from @dmitshur:

go: finding github.com/shurcooL/go-goon latest
go: finding github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470
go: finding github.com/shurcooL/component latest
go: finding github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50
go: finding honnef.co/go/tools latest
go: finding github.com/shurcooL/events latest
go: finding github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
go: finding github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191
go: finding github.com/gregjones/httpcache latest
go: finding sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4
go: finding github.com/neelance/sourcemap latest
go: finding github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc
go: finding github.com/jellevandenhooff/dkim latest
go: finding github.com/shurcooL/htmlg latest
go: finding github.com/shurcooL/highlight_diff latest
go: finding github.com/shurcooL/github_flavored_markdown latest
go: finding golang.org/x/exp v0.0.0-20190121172915-509febef88a4
go: finding github.com/shurcooL/go latest
go: finding github.com/russross/blackfriday v1.5.2
go: finding github.com/shurcooL/issues latest
go: finding sourcegraph.com/sqs/pbtypes v1.0.0
go: finding github.com/shurcooL/httperror latest
go: finding github.com/prometheus/procfs latest

You can reproduce this by running go get -u on https://github.com/xo/usql. What's more worrying, commands like go mod why github.com/shurcooL/httpgzip report that the main module doesn't need all those packages.

After some manual digging, we ended up tracing this to the cloud.google.com/go module: https://github.com/googleapis/google-cloud-go/blob/52299f000fab02ff521d180fd9f101ceb44d0f38/go.mod

And in particular, its requirement on the golang.org/x/build module: https://github.com/golang/build/blob/ad59fb13d31575de611cdf1252fa3fbfd8f07794/go.mod

It's in that last go.mod file where one can see over fifty third-party module requirements, counting // indirect lines.

Now, I would understand that the module which sets up golang.org's CI infrastructure needs to have lots of dependencies. After all, it needs to do lots of work. And that would be fine, as long as extremely few packages required the x/build module.

However, cloud.google.com/go, and by extension many other popular modules like google.golang.org/grpc and google.golang.org/api, require x/build. So any developer wishing to use Google Cloud with Go is forced to download dozens of third-party Go modules for no apparent reason. This can add up quickly even on a warm cache, as go get -u needs to do much more work.

I do understand that, once module mirrors are widespread, we'll only have to download go.mod files for these unnecessary modules, and not clone their entire git repositories. Still, I think it's bad design that cloud.google.com/go is pulling so many module requirements, some of which are exclusively for frontend/JS coding.

I can imagine multiple solutions, besides leaving the situation as it is now:

Actively discourage "library" modules from requiring x/build (though cloud.google.com/go imports non-trivial chunks of x/build)
Split up the library-useful bits of x/build into a separate module
Gradually remove the third party dependencies of x/build, bringing it more in line with other golang.org/x/... repos

I realise I could have raised this issue against the google-cloud-go repository, but I think it's best to open it here first. Developers also tend to trust golang.org/x repos, so I personally think it's reasonable of them to default to trusting x/build.

/cc @bradfitz @dmitshur @andybons @bcmills

The text was updated successfully, but these errors were encountered:

dmitshur · 2019-01-25T14:04:14Z

What's more worrying, commands like go mod why github.com/shurcooL/httpgzip report that the main module doesn't need all those packages.

Every single one of those packages coming from me are needed only by a single command in x/build:

https://godoc.org/golang.org/x/build/maintner/cmd/maintserve

See:

https://godoc.org/golang.org/x/build/maintner/cmd/maintserve?imports

It’s a small program that reuses a lot of my external code to visualize maintner data offline. It’s very dependency heavy.

Before modules, it wasn’t a problem to have it in x/build because you would only fetch its dependencies if you ran go get -u golang.org/x/build/maintner/cmd/maintserve. They wouldn’t be downloaded if you were installing any other libraries or commands in x/build or things that imported a library from x/build.

If it’s causing problems for people in module world, we can consider removing that command from x/build, or making it be a part of a separate module. /cc @bcmills

I’m happy to discuss this.

Some additional observations:

I’m surprised that google cloud libraries would be relying on x/build module
~~I don’t understand why dependencies of a single command in x/build module need to be downloaded when building something that only imports libraries from x/build~~
Update: I understand this by now
it’s unfortunate to hear that go mod why wasn’t more helpful for you to figure out where the httpgzip package is coming from

bcmills · 2019-01-25T14:31:08Z

(Responses in no particular order.)

commands like go mod why github.com/shurcooL/httpgzip report that the main module doesn't need all those packages.

go mod why gives you a path through the package-import graph, not through the module graph. (See #27900.)

bcmills · 2019-01-25T15:01:45Z

After some manual digging, we ended up tracing this to the cloud.google.com/go module

I note:

~/src/cloud.google.com/go$ go mod why -m golang.org/x/build
# golang.org/x/build
cloud.google.com/go/profiler/proftest
golang.org/x/build/kubernetes

Arguably cloud.google.com/go should use the official Kubernetes client (k8s.io/client-go) instead of the one from golang.org/x/build (which is specifically tailored for use within x/build, and to my knowledge carries no compatibility guarantees).

If the official kubernetes client is too heavy-weight for cloud.google.com, that's probably something they should take up with Kubernetes folks directly.

(CC @jadekler @enocom @jba @zombiezen for cloud.google.com; CC @liggitt for k8s.io/client-go)

bcmills · 2019-01-25T15:06:22Z

Note that x/build is almost entirely project-internal infrastructure.

x/tools is the more user-facing “utility junk drawer”, and it does depend on x/build (via golang.org/x/tools/cmd/tip), but we should arguably solve that by carving cmd/tip out of x/tools rather than worrying too much about the dependencies of x/build.

bcmills · 2019-01-25T15:08:22Z

So I guess that's all to say, the option I prefer is the first one that @mvdan mentioned:

Actively discourage "library" modules from requiring x/build

bradfitz · 2019-01-25T15:11:14Z

I'm also fine splitting out useful stuff elsewhere.

dmitshur · 2019-01-25T15:15:54Z

About x/tools/cmd/tip, we plan to move that out of x/tools (into x/website or x/build) as part of #29206. /cc @cnoellekb

enocom · 2019-01-25T15:31:54Z

/cc @aalexand whose team owns the cloud.google.com/go/profiler package.

mvdan · 2019-01-25T15:38:47Z

Thanks all for the replies :)

not through the module graph. (See #27900.)

Thanks - that will help dig through these issues in the future.

Note that x/build is almost entirely project-internal infrastructure.

I'm of course fine with this solution, as long as it's enforced properly. In that case, I'm happy for this issue to be forked into x/tools and cloud-go issues, if that would be better.

I don’t understand why dependencies of a single command in x/build module need to be downloaded when building something that only imports libraries from x/build

That's how modules and MVS work; #29832 is somewhat related. In short, go get -u needs to get all the go.mod files in the entire module graph. Perhaps @bcmills can clarify if I said anything wrong.

mvdan · 2019-01-25T16:07:39Z

when building something that only imports libraries from x/build

A clarification - note that my issue is about running go get -u, not go build or go test. Having to fetch the latest version of all these unnecessary modules is what's causing slowness and pain.

aalexand · 2019-01-25T16:23:58Z

It should be totally reasonable to get rid of golang.org/x/build/kubernetes dependency in cloud.google.com/go/profiler/proftest, probably replacing it with k8s.io/client-go. In fact, for now that code can just be removed as the profiler GKE tests are broken and that code needs to be updated anyway. So the immediate action is I think to remove the dependency and the code from proftest.go, then figure out how to do it proper.

Another thing to discuss here (and this becomes really cloud-go specific, so not sure how much we want to discuss it here) is that the proftest package is a package that is intended to be used only by tests in profiling agents. Currently it's just use by the Go profiling agent integration test plus by Node.js profiling agent integration test that we own at https://github.com/googleapis/cloud-profiler-nodejs. This package is intended to be useful to us and "invisible" / transparent to the rest of cloud-go users. If it's going to hurt users of cloud-go even with k8s.io/client-go dependency instead of golang.org/x/build/kubernetes, we should re-think the existence of this package.

@nolanmar511 @jqll @kalyanac FYI

bradfitz · 2019-01-25T16:42:31Z

Can you hide that package behind a build tag?

aalexand · 2019-01-25T16:51:36Z

@bradfitz If I add a build tag in proftest.go, will a test with a dependency on that package like https://github.com/googleapis/cloud-profiler-nodejs/blob/6190f20fa3cb24456d8d9ac37b0891b39ebb122a/testing/integration_test.go#L30 still build / run as is?

thepudds · 2019-01-25T17:14:15Z

go mod tidy looks across all build tags, so build tag alone might be insufficient to address the initial concern raised by @mvdan.

Edit: and sorry, not attempting to comment on all that’s been said here. Just wanted to inject one comment on current behavior.

bcmills · 2019-01-25T17:50:53Z

Parameterizing mod tidy on build tags is #25873.

bcmills · 2019-01-25T17:54:28Z

This package is intended to be useful to us and "invisible" / transparent to the rest of cloud-go users.

In that case, you could consider splitting it out into its own module.

jeanbza · 2019-01-25T18:03:48Z

That will happen soon (tm) but we're not ready with tooling for multiple modules in that repository yet, regrettably. By 1.13 we should hopefully be there.

gopherbot · 2019-02-13T19:32:21Z

Change https://golang.org/cl/162401 mentions this issue: godoc/dl, godoc/proxy, godoc/short, internal/memcache: delete

These packages existed only to power cmd/godoc for the purpose of serving the golang.org website. That functionality has moved into x/website as part of golang/go#29206. x/website has become the canonical source of golang.org in CL 162157, the golang.org-serving code was removed from cmd/godoc in CL 162400, and these packages can be deleted too now. This removes the last dependency on the cloud.google.com/go module, which results in a significant reduction of the number of indirect dependencies in x/tools (this is due to issue golang/go#29935, which affects the current version of the cloud.google.com/go module). Run go mod tidy (using Go 1.12 RC 1). Updates golang/go#29206 Updates golang/go#29981 Change-Id: If07e3ccae8538b3ebd51af64b6af5be5463f4906 Reviewed-on: https://go-review.googlesource.com/c/162401 Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Channing Kimble-Brown <channing@golang.org> Reviewed-by: Michael Matloob <matloob@golang.org>

dmitshur · 2019-03-01T04:35:45Z

@aalexand @jadekler If I understand correctly, the current plan on your side is to make the cloud.google.com/go module no longer require the golang.org/x/build module, is that right?

If so, it'd be helpful to open an issue in https://github.com/googleapis/google-cloud-go/issues to track that task, so people can subscribe to it for further updates on this topic. I can open the issue if that'd be helpful.

jeanbza · 2019-03-05T20:51:35Z

@dmitshur Absolutely. Please track in googleapis/google-cloud-go#1344

gopherbot · 2019-03-13T22:23:40Z

Change https://golang.org/cl/167461 mentions this issue: maintner/cmd/maintserve: carve out into a separate module

Now that a pseudo-version of golang.org/x/build module with maintserve carved out exists, the module golang.org/x/build/maintner/cmd/maintserve can depend on it. Do so now. This is analogous to CL 166857. Updates golang/go#29935 Change-Id: I661f32336ede26fde5581527959d31d164a00b63 Reviewed-on: https://go-review.googlesource.com/c/build/+/167463 Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>

gopherbot · 2019-03-28T21:23:13Z

Change https://golang.org/cl/170001 mentions this issue: [release-branch.go1.12] godoc/dl, godoc/proxy, godoc/short, internal/memcache: delete

gopherbot · 2019-04-16T14:29:21Z

Change https://golang.org/cl/170863 mentions this issue: cmd/gopherbot: CC triaged issues to owners

bradfitz · 2019-04-16T16:11:52Z

On a fresh machine with Go master (cb66462), I tried to go get a single binary out of x/build.

I don't have GOPROXY set. It took a long time downloading things and was a much worse experience than GOPATH mode:

ci20@ci20:~$ ./go/bin/go get -v golang.org/x/build/cmd/buildlet
Fetching https://golang.org/x/build/cmd/buildlet?go-get=1
Parsing meta tags from https://golang.org/x/build/cmd/buildlet?go-get=1 (status code 200)
get "golang.org/x/build/cmd/buildlet": found meta tag get.metaImport{Prefix:"golang.org/x/build", VCS:"git", RepoRoot:"https://go.googlesource.com/build"} at https://golang.org/x/build/cmd/buildlet?go-get=1
get "golang.org/x/build/cmd/buildlet": verifying non-authoritative meta tag
Fetching https://golang.org/x/build?go-get=1
Parsing meta tags from https://golang.org/x/build?go-get=1 (status code 200)
go: finding golang.org/x/build/cmd/buildlet latest
Fetching https://golang.org/x/build/cmd?go-get=1
Parsing meta tags from https://golang.org/x/build/cmd?go-get=1 (status code 200)
get "golang.org/x/build/cmd": found meta tag get.metaImport{Prefix:"golang.org/x/build", VCS:"git", RepoRoot:"https://go.googlesource.com/build"} at https://golang.org/x/build/cmd?go-get=1
get "golang.org/x/build/cmd": verifying non-authoritative meta tag
go: finding golang.org/x/build/cmd latest
Fetching https://golang.org/x/build?go-get=1
Parsing meta tags from https://golang.org/x/build?go-get=1 (status code 200)
get "golang.org/x/build": found meta tag get.metaImport{Prefix:"golang.org/x/build", VCS:"git", RepoRoot:"https://go.googlesource.com/build"} at https://golang.org/x/build?go-get=1
go: finding golang.org/x/build latest
go: downloading golang.org/x/build v0.0.0-20190416153556-d37bb9a5eaf1
go: extracting golang.org/x/build v0.0.0-20190416153556-d37bb9a5eaf1
Fetching https://go4.org?go-get=1
Fetching https://google.golang.org/appengine?go-get=1
Fetching https://golang.org/x/net?go-get=1
Fetching https://cloud.google.com/go?go-get=1
Fetching https://golang.org/x/sys?go-get=1
Fetching https://golang.org/x/time?go-get=1
Fetching https://grpc.go4.org?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
get "golang.org/x/net": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/time?go-get=1 (status code 200)
get "golang.org/x/time": found meta tag get.metaImport{Prefix:"golang.org/x/time", VCS:"git", RepoRoot:"https://go.googlesource.com/time"} at https://golang.org/x/time?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
get "golang.org/x/sys": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys?go-get=1
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
go: finding github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
go: finding golang.org/x/net v0.0.0-20181029044818-c44066c5c816
go: finding golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2
go: finding golang.org/x/sys v0.0.0-20181029174526-d69651ed3497
Parsing meta tags from https://google.golang.org/appengine?go-get=1 (status code 200)
get "google.golang.org/appengine": found meta tag get.metaImport{Prefix:"google.golang.org/appengine", VCS:"git", RepoRoot:"https://github.com/golang/appengine"} at https://google.golang.org/appengine?go-get=1
go: finding google.golang.org/appengine v1.2.0
Parsing meta tags from https://cloud.google.com/go?go-get=1 (status code 200)
get "cloud.google.com/go": found meta tag get.metaImport{Prefix:"cloud.google.com/go", VCS:"git", RepoRoot:"https://code.googlesource.com/gocloud"} at https://cloud.google.com/go?go-get=1
Parsing meta tags from https://go4.org?go-get=1 (status code 200)
get "go4.org": found meta tag get.metaImport{Prefix:"go4.org", VCS:"git", RepoRoot:"https://github.com/go4org/go4"} at https://go4.org?go-get=1
go: finding cloud.google.com/go v0.31.0
Parsing meta tags from https://grpc.go4.org?go-get=1 (status code 200)
get "grpc.go4.org": found meta tag get.metaImport{Prefix:"grpc.go4.org", VCS:"git", RepoRoot:"https://github.com/go4org/grpc"} at https://grpc.go4.org?go-get=1
go: finding go4.org v0.0.0-20180809161055-417644f6feb5
go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: finding github.com/google/martian v2.1.0+incompatible
go: finding github.com/googleapis/gax-go v2.0.0+incompatible
go: finding github.com/google/go-github v17.0.0+incompatible
Fetching https://golang.org/x/oauth2?go-get=1
Parsing meta tags from https://golang.org/x/oauth2?go-get=1 (status code 200)
get "golang.org/x/oauth2": found meta tag get.metaImport{Prefix:"golang.org/x/oauth2", VCS:"git", RepoRoot:"https://go.googlesource.com/oauth2"} at https://golang.org/x/oauth2?go-get=1
go: finding golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4
Fetching https://go.opencensus.io?go-get=1
go: finding github.com/google/go-cmp v0.2.0
Parsing meta tags from https://go.opencensus.io?go-get=1 (status code 200)
get "go.opencensus.io": found meta tag get.metaImport{Prefix:"go.opencensus.io", VCS:"git", RepoRoot:"https://github.com/census-instrumentation/opencensus-go"} at https://go.opencensus.io?go-get=1
go: finding go.opencensus.io v0.18.0
go: finding github.com/golang/protobuf v1.2.0
Fetching https://golang.org/x/perf?go-get=1
Parsing meta tags from https://golang.org/x/perf?go-get=1 (status code 200)
get "golang.org/x/perf": found meta tag get.metaImport{Prefix:"golang.org/x/perf", VCS:"git", RepoRoot:"https://go.googlesource.com/perf"} at https://golang.org/x/perf?go-get=1
go: finding golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
go: finding golang.org/x/net v0.0.0-20180724234803-3673e40ba225
Fetching https://gopkg.in/inf.v0?go-get=1
Parsing meta tags from https://gopkg.in/inf.v0?go-get=1 (status code 200)
get "gopkg.in/inf.v0": found meta tag get.metaImport{Prefix:"gopkg.in/inf.v0", VCS:"git", RepoRoot:"https://gopkg.in/inf.v0"} at https://gopkg.in/inf.v0?go-get=1
go: finding gopkg.in/inf.v0 v0.9.1
Fetching https://golang.org/x/text?go-get=1
Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
get "golang.org/x/text": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text?go-get=1
go: finding golang.org/x/text v0.3.0
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
get "golang.org/x/crypto": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto?go-get=1
go: finding golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16
Fetching https://google.golang.org/genproto?go-get=1
Parsing meta tags from https://google.golang.org/genproto?go-get=1 (status code 200)
get "google.golang.org/genproto": found meta tag get.metaImport{Prefix:"google.golang.org/genproto", VCS:"git", RepoRoot:"https://github.com/google/go-genproto"} at https://google.golang.org/genproto?go-get=1
go: finding google.golang.org/genproto v0.0.0-20180831171423-11092d34479b
Fetching https://google.golang.org/api?go-get=1
Parsing meta tags from https://google.golang.org/api?go-get=1 (status code 200)
get "google.golang.org/api": found meta tag get.metaImport{Prefix:"google.golang.org/api", VCS:"git", RepoRoot:"https://code.googlesource.com/google-api-go-client"} at https://google.golang.org/api?go-get=1
go: finding google.golang.org/api v0.0.0-20181030000543-1d582fd0359e
go: finding google.golang.org/api v0.0.0-20180910000450-7ca32eb868bf
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding github.com/google/go-querystring v1.0.0
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
Fetching https://gopkg.in/check.v1?go-get=1
Parsing meta tags from https://gopkg.in/check.v1?go-get=1 (status code 200)
get "gopkg.in/check.v1": found meta tag get.metaImport{Prefix:"gopkg.in/check.v1", VCS:"git", RepoRoot:"https://gopkg.in/check.v1"} at https://gopkg.in/check.v1?go-get=1
go: finding google.golang.org/genproto v0.0.0-20181029155118-b69ba1387ce2
go: finding golang.org/x/net v0.0.0-20180906233101-161cd47e91fd
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding github.com/grpc-ecosystem/grpc-gateway v1.5.0
Fetching https://golang.org/x/sync?go-get=1
Parsing meta tags from https://golang.org/x/sync?go-get=1 (status code 200)
get "golang.org/x/sync": found meta tag get.metaImport{Prefix:"golang.org/x/sync", VCS:"git", RepoRoot:"https://go.googlesource.com/sync"} at https://golang.org/x/sync?go-get=1
go: finding github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: finding golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: finding github.com/openzipkin/zipkin-go v0.1.1
go: finding golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e
go: finding github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273
go: finding github.com/ghodss/yaml v1.0.0
go: finding github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
go: finding github.com/prometheus/client_golang v0.8.0
go: finding github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e
go: finding github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
Fetching https://google.golang.org/grpc?go-get=1
Parsing meta tags from https://google.golang.org/grpc?go-get=1 (status code 200)
get "google.golang.org/grpc": found meta tag get.metaImport{Prefix:"google.golang.org/grpc", VCS:"git", RepoRoot:"https://github.com/grpc/grpc-go"} at https://google.golang.org/grpc?go-get=1
go: finding git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999
go: finding google.golang.org/grpc v1.14.0
go: finding github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
go: finding golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2
go: finding github.com/gliderlabs/ssh v0.1.1
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/kr/pty v1.1.3
go: finding github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07
go: finding github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d
go: downloading cloud.google.com/go v0.31.0
go: downloading google.golang.org/api v0.0.0-20181030000543-1d582fd0359e
go: downloading golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4
go: downloading golang.org/x/net v0.0.0-20181029044818-c44066c5c816
go: downloading gopkg.in/inf.v0 v0.9.1
go: extracting gopkg.in/inf.v0 v0.9.1
go: extracting golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4
go: extracting golang.org/x/net v0.0.0-20181029044818-c44066c5c816
go: extracting cloud.google.com/go v0.31.0
go: extracting google.golang.org/api v0.0.0-20181030000543-1d582fd0359e
golang.org/x/net/context
golang.org/x/build
golang.org/x/net/context/ctxhttp
golang.org/x/oauth2/jws
cloud.google.com/go/compute/metadata
golang.org/x/oauth2/internal
golang.org/x/oauth2
google.golang.org/api/googleapi/internal/uritemplates
google.golang.org/api/googleapi
golang.org/x/oauth2/jwt
golang.org/x/oauth2/google
google.golang.org/api/gensupport
golang.org/x/build/maintner/maintnerd/maintapi/version
golang.org/x/build/types
gopkg.in/inf.v0
google.golang.org/api/compute/v1
google.golang.org/api/oauth2/v2
golang.org/x/build/kubernetes/api
golang.org/x/build/kubernetes
golang.org/x/build/internal/httpdl
golang.org/x/build/pargzip
golang.org/x/build/revdial
golang.org/x/build/buildenv
golang.org/x/build/dashboard
golang.org/x/build/buildlet
golang.org/x/build/cmd/buildlet
ci20@ci20:~$

The actual deps are:

	"Deps": [
		"archive/tar",
		"bufio",
		"bytes",
		"cloud.google.com/go/compute/metadata",
		"compress/flate",
		"compress/gzip",
		"container/list",
		"context",
		"crypto",
		"crypto/aes",
		"crypto/cipher",
		"crypto/des",
		"crypto/dsa",
		"crypto/ecdsa",
		"crypto/elliptic",
		"crypto/hmac",
		"crypto/internal/randutil",
		"crypto/internal/subtle",
		"crypto/md5",
		"crypto/rand",
		"crypto/rc4",
		"crypto/rsa",
		"crypto/sha1",
		"crypto/sha256",
		"crypto/sha512",
		"crypto/subtle",
		"crypto/tls",
		"crypto/x509",
		"crypto/x509/pkix",
		"encoding",
		"encoding/asn1",
		"encoding/base64",
		"encoding/binary",
		"encoding/hex",
		"encoding/json",
		"encoding/pem",
		"errors",
		"flag",
		"fmt",
		"golang.org/x/build",
		"golang.org/x/build/buildenv",
		"golang.org/x/build/buildlet",
		"golang.org/x/build/dashboard",
		"golang.org/x/build/internal/httpdl",
		"golang.org/x/build/kubernetes",
		"golang.org/x/build/kubernetes/api",
		"golang.org/x/build/maintner/maintnerd/maintapi/version",
		"golang.org/x/build/pargzip",
		"golang.org/x/build/revdial",
		"golang.org/x/build/types",
		"golang.org/x/net/context",
		"golang.org/x/net/context/ctxhttp",
		"golang.org/x/oauth2",
		"golang.org/x/oauth2/google",
		"golang.org/x/oauth2/internal",
		"golang.org/x/oauth2/jws",
		"golang.org/x/oauth2/jwt",
		"google.golang.org/api/compute/v1",
		"google.golang.org/api/gensupport",
		"google.golang.org/api/googleapi",
		"google.golang.org/api/googleapi/internal/uritemplates",
		"google.golang.org/api/oauth2/v2",
		"gopkg.in/inf.v0",
		"hash",
		"hash/crc32",
		"internal/bytealg",
		"internal/cpu",
		"internal/errinternal",
		"internal/fmtsort",
		"internal/nettrace",
		"internal/oserror",
		"internal/poll",
		"internal/race",
		"internal/reflectlite",
		"internal/singleflight",
		"internal/syscall/unix",
		"internal/testlog",
		"io",
		"io/ioutil",
		"log",
		"math",
		"math/big",
		"math/bits",
		"math/rand",
		"mime",
		"mime/multipart",
		"mime/quotedprintable",
		"net",
		"net/http",
		"net/http/httptrace",
		"net/http/httputil",
		"net/http/internal",
		"net/textproto",
		"net/url",
		"os",
		"os/exec",
		"os/signal",
		"os/user",
		"path",
		"path/filepath",
		"reflect",
		"regexp",
		"regexp/syntax",
		"runtime",
		"runtime/cgo",
		"runtime/internal/atomic",
		"runtime/internal/math",
		"runtime/internal/sys",
		"sort",
		"strconv",
		"strings",
		"sync",
		"sync/atomic",
		"syscall",
		"time",
		"unicode",
		"unicode/utf16",
		"unicode/utf8",
		"unsafe",
		"vendor/golang.org/x/crypto/chacha20poly1305",
		"vendor/golang.org/x/crypto/cryptobyte",
		"vendor/golang.org/x/crypto/cryptobyte/asn1",
		"vendor/golang.org/x/crypto/curve25519",
		"vendor/golang.org/x/crypto/hkdf",
		"vendor/golang.org/x/crypto/internal/chacha20",
		"vendor/golang.org/x/crypto/internal/subtle",
		"vendor/golang.org/x/crypto/poly1305",
		"vendor/golang.org/x/net/dns/dnsmessage",
		"vendor/golang.org/x/net/http/httpguts",
		"vendor/golang.org/x/net/http/httpproxy",
		"vendor/golang.org/x/net/http2/hpack",
		"vendor/golang.org/x/net/idna",
		"vendor/golang.org/x/text/secure/bidirule",
		"vendor/golang.org/x/text/transform",
		"vendor/golang.org/x/text/unicode/bidi",
		"vendor/golang.org/x/text/unicode/norm"
	],

But note above that it was downloading go-smtpd, yaml, gopkg.in/check.v1`, and all sorts of stuff not required by this binary.

So I'm in favor of more sub-modules in x/build, as cmd/go apparently can't do any better.

thepudds · 2019-04-16T16:53:41Z

Most often, setting GOPROXY can help significantly, including because it allows downloading a go.mod file without needing to fetch a repo, and my understanding is source code is not downloaded with a GOPROXY if it is not needed for the package-level import graph. In addition, the HTTPS requests to a GOPROXY are generally more efficient than using git as the transport in the absence of a GOPROXY.

It could be interesting to check if you see better behavior if you do something like export GOPROXY='https://gocenter.io' or perhaps point towards any Athens instance you might be running.

A couple sample datapoints:

https://github.com/sagikazarmark/go-module-dependency-test

johanbrandhorst/certify#67 (comment)

thepudds · 2019-04-17T14:15:59Z

@bradfitz Just to give some rough numbers, I ran a quick test of go get golang.org/x/build/cmd/buildlet with and without setting GOPROXY='https://gocenter.io':

With GOPROXY='https://gocenter.io':
- 14 sec., 136 MB in GOPATH/pkg/mod
Without GOPROXY set:
- 58 sec., 381 MB in GOPATH/pkg/mod

YMMV. This was with a clean GOPATH each time (and hence no module cache), but not a clean machine.

You had cited gopkg.in/check.v1 as an example concern. If you inspect the module cache afterwards, you can see that when GOPROXY was not set, the code for gopkg.in/check.v1 ends up in the module cache (because the go command needed to fetch all the code via git to find its go.mod). On the other hand, when GOPROXY was set, the module cache has the go.mod but not the code for gopkg.in/check.v1.

Sample test steps:

export GOPROXY='https://gocenter.io'
export GOPATH=$(mktemp -d)
cd $(mktemp -d)
time gotip get golang.org/x/build/cmd/buildlet

Should speed up downloads. See golang/go#29935 (comment) for details

dmitshur · 2019-06-04T17:45:33Z

Is there anything else we want to do in x/build specifically? I'm not seeing more to be done, and I think the original issue is resolved by now.

The overall resolution seems to be:

many maintserve-only dependencies were removed from x/build module and moved into a nested module in CL 167461
the remaining module requirements in x/build seem reasonable, nothing jumps out as unexpected
library authors should not depending on x/build module because it's large and exists primarily for the needs of the Go build infrastructure
if there are some small and targeted Go packages in x/build that are desirable for external library consumption, they can be factored out if we're willing to support them; individual issues can be filed for that

bcmills · 2019-06-05T18:43:06Z

nothing jumps out as unexpected

I dunno, it at least seems weird that we have direct dependencies on two different go-shlex modules.

dmitshur · 2019-06-05T19:45:42Z

Both go-shlex module requirements I see are indirect:

github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 // indirect
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 // indirect

What makes you say they're direct dependencies?

We are directly importing and requiring github.com/gliderlabs/ssh v0.1.1, which in turn directly imports and uses github.com/anmitsu/go-shlex. anmitsu/go-shlex has an example that imports flynn/go-shlex:

https://github.com/anmitsu/go-shlex/blob/648efa622239a2f6ff949fed78ee37b48d499ba4/example_test.go#L8

I'm not seeing a problem here that we need to fix.

thepudds · 2019-09-13T22:20:51Z

Wanted to briefly bounce this. Is there anything left to do here?

bradfitz · 2019-09-13T22:47:51Z

What do you mean by "bounce this"? Close it? Work on it? "Ping"?

dmitshur · 2019-09-13T23:54:24Z

My interpretation of "bounce" is the same meaning as "ping".

Is there anything left to do here?

In #29935 (comment), I concluded that the original issue has been resolved and that I don't see anything actionable left to do. Bryan brought up one concern, and I addressed it. No one has brought anything up since then, and it's been 3 months.

I think this is done, so I'll close. Otherwise feel free to speak up or open a new issue. Thanks to everyone who helped with this.

gopherbot added this to the Unreleased milestone Jan 25, 2019

gopherbot added the Builders x/build issues (builders, bots, dashboards) label Jan 25, 2019

bcmills added the modules label Jan 25, 2019

bcmills mentioned this issue Jan 25, 2019

x/tools: define module boundaries #27858

Closed

bcmills mentioned this issue Feb 12, 2019

x/*: define module boundaries for all the repos under golang.org/x/ #28136

Closed

mvdan mentioned this issue Mar 1, 2019

x/tools: vet brings massive amount of dependencies, errors when building #30506

Closed

jeanbza mentioned this issue Mar 5, 2019

profiler: remove x/build dependency googleapis/google-cloud-go#1344

Closed

dmitshur mentioned this issue Mar 8, 2019

x/net/http2/h2demo: should not be single-handedly responsible for adding 3 required modules to x/net #30685

Closed

bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 12, 2019

fkorotkov added a commit to cirruslabs/cirrus-ci-docs that referenced this issue Apr 22, 2019

Add $GOPROXY to Golang example

a16d629

Should speed up downloads. See golang/go#29935 (comment) for details

fkorotkov mentioned this issue Apr 22, 2019

Add $GOPROXY to Golang example cirruslabs/cirrus-ci-docs#259

Merged

fkorotkov added a commit to cirruslabs/cirrus-ci-docs that referenced this issue Apr 22, 2019

Add $GOPROXY to Golang example (#259)

7a07c77

Should speed up downloads. See golang/go#29935 (comment) for details

aeneasr mentioned this issue May 10, 2019

cmd/go: git tags with build metadata do not resolve #31713

Closed

dmitshur closed this as completed Sep 13, 2019

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Sep 13, 2019

dmitshur mentioned this issue Dec 13, 2019

x/website/_content: consider reusing repo list from x/build/repos instead of maintaining own hard-coded copy #36136

Open

dmitshur mentioned this issue Jan 10, 2020

x/build: document that it's a special internal repo, doesn't provide same level of stability as other golang.org/x repos #36505

Closed

rhcarvalho mentioned this issue Feb 4, 2020

Split sentry-go into multiple submodules getsentry/sentry-go#156

Closed

dmitshur mentioned this issue Feb 11, 2020

x/exp: reconsider using single module for all unrelated experimental packages of various maturity levels #37175

Closed

golang locked and limited conversation to collaborators Sep 13, 2020

gopherbot added the FrozenDueToAge label Sep 13, 2020

verdverm mentioned this issue Apr 6, 2021

Proposal: package management cuelang/cue#851

Closed

cueckoo mentioned this issue Jul 3, 2021

Proposal: package management cue-lang/cue#851

Closed

golang unlocked this conversation Jul 10, 2023

golang locked as resolved and limited conversation to collaborators Jul 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build: reconsider the large number of third-party dependencies #29935

x/build: reconsider the large number of third-party dependencies #29935

mvdan commented Jan 25, 2019 •

edited

Loading

dmitshur commented Jan 25, 2019 •

edited

Loading

bcmills commented Jan 25, 2019

bcmills commented Jan 25, 2019 •

edited

Loading

bcmills commented Jan 25, 2019

bcmills commented Jan 25, 2019 •

edited

Loading

bradfitz commented Jan 25, 2019

dmitshur commented Jan 25, 2019 •

edited

Loading

enocom commented Jan 25, 2019 •

edited

Loading

mvdan commented Jan 25, 2019

mvdan commented Jan 25, 2019

aalexand commented Jan 25, 2019

bradfitz commented Jan 25, 2019

aalexand commented Jan 25, 2019

thepudds commented Jan 25, 2019 •

edited

Loading

bcmills commented Jan 25, 2019

bcmills commented Jan 25, 2019

jeanbza commented Jan 25, 2019

gopherbot commented Feb 13, 2019

dmitshur commented Mar 1, 2019

jeanbza commented Mar 5, 2019

gopherbot commented Mar 13, 2019

gopherbot commented Mar 28, 2019

gopherbot commented Apr 16, 2019

bradfitz commented Apr 16, 2019

thepudds commented Apr 16, 2019 •

edited

Loading

thepudds commented Apr 17, 2019 •

edited

Loading

dmitshur commented Jun 4, 2019

bcmills commented Jun 5, 2019

dmitshur commented Jun 5, 2019

thepudds commented Sep 13, 2019

bradfitz commented Sep 13, 2019

dmitshur commented Sep 13, 2019 •

edited

Loading

x/build: reconsider the large number of third-party dependencies #29935

x/build: reconsider the large number of third-party dependencies #29935

Comments

mvdan commented Jan 25, 2019 • edited Loading

dmitshur commented Jan 25, 2019 • edited Loading

bcmills commented Jan 25, 2019

bcmills commented Jan 25, 2019 • edited Loading

bcmills commented Jan 25, 2019

bcmills commented Jan 25, 2019 • edited Loading

bradfitz commented Jan 25, 2019

dmitshur commented Jan 25, 2019 • edited Loading

enocom commented Jan 25, 2019 • edited Loading

mvdan commented Jan 25, 2019

mvdan commented Jan 25, 2019

aalexand commented Jan 25, 2019

bradfitz commented Jan 25, 2019

aalexand commented Jan 25, 2019

thepudds commented Jan 25, 2019 • edited Loading

bcmills commented Jan 25, 2019

bcmills commented Jan 25, 2019

jeanbza commented Jan 25, 2019

gopherbot commented Feb 13, 2019

dmitshur commented Mar 1, 2019

jeanbza commented Mar 5, 2019

gopherbot commented Mar 13, 2019

gopherbot commented Mar 28, 2019

gopherbot commented Apr 16, 2019

bradfitz commented Apr 16, 2019

thepudds commented Apr 16, 2019 • edited Loading

thepudds commented Apr 17, 2019 • edited Loading

dmitshur commented Jun 4, 2019

bcmills commented Jun 5, 2019

dmitshur commented Jun 5, 2019

thepudds commented Sep 13, 2019

bradfitz commented Sep 13, 2019

dmitshur commented Sep 13, 2019 • edited Loading

mvdan commented Jan 25, 2019 •

edited

Loading

dmitshur commented Jan 25, 2019 •

edited

Loading

bcmills commented Jan 25, 2019 •

edited

Loading

bcmills commented Jan 25, 2019 •

edited

Loading

dmitshur commented Jan 25, 2019 •

edited

Loading

enocom commented Jan 25, 2019 •

edited

Loading

thepudds commented Jan 25, 2019 •

edited

Loading

thepudds commented Apr 16, 2019 •

edited

Loading

thepudds commented Apr 17, 2019 •

edited

Loading

dmitshur commented Sep 13, 2019 •

edited

Loading