cmd/go: add -reuse flag [freeze exception] #53644
Labels
FrozenDueToAge
GoCommand
cmd/go
NeedsFix
The path to resolution is known, but the work has not been done.
release-blocker
Milestone
Comments
|
Change https://go.dev/cl/411397 mentions this issue: |
|
Change https://go.dev/cl/415678 mentions this issue: |
|
Change https://go.dev/cl/411398 mentions this issue: |
|
Bryan agrees with this request (personal communication and also a thumbs-up above) but is now on a brief vacation. I'm not sure what else needs to happen for formal approval, but I'm going to submit the CLs once I get the TryBots happy so as not to miss the Go 1.19 release candidate later this week. |
gopherbot
pushed a commit
that referenced
this issue
Jul 5, 2022
This change adds an "Origin" JSON key to the output of go list -json -m and go mod download -json. The associated value is a JSON object with metadata about the source control system. For Git, that metadata is sufficient to evaluate whether the remote server has changed in any interesting way that might invalidate the cached data. In most cases, it will not have, and a fetch could then avoid downloading a full repo from the server. This origin metadata is also now recorded in the .info file for a given module@version, for informational and debugging purposes. This change only adds the metadata. It does not use it to optimize away unnecessary git fetch operations. (That's the next change.) For #53644. Change-Id: I4a1712a2386d1d8ab4e02ffdf0f72ba75d556115 Reviewed-on: https://go-review.googlesource.com/c/go/+/411397 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
gopherbot
pushed a commit
that referenced
this issue
Jul 5, 2022
The go list -m and go mod download commands now have a -reuse flag, which is passed the name of a file containing the JSON output from a previous run of the same command. (It is up to the caller to ensure that flags such as -versions or -retracted, which affect the output, are consistent between the old and new run.) The new run uses the old JSON to evaluate whether the answer is unchanged since the old run. If so, it reuses that information, avoiding a costly 'git fetch', and sets a new Reuse: true field in its own JSON output. This dance with saving the JSON output and passing it back to -reuse is not necessary on most systems, because the go command caches version control checkouts in the module cache. That cache means that a new 'git fetch' would only download the commits that are new since the previous one (often none at all). The dance becomes important only on systems that do not preserve the module cache, for example by running 'go clean -modcache' aggressively or by running in some environment that starts with an empty file system. For #53644. Change-Id: I447960abf8055f83cc6dbc699a9fde9931130004 Reviewed-on: https://go-review.googlesource.com/c/go/+/411398 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
|
Does this address #44577? If so, we may want to unlock that thread and post a brief update when proxy.golang.org starts using |
dmitshur
changed the title
dmitshur
added
NeedsFix
|
Thanks for letting us know.
It's documented at https://go.dev/s/release#freeze-exceptions. I've retitled to fit the format (so this freeze exception is findable if we want to count them per release, etc.) and added CC @golang/release here.
I think that rationale is reasonable, it's done, and it's fine to consider it approved. |
jproberts
pushed a commit
to jproberts/go
that referenced
this issue
Aug 10, 2022
This change adds an "Origin" JSON key to the output of go list -json -m and go mod download -json. The associated value is a JSON object with metadata about the source control system. For Git, that metadata is sufficient to evaluate whether the remote server has changed in any interesting way that might invalidate the cached data. In most cases, it will not have, and a fetch could then avoid downloading a full repo from the server. This origin metadata is also now recorded in the .info file for a given module@version, for informational and debugging purposes. This change only adds the metadata. It does not use it to optimize away unnecessary git fetch operations. (That's the next change.) For golang#53644. Change-Id: I4a1712a2386d1d8ab4e02ffdf0f72ba75d556115 Reviewed-on: https://go-review.googlesource.com/c/go/+/411397 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
jproberts
pushed a commit
to jproberts/go
that referenced
this issue
Aug 10, 2022
The go list -m and go mod download commands now have a -reuse flag, which is passed the name of a file containing the JSON output from a previous run of the same command. (It is up to the caller to ensure that flags such as -versions or -retracted, which affect the output, are consistent between the old and new run.) The new run uses the old JSON to evaluate whether the answer is unchanged since the old run. If so, it reuses that information, avoiding a costly 'git fetch', and sets a new Reuse: true field in its own JSON output. This dance with saving the JSON output and passing it back to -reuse is not necessary on most systems, because the go command caches version control checkouts in the module cache. That cache means that a new 'git fetch' would only download the commits that are new since the previous one (often none at all). The dance becomes important only on systems that do not preserve the module cache, for example by running 'go clean -modcache' aggressively or by running in some environment that starts with an empty file system. For golang#53644. Change-Id: I447960abf8055f83cc6dbc699a9fde9931130004 Reviewed-on: https://go-review.googlesource.com/c/go/+/411398 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
jproberts
pushed a commit
to jproberts/go
that referenced
this issue
Aug 10, 2022
CL 411398 added the -reuse flag for reusing cached JSON output when the remote Git repository has not changed. One case that was not yet cached is a lookup of a nonexistent version. This CL adds caching of failed lookups of nonexistent versions, by saving a checksum of all the heads and tags refs on the remote server (we never consider other kinds of refs). If none of those have changed, then we don't need to download the full server. Fixes golang#53644. Change-Id: I428bbc8ec8475bd7d03788934d643e1e2be3add0 Reviewed-on: https://go-review.googlesource.com/c/go/+/415678 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
We have been working on a
-reuseflag for use by module proxies to reduce redundant 'git fetch' / 'git clone' traffic. The code paths only trigger when the new flag is used, so the chance of incidental problems should be quite low. There is some more provenance information in the .info files too, which may make people happy. We want to get it into Go 1.19 so that proxy.golang.org can start using it once Go 1.19 is out in August, rather than having to wait for Feb 2023 (Go 1.20).Filing for freeze exception.
CL 411397 and CL 411398.
/cc @bcmills
The text was updated successfully, but these errors were encountered: