proposal: remote cache (stepstone to central repository)

[aaronkavlie-wf]

Background

Central package repositories are commonplace in package managers for other languages.
In Python (the language I'm most familiar with), packages rarely include code from other projects in their source tree.

Many Go developers would prefer to avoid checking in 3rd-party packages, to avoid the bloat and messy diffs that leads to. However, without vendored dependencies Go tools usually fetch packages from multiple independent hosts. This causes problems if projects move, hosts disappear (I'm looking at you, Google Code), repositories have downtime, etc. Central package repositories are typically much more reliable.

Solution

Following the lead of the caching functionality already in glide, I have a WIP implementation of a remote-cache option (name up for debate). This option would attempt to check out the package from a user-supplied URL first. Unlike a local cache it could not write to the cache, but it could supply the package name or URL (perhaps in the Referer header), so that a server could, in turn, check out (or pull updates) for the package so it's available for the next user.

Ideally you would be able to save this setting in a config file in your home directory (like pip) to avoid setting it on every usage.
It doesn't look like Glide has support for config files (discounting per-project glide.yaml files) yet, but this is pretty essential to make the feature usable.

[technosophos]

I am very interested in this idea. A few others in the community have expressed interest in something like this too.

[mattfarina]

I'm up for this and have been for some time. To link in other issues, #39 and #128 are related to this.

[aaronkavlie-wf]

I can put up my WIP code. Didn't seem too complicated to add, though I got tripped up a bit by the implementation details in repo/vcs.go.

Thinking about it more, I'm not sure that "remote cache" adequately describes this functionality. A cache is often temporary and ephemeral. It would act like a cache only in that you fall back to fetching as usual (and that could even be potentially disabled, for companies with strict requirements for 3rd-party packages).

[sdboyer]

yes, WIP code would be beneficial, though...

what amounts to an engine rewrite for glide over in vsolver would likely change the dynamics of how this might work. in the parlance of that system, it could probably be implemented as an alternate SourceManager (compare to the - still very much WIP - built-in, local-cache-oriented default)

[aaronkavlie-wf]

@sdboyer are @mattfarina & co planning to use vsolver for glide at some point, or is that speculative?

[sdboyer]

@aaronkavlie-wf yes, it's the formal plan. with any luck, i'll have a PR up sometime next week that gets things started.

[mattfarina]

This has been implemented. See the mirror command and subcommands.

In the future I would like to add patterns to the mirror setup. Now that the groundwork has been laid it should be easy. Any takers?