Support for pull-through cached mirroring #19
Comments
2 cents... |
|
@dmp42 Regarding item 2, above, by ACLs, I meant do we provide control over which names would actually be forwarded? For example, one may not want to ever forward requests for "mydomain.com/" but allow "library/" to be forwarded. However, local credential configuration for remote registry makes a lot of sense. |
|
@dmp42 It seems like we should employ conditional http requests to the remote for manifest caching. I'll update the description above to capture this. |
|
cc @samalba |
|
Sorry to be late on this... I agree the credentials should not be sent upstream, the Mirror should be anonymous (at least for short term). However it's important to have some identification (like a modified User-Agent?) to be able to identify that the request is coming from a Mirror. I would not make this configurable for the sake of simplicity. About the TTL, not configurable either. It would be set from the cache TTL provided upstream (Cache-Control). In this case. |
|
Interesting feedback on this: the TTL (even for long-lived resources) has to be set to a relatively short value (eg: < 2 weeks), otherwise from a legal POV the mirror is no longer considered "caching" but "hosting". |
|
@dmp42 The main use case that this proposal covers is for an on-premise, private registry. Users can push and pull content to the "local" storage. If content is not found during a pull, the data can be requested from upstream. If the upstream content is found, it is "cached" locally. It's a cache, whether there is a TTL or not. A cache is a place to store things conveniently. A TTL is merely one technique to manage the a cache. Conversely, putting a TTL on a datastore does not make it a "cache". What are the ramifications of "hosting" vs "caching" in relation to docker images? |
|
@stevvooe I will not speak to you about that without my lawyer being present :-) |
|
+1 This is "must have" for me! |
|
+1 |
|
+1 |
|
@stevvooe what I would love to see is the registry mirror being able to store a push and make it available for use while still pushing it to the hub. I'm in a situation currently where it's taking a significant amount of time to push images to the hub and need the speed of pushing/pulling to/from a local registry, but still want the images to make their way to the Docker Hub. |
|
@jalmansor That use case is outside the scope of the proxy caching feature. Once the feature is available, it should be trivial to setup such a system. Simply setup the registry as a proxy cache and push your image. It will then be available locally. Have another process come by and push that to the hub, separately. As long as they have the same name, the caching will work correctly. |
|
An important part of this is to allow the Docker client to have configured fallback mirrors -- once I have a "central" registry and "regional" mirrors, I want to configure fallback to central registry if a regional mirror should fail. |
|
@stevenschlansker definitely an interesting idea. Maybe this can be worked on as a second step? At first, it can act like a single reverse proxy cache, where failure handling should be done at the proxy level with proper load balancing etc. |
|
Especially once you are geographically diverse, client-side fallback will be much easier to configure and administer than server-side. We already have to manage our Docker daemon configuration across all our machines. We do not have a way to set up HA spanning, say, US west coast and London. |
|
@stevenschlansker Agreed this is a valid model but its outside the scope of this particular feature. This proposal is for proxy-caching, which is different from intelligent mirror selection. If you're interested in this kind of support, please see the proposal on namespaces, issue #303. We are focused on making the client smarter and more configurable to handle the kinds of topologies you describe. |
|
Is it possible right now to do mirroring like this on docker-distribution? |
|
@noisy not yet - but it's coming. |
|
Looking forward to this feature! I noticed the patch was merged. When can we use it? @RichardScothern @dmp42 |
|
The intention is for 1.7. Watch this PR for updates @tobegit3hub |
|
Remarkable feature! Is it good to go? @RichardScothern |
|
Saw it. But it's kinda heavy for me to read through so i just picked up am easy patch from here. |
|
Fair enough @dalanlan There is a markdown documentation file in the pull request that should answer your questions. Please comment there if its not clear. |
|
Merged |
Use docker client in the integration tests Image-registry-commit: 0beb7ee80381f3ebc49aa17e3a9a4391e0f55934
Use docker client in the integration tests Image-registry-commit: 0beb7ee80381f3ebc49aa17e3a9a4391e0f55934
The registry should be able to operate as a pull-through mirroring cache. This means that if a pull operation cannot proceed locally due to missing content, the registry should be able to defer the request to an upstream registry.
Please see docker-archive/docker-registry#658 for some background information on this issue.
Pull
When pulling a manifest, if the content is available locally, it will be served as is. Optionally, the local may check the remote if the content has been updated, using conditional http requests and update the local content. If it is not available locally, the request should be forwarded to the remote registry. If the remote request proceeds, the manifest should be stored locally then served in response to the local request.
When pulling a blob, if the content is available locally, serve it as is. If not available locally, forward the request to the remote registry. If the data is directly available, the data should be forwarded to the client and stored locally, concurrently. If the remote issues a redirect, the local registry should download the data into the local cache and serve the data directly.
Push
All push operations are only attempted on the "local" registry. If they fail, they will not be forwarded to the remote registry.
Open Questions
The text was updated successfully, but these errors were encountered: