As a user I can mount blobs across repositories

[pulpbot]

Author: @ipanova (ipanova@redhat.com)

Redmine Issue: 9241, https://pulp.plan.io/issues/9241

---

Given that the user, who is currently pushing an image, has READ access to another repository that already contains the blob, he should be able to mount blobs across repositories. The ability, that registry may provide is beneficial to save on network speed.

Cross blob mount works in the following way:
For every repository that container blob X will be asked a token with mount permission. If the user has no READ permission to any of those repos it will fall back and create the blob upload and upload the blob bits.

In these logs the are already existing repos busybox, busybox1, busybox2 that share same blob. Repo ina/busybox3 is being pushed and it has same blob that is contained in busybox, busybox1 and busybox2.
It tries to mount blob against each repo, fails and falls back to the regular blob upload.

.0.0.1 - - [16/Aug/2021:14:37:08 +0000] "PUT /v2/ina/busybox3/blobs/uploads/5e6887df-78f7-4908-ab7d-1e03a97754fd?digest=sha256%3Aa9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045 HTTP/1.1" 201

.0.0.1 - - [16/Aug/2021:14:37:07 +0000] "PATCH /v2/ina/busybox3/blobs/uploads/5e6887df-78f7-4908-ab7d-1e03a97754fd HTTP/1.1" 202 
.0.0.1 - - [16/Aug/2021:14:37:07 +0000] "POST /v2/ina/busybox3/blobs/uploads/ HTTP/1.1" 202 

.0.0.1 - - [16/Aug/2021:14:37:06 +0000] "POST /v2/ina/busybox3/blobs/uploads/?from=busybox&mount=sha256%3Aa9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045 HTTP/1.1" 401 
ngo.request:WARNING: Unauthorized: /v2/ina/busybox3/blobs/uploads/
 /pulp/container/busybox/blobs/sha256:a9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045?validate_token=bb5a4a4fffc5418d6b8d8f735e49caf38bb4660cf68a4a6fca3bc6e196550e60:fb629a583d896ba3e1a4888a83153a733a85e677771c2fe5a27342>
.0.0.1 - - [16/Aug/2021:14:37:06 +0000] "HEAD /v2/busybox/blobs/sha256:a9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045 HTTP/1.1" 302 
.0.0.1 - ina [16/Aug/2021:14:37:06 +0000] "GET /token/?account=ina&scope=repository%3Aina%2Fbusybox3%3Apull%2Cpush&scope=repository%3Abusybox%3Apull&service=localhost%3A24817 HTTP/1.0" 200 
.0.0.1 - - [16/Aug/2021:14:37:06 +0000] "POST /v2/ina/busybox3/blobs/uploads/?from=busybox1&mount=sha256%3Aa9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045 HTTP/1.1" 401 
ngo.request:WARNING: Unauthorized: /v2/ina/busybox3/blobs/uploads/
 /pulp/container/busybox1/blobs/sha256:a9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045?validate_token=d38d17fd475a24e228ed9d0418b937d7011954227d42945d9f5c14cca4afa50c:23939ede17923e540b63991e7372e1abed2223a26497c0581531b>
.0.0.1 - - [16/Aug/2021:14:37:06 +0000] "HEAD /v2/busybox1/blobs/sha256:a9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045 HTTP/1.1" 302 
.0.0.1 - ina [16/Aug/2021:14:37:06 +0000] "GET /token/?account=ina&scope=repository%3Aina%2Fbusybox3%3Apull%2Cpush&scope=repository%3Abusybox1%3Apull&service=localhost%3A24817 HTTP/1.0" 200 
.0.0.1 - - [16/Aug/2021:14:37:06 +0000] "POST /v2/ina/busybox3/blobs/uploads/?from=busybox2&mount=sha256%3Aa9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045 HTTP/1.1" 401 
ngo.request:WARNING: Unauthorized: /v2/ina/busybox3/blobs/uploads/
 /pulp/container/busybox2/blobs/sha256:a9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045?validate_token=4148bac928c3c8641bf5a40839c4e082334fc5242a787f99117eb956d4d99906:59026249fe55842485fbecd279c8b98e33df47c168ecf02cd6df4>
.0.0.1 - - [16/Aug/2021:14:37:05 +0000] "HEAD /v2/busybox2/blobs/sha256:a9efad4c351baa14138b2ddca283c33974c1957b59981f77642ccf4398981045 HTTP/1.1" 302 
.0.0.1 - ina [16/Aug/2021:14:37:05 +0000] "GET /token/?account=ina&scope=repository%3Aina%2Fbusybox3%3Apull%2Cpush&scope=repository%3Abusybox2%3Apull&service=localhost%3A24817 HTTP/1.0" 200
.0.0.1 - - [16/Aug/2021:14:37:05 +0000] "HEAD /v2/ina/busybox3/blobs/sha256:5b8c72934dfc08c7d2bd707e93197550f06c0751023dabb3a045b723c5e7b373 HTTP/1.1" 404 
ngo.request:WARNING: Not Found: /v2/ina/busybox3/blobs/sha256:5b8c72934dfc08c7d2bd707e93197550f06c0751023dabb3a045b723c5e7b373
.0.0.1 - ina [16/Aug/2021:14:37:05 +0000] "GET /token/?account=ina&scope=repository%3Aina%2Fbusybox3%3Apull%2Cpush&service=localhost%3A24817 HTTP/1.0" 200 
.0.0.1 - - [16/Aug/2021:14:37:05 +0000] "GET /v2/ HTTP/1.1" 401 
ngo.request:WARNING: Unauthorized: /v2/

https://docs.docker.com/search/?q=Cross%20Repository%20Blob%20Mount

[pulpbot]

From: mgoddard (mgoddard)
Date: 2021-12-07T15:03:33Z

---

Would love to see this!

We have many images which are part of a hierarchy, and have many shared layers. Currently pushing all of these to pulp takes over 3 hours since we have to push all layers for each image.

[markgoddard]

Hooray!