Add ability to download/upload one or more resources to/from disk

[TerryHowe]

What is the version of your ORAS CLI

0.16.0

What would you like to be added?

Ability to download and upload one or more resources to and from disk.

• In air gapped environments, this may be the only way replicate a registry. The data would be saved to say a thumb drive which is physically moved into the air gapped environment and uploaded to a private registry.
• Sometimes it may be more convenient to dump all the data onto a local file and upload later rather than use the copy command.

Why is this needed for ORAS?

In order to mirror registries, we need the ability to download one or more resources from a registry to disk. We also need the ability to upload the data saved on disk to a registry.

• Manual backup and restore of registries
• Mirroring registries to for air gapped environments
• Mirroring registries for performance reasons
• Convenient way to promote artifacts from dev to staging to prod
• The destination of the data could be a directory (containing directories and files), a tgz file or a stream of tgz data. I think the gzip compressed tar file (tgz) would be the most convenient.
• The ability to to specify multiple resources to copy to a single disk output is needed.
• Need to be able to specify credentials for source or destination registry similar to ~/.docker/config.json
• Need to be able to specify TLS cert for source or destination registry.
• Obviously needs to be optimized not to contain the same blob more than once.
• Given a list of image:tags, the new uploaded registry images should have the same shasums

Are you willing to submit PRs to contribute to this feature?

• Yes, I am willing to implement it.

[FeynmanZhou]

Hi @TerryHowe ,

Thanks for the feature request. It looks like the OCI image layout #378 can cover the core ability to upload the data saved on disk to a registry as you described above.

ORAS supported specifying an authentication file when pulling/pushing files from/to the registry. Can you elaborate more on "Need to be able to specify credentials and TLS for source or destination registry"? Does it mean the ability to use certificates at path (.crt, .cert, .key) to connect to the destination registry and require HTTPS and verify certificates when interacting with the registry respectively?

[TerryHowe]

Credentials I would expect similar to the docker config.json format.

[TerryHowe]

I don't have any firm TLS cert requirements because there are a lot of options there, but need to support self signed certs and private CAs.

[qweeah]

Credentials I would expect similar to the docker config.json format.

oras reuses the same credential schema of docker config.json file

need to support self signed certs and private CAs.

--ca-file do support private CAs, and you can add self-signed cert to the CA file so self-singed certs are also supported.

[shizhMSFT]

With oras v1.0.0-rc.1, this feature request is fulfilled by using oras copy --to-oci-layout and oras copy --from-oci-layout where OCI layout supports multiple resources in a single folder.

[TerryHowe]

With oras v1.0.0-rc.1, this feature request is fulfilled by using oras copy --to-oci-layout and oras copy --from-oci-layout where OCI layout supports multiple resources in a single folder.

oras copy only copies one resource and it does not preserve repository name. If I want to create a mirror, particularly in air gapped environment, I want to download a long list of images and then sneaker net them to my air gapped environment and upload them to my private registry.

[qweeah]

oras copy ... does not preserve repository name.

The repository name can be preserved if you use the registry full FQDN as the layout path, e.g. below command will download the image to an OCI layout in ${pwd}/docker.io/library/hello-world.

oras cp docker.io/library/hello-world:latest --to-oci-layout docker.io/library/hello-world:latest --platform linux/amd64

The repository is in the relative path from docker.io to the oci root directory hello-world

The idea of batched uploading/downloading sounds interesting to me. Providing batched operations is a brand-new feature to oras CLI so it's better to start with a design discussion.

[shizhMSFT]

@TerryHowe Downloading multiple resources to local filesystem is as simple as saving the following bash script as oras_mirror_backup.sh

#!/bin/bash
for ref in $@; do
    echo ">>> Backing up $ref ..."
    oras cp $ref --to-oci-layout $ref
done

and then do

$ ./oras_mirror_backup.sh docker.io/library/hello-world:linux docker.io/library/busybox:latest
>>> Backing up docker.io/library/hello-world:linux ...
Copied [registry] docker.io/library/hello-world:linux => [oci-layout] docker.io/library/hello-world:linux
Digest: sha256:1b4338ade6d6d9eff53264cdc0f9caf1d717cf89dbfcb1cb13c57bc0a3c199a0
>>> Backing up docker.io/library/busybox:latest ...
Copied [registry] docker.io/library/busybox:latest => [oci-layout] docker.io/library/busybox:latest
Digest: sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c

As you may noticed, the blobs are not de-duplicated across the repositories. Here's an advanced script

#!/bin/bash

function extract_path {
    local ref="$1"
    if [[ "$ref" == *@* ]]; then
        echo "${ref%@*}"
    else
        echo "${ref%:*}"
    fi
}

# create a shared blobs folder and a repositories folder
mkdir -p blobs repositories
cd repositories

# start mirroring
for ref in $@; do
    echo ">>> Backing up $ref ..."
    path=$(extract_path $ref)
    if [[ ! -L $path/blobs ]]; then
        mkdir -p $path
        ln -rs ../blobs $path/blobs
    fi
    oras cp $ref --to-oci-layout $ref
done

If you run the above script, you will get

$ ./oras_mirror_backup.sh docker.io/library/hello-world:linux docker.io/library/busybox:latest
>>> Backing up docker.io/library/hello-world:linux ...
Copied [registry] docker.io/library/hello-world:linux => [oci-layout] docker.io/library/hello-world:linux
Digest: sha256:1b4338ade6d6d9eff53264cdc0f9caf1d717cf89dbfcb1cb13c57bc0a3c199a0
>>> Backing up docker.io/library/busybox:latest ...
Copied [registry] docker.io/library/busybox:latest => [oci-layout] docker.io/library/busybox:latest
Digest: sha256:7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c

Here's the final folder structure:

$ tree
.
├── blobs
│   └── sha256
│       ├── 04341b189be695acecd201a36cdf9dd99b8b0c338075000f555d5adf8e9c0547
│       ├── 069e43a261e5dd787655dbeba5eed96e40f4c9f80f024ecd5d2bd17aab357204
│       ├── 0af8c5262529b2acebe9e308296ea619f25b2b3b47c632f7ff154e931d18064c
│       ├── 0b41f34c76745eef3d807afe679bea40ad0d84c1f109578ff017c870f2137589
│       ├── 1b4338ade6d6d9eff53264cdc0f9caf1d717cf89dbfcb1cb13c57bc0a3c199a0
│       ├── 1c8bbeaff20b74c3918ae3da99db0f0d8563adb33fcb346592e2882d82c28ab5
│       ├── 1d57ab16f681953c15d7485bf3ee79a49c2838e5f9394c43e20e9accbb1a2b20
│       ├── 205dae5015e78dd8c4d302e3db4eb31576fac715b46d099fe09680ba28093a7a
│       ├── 2bc9dea49d1a226db134bce761bfa89dd456109555c3ee4c490db84ad48d53b0
│       ├── 2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54
│       ├── 33450689bfb495ed64ead935c9933f1d6b3e42fe369b8de9680cf4ff9d89ce5c
│       ├── 36d89aa75357c8f99e359f8cabc0aae667d47d8f25ed51cbe66e148e3a77e19c
│       ├── 3caa6dc69d0b73f21d29bfa75356395f2695a7abad34f010656740e90ddce399
│       ├── 40d0cfd0861719208ff9f7747ab3f97844eeca509df705db44a736df863b76af
│       ├── 432f982638b3aefab73cc58ab28f5c16e96fdb504e8c134fc58dff4bae8bf338
│       ├── 46331d942d6350436f64e614d75725f6de3bb5c63e266e236e04389820a234c4
│       ├── 46758452d3eef8cacb188405495d52d265f0c3a7580dfec51cb627c04c7bafc4
│       ├── 4c45e4bb3be9dbdfb27c09ac23c050b9e6eb4c16868287c8c31d34814008df80
│       ├── 4ff685e2bcafdab0d2a9b15cbfd9d28f5dfe69af97e3bb1987ed483b0abf5a99
│       ├── 5004e9d559e7a75f42249ddeca4d5764fa4db05592a7a9a641e4ac37cc619ba1
│       ├── 5e42fbc46b177f10319e8937dd39702e7891ce6d8a42d60c1b4f433f94200bd2
│       ├── 61fff98d5ca765a4351964c8f4b5fb1a0d2c48458026f5452a389eb52d146fe8
│       ├── 66ba00ad3de8677a3fa4bc4ea0fc46ebca0f14db46ca365e7f60833068dd0148
│       ├── 688cd001103a44dc582d4fdc4647517422c0be7942c1278b5bb748395265375d
│       ├── 7050e35b49f5e348c4809f5eff915842962cb813f32062d3bbdd35c750dd7d01
│       ├── 7066d68bd2f224dbb7c3332da105b1dac81a75b47a869602096c27b6a75a525c
│       ├── 77ed5ebc3d9d48581e8afcb75b4974978321bd74f018613483570fcd61a15de8
│       ├── 7b3ccabffc97de872a30dfd234fd972a66d247c8cfc69b0550f276481852627c
│       ├── 7b8b7289d0536a08eabdf71c20246e23f7116641db7e1d278592236ea4dcb30c
│       ├── 7ef0bcd6b4899cdcc5d2c4e97aba3e60a0153a3201c1a9c810ed915975f3833e
│       ├── 7f0d4fad461d1ac69488092b5914b5ec642133c0fb884539045de33fbcd2eadb
│       ├── 8f23e10f4610afdde9b856b9367742f1f5ded5c35e2aaa0630d3c5d9ebc2e4cf
│       ├── 907ca53d7e2947e849b839b1cd258c98fd3916c60f2e6e70c30edbf741ab6754
│       ├── 93f830f96e6d4290268f3d7adb078a66ddc24c23dddbd4899fd72a8041a5a1c8
│       ├── 95a526907ab34a09463a07cd768039ac815d433029f181a7731ef8dba3095bc4
│       ├── 98c9722322be649df94780d3fbe594fce7996234b259f27eac9428b84050c849
│       ├── 995efde2e81b21d1ea7066aa77a59298a62a9e9fbb4b77f36c189774ec9b1089
│       ├── 99ee43e96ff50e90c5753954d7ce2dfdbd7eb9711c1cd96de56d429cb628e343
│       ├── 9af22d424aada215bab8c43d48ba6c8e4ddae9018628ab2098f16520bfdcd6d8
│       ├── a22ab831b2b2565a624635af04e5f76b4554d9c84727bf7e6bc83306b3b339a9
│       ├── abaa813f94fdeebd3b8e6aeea861ab474a5c4724d16f1158755ff1e3a4fde8b0
│       ├── abc70fcc95b2f52b325d69cc5c259dd9babb40a9df152e88b286fada1d3248bd
│       ├── b203a35cab50f0416dfdb1b2260f83761cb82197544b9b7a2111eaa9c755dbe7
│       ├── b3593dab05491cdf5ee88c29bee36603c0df0bc34798eed5067f6e1335a9d391
│       ├── b49eda688ce8c1226b6d7e02969f22361a8874cfee14c603e98ad855f1267a94
│       ├── b836bb24a270b9cc935962d8228517fde0f16990e88893d935efcb1b14c0017a
│       ├── b921b04d0447ddcd82a9220d887cd146f6ef39e20a938ee5e19a90fc3323e030
│       ├── bbc6052697e5fdcd1b311e0b3f65189ffbe354cf8ae97e7a55d588e855097174
│       ├── c0218de6585df06a66d67b25237bdda42137c727c367373a32639710c7a9fa94
│       ├── c3505dfdb7a6ef524d17d0ee391749f94de950c43642e3286e06172577e184a3
│       ├── c7b6944911848ce39b44ed660d95fb54d69bbd531de724c7ce6fc9f743c0b861
│       ├── db6ea0cbfcdfe2e7fff3f36b40c2c6ac27933977d71317b30c1905675ec29349
│       ├── dde8e930c7b6a490f728e66292bc9bce42efc9bbb5278bae40e4f30f6e00fe8c
│       ├── df5477cea5582b0ae6a31de2d1c9bbacb506091f42a3b0fe77a209006f409fd8
│       ├── e954aa43bc3d58a30a967d36b0b0ebf408eea4b1283106d2ca553b0243858d6b
│       ├── eb11b1a194ff8e236a01eff392c4e1296a53b0fb4780d8b0382f7996a15d5392
│       ├── f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4
│       ├── f78e6840ded1aafb6c9f265f52c2fc7c0a990813ccf96702df84a7dcdbe48bea
│       └── feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
├── oras_mirror_backup.sh
└── repositories
    └── docker.io
        └── library
            ├── busybox
            │   ├── blobs -> ../../../../blobs
            │   ├── index.json
            │   ├── ingest
            │   └── oci-layout
            └── hello-world
                ├── blobs -> ../../../../blobs
                ├── index.json
                ├── ingest
                └── oci-layout

11 directories, 64 files

/cc @sajayantony @qweeah

[TerryHowe]

I was going to add optimization to this as a follow-up, just trying to keep it simple for now. This is supposed to be a POC.

[TerryHowe]

Sure, everyone could write their own shell script, but there are a lot of use cases for this. I'd much rather throw a tar ball in a bucket and tell a customer to oras import it and know it will work.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[shizhMSFT]

The discussion is continued in #1366