multi-arch-image-mgmt.md

Multi-arch image management with ORAS

Note

The version of this specification is 1.3.0 Beta.2. It is subject to change until ORAS v1.3.0 is officially released.

Overview

A multi-architecture (multi-arch) image is a type of container image that may combine variants for different architectures, and sometimes for different operating systems. In general, multi-arch images are commonly used in IoT and Edge computing, particularly heterogeneous deployments, as well as cross-platform environments.

When running an image with multi-architecture support, container clients will automatically select an image variant that matches your OS and architecture. Multi-arch images have been widely used in industry.

There are two formats of implementation in industry to create a multi-arch image:

• Docker manifest list: a manifest list is created from images that are identical in function for different OS/Arch combinations, e.g. docker.io/bitnami/kubectl:1.31. Note that there are some limitations to use Docker manifest list as articulated in the problems statement section below.
• OCI image index: a higher-level manifest which points to specific image manifests, ideal for one or more platforms, e.g. ghcr.io/oras-project/oras:v1.2.0.

As more and more container registries are fully compliant with the OCI specifications, OCI image index becomes a popular format to create a multi-arch image.

This document aims to elaborate on the scenarios and problems of creating and managing multi-arch images, propose a solution to resolve these problems and help users to effectively create and manage multi-arch images with ORAS.

Target users

For users who need to create, store, update, and push multi-arch images locally and in OCI registries. The primary user persona includes cloud-native developers, DevOps engineers, and security enigineers.

Problem Statement

The oras manifest provides subcommands to push, fetch, and delete an image manifest, but it doesn’t support composing an image index for a multi-arch image. This causes some problems and limitations to users as articulated below when creating multi-arch images.

Problem statement

Specifically, there are limitations and problems to create and manage multi-arch images using docker or docker buildx:

Rely on remote registry to create a multi-arch image

docker manifest and docker buildx imagetools create requires the separate arch-specific images (with different architectures/variants) to be pushed to a registry before creating an image index or manifest list. Normally, developers want to create a multi-arch image locally so that they can test in a development environment before publishing to the registry. This is not efficient for local development and testing. Moreover, relying on a remote registry to create a multi-arch image is not acceptable in an air-gapped environment.

In addition, docker buildx relies on a remote builder for the target architectures which do not match the build machine. Using a remote builder is not convenient and secure enough for testing changes.

Hard to compose and update a multi-arch image

To create a multi-arch image, ORAS users have to manually compose an OCI image index from one or several manifest JSON files, then use "oras manifest push" to push it to an OCI layout or a registry. Typical examples can be found from these two scripts:

• ORAS user A: POC: Create and install Postgres Trunk Binaries (github.com) and his blog post POC: Distributing Trunk Binaries via OCI (justatheory.com)
• ORAS user B: Script to help create a simple multi-arch artifact (github.com)

docker buildx/docker is unavailable

Users are not able to install or use docker buildx or even no docker and its daemon service in some strict environments. Users need to seek for an alternative tool to create a multi-arch image without relying on docker or docker buildx.

Scenarios

Create a multi-arch image in local filesystem

Bob needs to create a multi-arch image in an air-gapped environment with strict isolation, meaning all resources must reside in the local filesystem, with no access to a container registry. Each architecture-specific image manifest has already been created and stored locally. Bob wants to use the oras client to assemble a multi-arch image by creating an OCI image index that references these existing arch-specific images. Once the multi-arch image is assembled, Bob will verify and test it locally to ensure correctness.

Create a multi-arch image from local and registry

The user Bob wants to create a multi-arch image by combining architecture-specific images stored in both a local filesystem and a remote registry. Bob has already created some arch-specific image manifests and stored them locally, while others exist in a container registry. He wants to use the oras client to create a multi-arch image using an OCI image index, incorporating both local and remote images. Once created, Bob will push the final multi-arch image to a remote registry for distribution and testing.

Create a multi-arch image from registry only

Bob wants to create a multi-arch image using arch-specific images stored in a remote container registry. Each architecture-specific image has already been pushed to the registry. Bob aims to use the oras client to assemble these remote images into a multi-arch image using an OCI image index, without downloading them to the local filesystem. After successfully creating the multi-arch image, Bob will push the final multi-arch to the registry for further use.

Update an existing multi-arch image

Bob needs to update an existing multi-arch image by replacing or adding architecture-specific images. The current multi-arch image is stored either in a local filesystem or a remote registry. Bob has created new arch-specific images that need to be incorporated into the existing OCI image index. Using the oras client, he plans to:

• Retrieve the existing multi-arch image to local
• Modify the image index: He will replace outdated architecture-specific images with new ones or append additional architectures as needed.
• Push the updated multi-arch image: Once modified, he will save the updated image index locally or push it to a registry for further use.
• Verify the update: Finally, Bob will inspect the updated image to confirm that the changes were applied successfully.

Bob expects oras client to streamline the updating process above. This ensures that Bob can efficiently manage multi-arch images while maintaining version consistency.

Annotate a multi-arch image

This scenario will be included in a separate proposal doc.

Proposals

Ideally, if ORAS extends the ability to create and manage a multi-arch image from either local environment or remote registry, the problems listed above can be resolved. Creating and managing a multi-arch image using image index format should be as easy as playing Legos.

The proposed CLI commands for managing a multi-arch image are listed below. The detailed use cases and subcommands are articulated in the CLI Spec section.

• Create a multi-arch image: oras manifest index create
  • Add annotations to a multi-arch image during creation: oras manifest index create --annotation
• Update a multi-arch image: oras manifest index update

The proposal creates a multi-arch image using an OCI image index in an OCI image layout as a local storage, then push the multi-arch image to the registry with ORAS.

Desired experience

Create a multi-arch image in local filesystem

Here is the sample workflow to create a multi-arch image using an image index locally and push it to the registry for deployment:

1. Assume there are two arch-specific images tagged as v1-linux-amd64 and v1-linux-arm64 in an OCI image layout called layout-dir. List the tags in the OCI image layout:

$ oras repo tags --oci-layout layout-dir 

v1-linux-amd64 
v1-linux-arm64
v1-linux-armv7

2. Create a multi-arch image by combining two image manifests into an image index, tag it with v1 and push the tagged image index to an OCI image layout layout-dir automatically. Add annotations to this image index at the same time:

$ oras manifest index create v1-linux-amd64 v1-linux-arm64 --annotation "com.example.key=value" --oci-layout layout-dir:v1 

Fetching  v1-linux-amd64 
Fetched   sha256:42c524c48e0672568dbd2842d3a0cb34a415347145ee9fe1c8abaf65e7455b46 v1-linux-amd64 
Fetching  v1-linux-arm64
Fetched   sha256:965945e1a08031a63d5970c1da7c39af231c36e4c0a5a3cc276d02a3e06513ee v1-linux-arm64 
Packed    edb5bc1f0b5c application/vnd.oci.image.index.v1+json
Pushed    [oci-layout] layout-dir:v1
Digest: sha256:edb5bc1f0b5c21e9321b34e50c92beae739250fb88409056e8719d9759f6b5b4

Status: An image index has been created and pushed to layout-dir:v1

3. View the image index in the OCI image layout in a pretty JSON output:

$ oras manifest fetch --oci-layout layout-dir:v1 --pretty  

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:42c524c48e0672568dbd2842d3a0cb34a415347145ee9fe1c8abaf65e7455b46",
      "size": 1239,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:965945e1a08031a63d5970c1da7c39af231c36e4c0a5a3cc276d02a3e06513ee",
      "size": 1239,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    }
  ],
  "annotations": {
    "com.example.key": "value"
  }
}

Create a multi-arch image from local and registry

Create a multi-arch image by combining two architecture-specific images stored in a local filesystem and a remote registry respectively, tag it with v1 and push the tagged image index to an OCI image layout layout-dir automatically:

$ oras manifest index create v1-linux-amd64 localhost:5000/v1-linux-arm64 --oci-layout layout-dir:v1 

Fetching  v1-linux-amd64 
Fetched   sha256:42c524c48e0672568dbd2842d3a0cb34a415347145ee9fe1c8abaf65e7455b46 v1-linux-amd64 
Fetching  v1-linux-arm64
Fetched   sha256:965945e1a08031a63d5970c1da7c39af231c36e4c0a5a3cc276d02a3e06513ee v1-linux-arm64 
Packed    edb5bc1f0b5c application/vnd.oci.image.index.v1+json
Pushed    [oci-layout] layout-dir:v1
Digest: sha256:edb5bc1f0b5c21e9321b34e50c92beae739250fb88409056e8719d9759f6b5b4

Status: An image index has been created and pushed to layout-dir:v1

Create a multi-arch image from a registry only

Create a multi-arch image using two arch-specific images stored in a registry only, tag it with v1 and push the tagged image index to an OCI image layout layout-dir automatically.

$ oras manifest index create localhost:5000/v1-linux-arm64 localhost:5000/v1-linux-arm64 --oci-layout layout-dir:v1 

Fetching  v1-linux-amd64 
Fetched   sha256:42c524c48e0672568dbd2842d3a0cb34a415347145ee9fe1c8abaf65e7455b46 v1-linux-amd64 
Fetching  v1-linux-arm64
Fetched   sha256:965945e1a08031a63d5970c1da7c39af231c36e4c0a5a3cc276d02a3e06513ee v1-linux-arm64 
Packed    edb5bc1f0b5c application/vnd.oci.image.index.v1+json
Pushed    [oci-layout] layout-dir:v1
Digest: sha256:edb5bc1f0b5c21e9321b34e50c92beae739250fb88409056e8719d9759f6b5b4

Status: An image index has been created and pushed to layout-dir:v1

Update an axisting multi-Arch image

Update the image index by adding a new architecture from a registry in the OCI image layout:

$ oras manifest index update --add localhost:5000/v1-linux-arm64 --oci-layout layout-dir:v1

Fetching  v1
Fetched   sha256:edb5bc1f0b5c21e9321b34e50c92beae739250fb88409056e8719d9759f6b5b4 v1
Fetching  v1-linux-armv7
Fetched   sha256:965945e1a08031a63d5970c1da7c39af231c36e4c0a5a3cc276d02a3e06513ee v1-linux-armv7
Added     sha256:965945e1a08031a63d5970c1da7c39af231c36e4c0a5a3cc276d02a3e06513ee v1-linux-armv7
Updated   sha256:6a165dbdc7a24e677e7ec0748457604ba143ae74e5b27a19789b88b41bf49bb0
Pushed    [oci-layout] layout-dir:v1
Digest: sha256:6a165dbdc7a24e677e7ec0748457604ba143ae74e5b27a19789b88b41bf49bb0

CLI Specs for new subcommands

Create a multi-arch image

# Create an index from source manifests tagged 'linux-amd64' and 'linux-arm64', and push without tagging:
oras manifest index create localhost:5000/hello linux-amd64 linux-arm64

# Create an index from source manifests tagged 'linux-amd64' and 'linux-arm64', and push with the tag 'v1':
oras manifest index create localhost:5000/hello:v1 linux-amd64 linux-arm64

# Create an index from source manifests using both tags and digests, and push with tag 'v1':
oras manifest index create localhost:5000/hello:v1 linux-amd64 sha256:99e4703fbf30916f549cd6bfa9cdbab614b5392fbe64fdee971359a77073cdf9

# Create an index and push it with multiple tags:
oras manifest index create localhost:5000/hello:tag1,tag2,tag3 linux-amd64 linux-arm64 sha256:99e4703fbf30916f549cd6bfa9cdbab614b5392fbe64fdee971359a77073cdf9

# Create and push an index with annotations:
oras manifest index create localhost:5000/hello:v1 linux-amd64 --annotation "key=val"

# Create an index and push to an OCI image layout folder 'layout-dir' and tag with 'v1':
oras manifest index create layout-dir:v1 linux-amd64 sha256:99e4703fbf30916f549cd6bfa9cdbab614b5392fbe64fdee971359a77073cdf9

# Create an index and save it locally to index.json, auto push will be disabled:
oras manifest index create --output index.json localhost:5000/hello linux-amd64 linux-arm64

# Create an index and output the index to stdout, auto push will be disabled:
oras manifest index create localhost:5000/hello linux-arm64 --output - --pretty

Update a multi-arch image

# Remove a manifest and add two manifests from an index tagged 'v1'. The tag will point to the updated index:
oras manifest index update localhost:5000/hello:v1 --add linux-amd64 --add linux-arm64 --remove sha256:99e4703fbf30916f549cd6bfa9cdbab614b5392fbe64fdee971359a77073cdf9

# Create a new index by updating an existing index specified by its digest:
oras manifest index update localhost:5000/hello@sha256:99e4703fbf30916f549cd6bfa9cdbab614b5392fbe64fdee971359a77073cdf9 --add linux-amd64 --remove sha256:fd6ed2f36b5465244d5dc86cb4e7df0ab8a9d24adc57825099f522fe009a22bb

# Merge manifests from the index 'v2-windows' to the index 'v2':
oras manifest index update localhost:5000/hello:v2 --merge v2-windows

# Update an index and tag the updated index as 'v2.1.0' and 'v2':
oras manifest index update localhost:5000/hello@sha256:99e4703fbf30916f549cd6bfa9cdbab614b5392fbe64fdee971359a77073cdf9 --add linux-amd64 --tag "v2.1.0" --tag "v2"

# Update an index and save it locally to index.json, auto push will be disabled:
oras manifest index update --output index.json localhost:5000/hello:v2 --add v2-linux-amd64

# Update an index and output the index to stdout, auto push will be disabled:
oras manifest index update --output - --pretty localhost:5000/hello:v2 --remove sha256:99e4703fbf30916f549cd6bfa9cdbab614b5392fbe64fdee971359a77073cdf9

View a multi-arch image

To make the inspection operation more intuitive to users, add one alias oras manifest show to the existing command oras manifest fetch:

Usage:
  oras manifest fetch [flags] <name>{:<tag>|@<digest>}

Aliases:
  fetch, get, show

Investigation on other client tools and industry

Most of popular container client tools support create and push a multi-arch image using docker manifest list or OCI image index format, but these tools require users to push platform-specific image push to the registry separately. They don’t provide native support for local environment.

• docker:
  • docker buildx build
  • docker buildx imagetool
  • docker manifest create
• podman (Backed by Red Hat): similar with docker manifest, it provides podman manifest with subcommands to create and manipulate manifest lists and image indexes.
• buildah: supports building a multi-platform image to local filesystem.
• crane(Backed by Google): provides a single command crane index append to compose an image index. crane index filter supports filters the image index to include only platforms that are relevant to you.
• regctl (Individual) provides regctl index add/create/delete to creates or manages OCI image index and manifest list, regctl artifact put --index supports pushing the artifact and add it to an index in a single operation. It also provides native support for local environments in all commands with the OCI Layout.
• manifest-tool (Individual from AWS): create docker manifest list or OCI image index in a registry by using the manifest-tool push command with either a YAML file describing the images to assemble or by using a series of parameters.
• skopeo (Backed by Red Hat): Skopeo can consume and copy image index or manifest list, but it doesn't support generating or modifying them.

In addition, docker buildx supports building a multi-arch image using the OCI image index format. Even Docker Official Images are using the OCI image index format to create multi-platform images. Homebrew publishes all images on ghcr.io using OCI image index.

Follow-up enhancements in the future releases

• Support showing platform information of tags in formatted output: oras-project#1547
• Support attach annotations to image index and its child image manifest in oras attach: oras-project#1531
• Add status output to show the command execution result: oras-project#1575
• Add two alias to "oras manifest fetch": oras-project#1574