how to build multi-arch image using kaniko

[jinchihe]

/question

Hello, this is a question, is that kaniko support build multi-arch images? how to build multi-arch image using kaniko? Any docs or best practice? Thanks a lot!

Refer to build multi-arch image by using Google Cloud Build: https://medium.com/@bamnet/building-multiarch-docker-images-8a70002b3476

Thank you again!

[tejal29]

Thank you for your question.
You can pass in docker args to kaniko --build-arg opts="CGO_ENABLED=0 GOARCH=amd64" in the kaniko pod spec https://github.com/GoogleContainerTools/kaniko#using-kaniko

You will have create one kaniko pod per architecture.

Does that help?

[jinchihe]

@tejal29 Thanks. That's crosee building you pointed out. If we want to support multi-arch image, I think kaniko need support to create minifest, please see here for details abour multi-arch image: https://lobradov.github.io/Building-docker-multiarch-images/

[tejal29]

@jinchihe I am not sure, what you mean by minifest.
I went through the doc and looks like the guidance is to create a 1 docker file per architecture.
Kaniko currently only supports 1 dockerfile per run.
You will have to create a build.sh which runs a kaniko pod instead
https://lobradov.github.io/Building-docker-multiarch-images/#same-dockerfile-template

I am going to close this for now. Please re-open if this does not answer your question.

[yangm97]

@tejal29 take a look at how buildkit is doing it. You can use a single Dockerfile for multiple architectures and then all images get pushed into the same tag. https://engineering.docker.com/2019/04/multi-arch-images/

[darewreck54]

@tejal29 should this be re-opened or point out if kaniko supports it or not.

[MatthiasLohr]

@tejal29, any news on that?

[tejal29]

currently we don't support building multi-arch docker images.
We love any contributions or designs for this feature.

[mickkael]

This can be achieved within a K8s cluster with Argo to orchestrate.
https://flavio.castelli.me/2020/10/05/build-multi-architecture-container-images-using-argo-workflow/
in this example, buildah can be replaced by kaniko for the image build, but I'm not sure for the manifest, need to check that.

[morganchristiansson]

#1102 is related

[ericbl]

any improvement here? Is it still not possible to build a multi arch image with Kaniko?
We could imagine allowing multiple target in the
--customPlatform
parameter, similar to
docker buildx build --platform "linux/amd64,linux/arm64"

At least, any additional info in the README would help.
(and avoid some confusion as I wrote in #1746 on the same question)

[imjasonh]

Multi-arch support in Kaniko is fundamentally difficult because of constraints in the way Kaniko works -- instead of starting containers for each step, the RUN directives in the Dockerfile are run inside Kaniko's own container, which is naturally running in only one arch. Kaniko's value comes from not needing to spawn new containers, but that's also what keeps it from making multi-arch builds easier.

It may be theoretically possible to change this, but it would be a large architectural undertaking in Kaniko, which is currently understaffed.

In the meantime, I suggest using docker buildx, buildah, etc., each of which have different tradeoffs in terms of required privilege, performance, and maintainer investment.

[gilbsgilbs]

Another workaround that probably works if one cannot afford to leave userspace:

0. Create all the desired images on native platforms and tag them with the architecture suffixed <version>-<arch> (e.g. v14-arm64, v14-amd64, etc…)
1. Once everything is tagged, use some software push a multi-arch manifest under the desired tag (e.g. v14). Turns out manifest-tool can do that natively: manifest-tool push from-args --platforms linux/amd64,linux/arm64 --template myimage:v14-ARCH --target myimage:v14)

The main downside I see is that you need some orchestration capability to 1) run kaniko on each target platform 2) likely in parallel 3) push the manifest afterwards. Yet I think building on native platforms should be faster than building within cross-arch VMs (like docker buildx and probably buildah do).

For actual cross-arch builds, Kaniko's container could embed qemu and execute kaniko within some VM. Not only this would be a “large architectural undertaking” as @imjasonh said, it would also be a somewhat questionable move IMO given this probably requires some extra capabilities on the container (at least for networking). However, I wonder to which extent Kaniko could pick up some of manifest-tool's job in my suggested workaround, for instance by implementing some --merge-multi-arch-manifest=<image:tag> option, similar to --push=<image:tag>, that'd upload a multi-arch manifest for the current architecture, creating if it doesn't exist or merging with any already existing multi-arch manifest if it exists. Yet this sounds very prone to races (both for kaniko and for the image user).

edit: related to #1102 (comment)

[fgreinacher]

Another workaround that probably works if one cannot afford to leave userspace:

0. Create all the desired images on native platforms and tag them with the architecture suffixed <version>-<arch> (e.g. v14-arm64, v14-amd64, etc…)

1. Once everything is tagged, use some software push a multi-arch manifest under the desired tag (e.g. v14). Turns out manifest-tool can do that natively: manifest-tool push from-args --platforms linux/amd64,linux/arm64 --template myimage:v14-ARCH --target myimage:v14)

That's what we went for, and it's pretty straightforward in GitLab CI: https://ingenuity.siemens.com/2022/07/building-a-multi-arch-container-image-in-unprivileged-containers/

[gsquared94]

Skaffold's latest beta-release launched support for building multi-arch images, and I have this working for Kaniko as well. You'll need to use a mixed node Kubernetes cluster containing nodes matching the build target architectures. Skaffold will iteratively run the Kaniko builder for each architecture against that architecture Pod, stitch it into a ManifestList and push the image.

For example, against this test kaniko project:

❯ skaffold build --default-repo gcr.io/k8s-skaffold --cache-artifacts=false --platform=linux/arm64,linux/amd64
Generating tags...
 - skaffold-example-sub -> gcr.io/k8s-skaffold/skaffold-example-sub:v2.0.0-beta2-16-g796ebe417
Starting build...
Checking for kaniko secret [default/e2esecret]...
WARN[0002] Assuming the secret e2esecret is mounted inside Kaniko pod with the filename kaniko-secret. If your secret is mounted at different path, please specify using config key `pullSecretPath`.
See https://skaffold.dev/docs/references/yaml/#build-cluster-pullSecretPath  subtask=-1 task=Build
Building [skaffold-example-sub]...
Target platforms: [linux/arm64,linux/amd64]
INFO[0000] Resolved base name golang:1.15-alpine to builder 
INFO[0000] Retrieving image manifest golang:1.15-alpine 
INFO[0000] Retrieving image golang:1.15-alpine from registry index.docker.io 
INFO[0000] Retrieving image manifest alpine:3           
INFO[0000] Retrieving image alpine:3 from registry index.docker.io 
INFO[0001] Built cross stage deps: map[0:[/app]]        
INFO[0001] Retrieving image manifest golang:1.15-alpine 
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'golang:1.15-alpine' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd COPY main.go . requires it. 
INFO[0006] COPY main.go .                               
INFO[0006] Taking snapshot of files...                  
INFO[0006] RUN go build -o /app main.go                 
INFO[0006] Initializing snapshotter ...                 
INFO[0006] Taking snapshot of full filesystem...        
INFO[0008] Cmd: /bin/sh                                 
INFO[0008] Args: [-c go build -o /app main.go]          
INFO[0008] Running: [/bin/sh -c go build -o /app main.go] 
INFO[0009] Taking snapshot of full filesystem...        
INFO[0009] Saving file app for later use                
INFO[0009] Deleting filesystem...                       
INFO[0010] Retrieving image manifest alpine:3           
INFO[0010] Returning cached image manifest              
INFO[0010] Executing 0 build triggers                   
INFO[0010] Building stage 'alpine:3' [idx: '1', base-idx: '-1'] 
INFO[0010] Unpacking rootfs as cmd COPY --from=builder /app . requires it. 
INFO[0010] CMD ["./app"]                                
INFO[0010] COPY --from=builder /app .                   
INFO[0010] Taking snapshot of files...                  
INFO[0010] Pushing image to gcr.io/k8s-skaffold/skaffold-example-sub:v2.0.0-beta2-16-g796ebe417_linux_arm64 
INFO[0012] Pushed gcr.io/k8s-skaffold/skaffold-example-sub@sha256:ee161ccaf7303116d0e2455fc3b71c11554ec12d545bd650c7e7db8f0257ccca 
INFO[0000] Resolved base name golang:1.15-alpine to builder 
INFO[0000] Retrieving image manifest golang:1.15-alpine 
INFO[0000] Retrieving image golang:1.15-alpine from registry index.docker.io 
INFO[0000] Retrieving image manifest alpine:3           
INFO[0000] Retrieving image alpine:3 from registry index.docker.io 
INFO[0001] Built cross stage deps: map[0:[/app]]        
INFO[0001] Retrieving image manifest golang:1.15-alpine 
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'golang:1.15-alpine' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd COPY main.go . requires it. 
INFO[0006] COPY main.go .                               
INFO[0006] Taking snapshot of files...                  
INFO[0006] RUN go build -o /app main.go                 
INFO[0006] Initializing snapshotter ...                 
INFO[0006] Taking snapshot of full filesystem...        
INFO[0010] Cmd: /bin/sh                                 
INFO[0010] Args: [-c go build -o /app main.go]          
INFO[0010] Running: [/bin/sh -c go build -o /app main.go] 
INFO[0010] Taking snapshot of full filesystem...        
INFO[0010] Saving file app for later use                
INFO[0010] Deleting filesystem...                       
INFO[0010] Retrieving image manifest alpine:3           
INFO[0010] Returning cached image manifest              
INFO[0010] Executing 0 build triggers                   
INFO[0010] Building stage 'alpine:3' [idx: '1', base-idx: '-1'] 
INFO[0010] Unpacking rootfs as cmd COPY --from=builder /app . requires it. 
INFO[0011] CMD ["./app"]                                
INFO[0011] COPY --from=builder /app .                   
INFO[0011] Taking snapshot of files...                  
INFO[0011] Pushing image to gcr.io/k8s-skaffold/skaffold-example-sub:v2.0.0-beta2-16-g796ebe417_linux_amd64 
INFO[0012] Pushed gcr.io/k8s-skaffold/skaffold-example-sub@sha256:4c0efe6eada6276e6afe51204132f1923810155bb1925316d6e1da96e69ce8e8 
Build [skaffold-example-sub] succeeded

❯ docker manifest inspect gcr.io/k8s-skaffold/skaffold-example-sub:v2.0.0-beta2-16-g796ebe417
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 591,
         "digest": "sha256:ee161ccaf7303116d0e2455fc3b71c11554ec12d545bd650c7e7db8f0257ccca",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 591,
         "digest": "sha256:4c0efe6eada6276e6afe51204132f1923810155bb1925316d6e1da96e69ce8e8",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

[ozbillwang]

@gsquared94

Thanks for the confirmation.

can I get full supported platform list?

--platform=linux/arm64,linux/amd64

Any other platforms as well?