[FR] Multi architecture build with only Kaniko

[ericbl]

In the issue #1474 and #1531 a multi arch support was added by using Docker BuildX and QEMU.
In our team, the main motivation to use kaniko is to get rid of direct docker commands.

Is there any better plan to build multi architecture image WITHOUT any BuildX use?

In our teams, we have pipelines using the plain docker BuildX without Kaniko for one single image name for 2 architectures.
and others using Kaniko to build separate images: one for AMD64, the other for ARM64.
(with e.g. --build-arg opts='GOARCH=arm64' --customPlatform=linux/arm64/v8 to build ARM64 from an Intel runner, as accepted as a solution of the issue #1587 )
btw, the Readme concerning the customPlatform is a bit confusing: it does work for ARM image on Intel runner!

Kaniko with one image for both architecture would be the best of both world.

Any input, suggestion and roadmap would be highly appreciated.

[mickkael]

About the custom platform, it may or may not work on Intel. It works for you as you have Go code, which can be cross compiled. It would not work in some other use cases.
For the multplatform image, I use argo workflows to build amd64, then arm64 and finally build the multi-manifest using
mplatform/manifest-tool

The only difference with dockerx is that you would have 3 images/tag in the repo. arm64, amd64, and the multi (let's say latest)

[ericbl]

I am still confused. Who has a running example with Kaniko multi arch used to build both AMD64 and ARM64 images?

Could the Kaniko team please update the doc with such an exemple?
Or cleary set in the documentation (Readme.md) how the multi arch can be used.

[lodotek]

bump

[Kovah]

We are confused by the docs too. Theoretically, Docker can build multiple platforms with buildx, independent from the host. Does Kaniko use buildx or not?

[autarchprinceps]

For anybody like me finding this thread, I think I got it to work by calling kaniko executor with --customPlatform once per platform and combining them with docker manifest. My kaniko host is just amd64, but I got an image that ran as arm64 on my M1 macbook, and that manifest inspect would show as a list with both architectures.

docker manifest create \
your-username/multiarch-example:manifest-latest \
--amend your-username/multiarch-example:manifest-amd64 \
--amend your-username/multiarch-example:manifest-arm64
docker manifest push your-username/multiarch-example:manifest-latest

Correction: That only works with either pure docker commands, or a native worker for the kaniko executor. Kaniko is not capable of executing non-native code emulated in the way buildx is. But if you have a Kubernetes with nodes of both types, you can build one image on each and manifest it together, so at least then this helps.

[aaron-prindle]

Related:
#2306

Recently some documentation was added in Kaniko's README.md regarding how to create multi-arch images with Kaniko with the caveat being that this is done by stitching N kaniko runs/images together into a final image (similar to what @autarchprinceps mentions in the comment above). Link here:
https://github.com/GoogleContainerTools/kaniko#creating-multi-arch-container-manifests-using-kaniko-and-manifest-tool

I will re-title this issue to be around the feature-request to add a way for Kaniko to do this on its own

[skyforce77]

Related to this issue, I'm wondering if it's currently even possible to come up with a solution for multi stage builds.
Simple images can easily get built using custom-platform and build-arg, but I have some trouble trying to get a stage running on host architecture and then building image for the target

Example working using docker build, but that's coming up with exec format error with kaniko running on amd64 host, to build arm64 target (Which I think shouldn't have any trouble, since it shouldn't even run target platform binaries)

## Cross compiled image base ##
# Build from host platform image
FROM --platform=${BUILDPLATFORM:-linux/amd64} docker.io/library/golang:1.22.1-bookworm as base
WORKDIR /app
COPY go.mod ./
RUN go mod download
COPY *.go ./
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -o /hello-world

# Target infrastructure packaging
FROM --platform=${TARGETPLATFORM:-linux/amd64} scratch
COPY --from=base /hello-world .
CMD ["/hello-world"]

My build is also using these generated arguments to try keeping things compatible with docker

/kaniko/executor --context tests/hello-world --dockerfile tests/hello-world/Dockerfile --cache=true --cache-ttl=24h --cache-repo=xxx/hello-world --destination xxx/hello-world:xxx-arm64 --tar-path hello-world-arm64.tar --no-push --custom-platform=linux/arm64 --build-arg='TARGETOS=linux' --build-arg='TARGETARCH=arm64' --build-arg='TARGETPLATFORM=linux/arm64' --build-arg='BUILDOS=linux' --build-arg='BUILDARCH=amd64' --build-arg='BUILDPLATFORM=linux/amd64'

So well, just to say, it's currently really tricky to get things done and I would be definitely happy to see a proper solution to build multi arch images, since my environment currently can't host arm64 servers

[PigeonF]

Related to this issue, I'm wondering if it's currently even possible to come up with a solution for multi stage builds.
Simple images can easily get built using custom-platform and build-arg, but I have some trouble trying to get a stage running on host architecture and then building image for the target

Looking at

kaniko/pkg/image/image_util.go

Lines 90 to 92 in 9c08f1c

	// Otherwise, initialize image as usual
	return RetrieveRemoteImage(currentBaseName, opts.RegistryOptions, opts.CustomPlatform)
	}

it seems that FROM --platform is not supported (since it always passes the customPlatform CLI arg, which defaults to the current platform, so I don't think what you are trying to achieve is possible unless kaniko gains support for reading the --platform parameter.