Multi-arch builds are failing

[msau42]

We successfully built amd64, but ppc64le is failing:

 > [1/2] FROM gcr.io/distroless/static:latest:

failed to solve: rpc error: code = Unknown desc = failed to load cache key: no match for platform in manifest sha256:7d687a1d8809b89c60211ff08a9277ffbe76895907aedffac4d2400a4e1a2dc7: not found
+ docker buildx rm multiarchimage-buildertest
make: *** [release-tools/build.make:128: push-multiarch-csi-provisioner] Error 1
ERROR
ERROR: build step 0 "gcr.io/k8s-testimages/gcb-docker-gcloud:v20200421-a2bf5f8" failed: step exited with non-zero status: 2

[msau42]

https://storage.googleapis.com/kubernetes-jenkins/logs/post-external-provisioner-push-images/1312087665054584832/build-log.txt

[msau42]

This may be related to changes made for:
GoogleContainerTools/distroless#583

[namrata-ibm]

@msau42 It does seem to be breaking since above change. The latest gcr.io/distroless/static image shows only amd and arm manifest. Pulling this image gives same error on s390x machine as seen in the Prow job.
We would need to extend support for ppc64le+s390x in distroless image.

[msau42]

@namrata-ibm would you be able to help with adding support for those other platforms to the distroless build?

[namrata-ibm]

@msau42 Adding support to distroless would need some time to sort Bazel related support first. Will work on that as a long term solution in parallel.
However to get the builds passing again, can we make changes to Dockerfile to use gcr.io/distroless/static:latest-amd64 image as suggested here? I can raise PRs for that.
cc @pohly

[msau42]

Yes, the workaround sounds fine to me. I'm only concerned that we will stop getting updates to the image so we should actively work on adding support for the missing platforms

[stuartpb]

This workaround is no good - switching the build to amd64 just broke all the pods running on my arm64 nodes. (Or maybe it's a coincidence, but in any case, I'm getting exec format errors, and I'm pretty sure I wasn't before.)

[pohly]

@stuartpb can you be more specific? Which containers broke (name, image version, etc.) and which binary in them cannot be executed?

[stuartpb]

These were the images I saw exec format errors from - I believe it was in every container spawned by Rook using any one of them:

k8s.gcr.io/sig-storage/csi-provisioner:v1.6.0
k8s.gcr.io/sig-storage/csi-snapshotter:v2.1.1
k8s.gcr.io/sig-storage/csi-attacher:v2.2.0
k8s.gcr.io/sig-storage/csi-resizer:v0.4.0

[msau42]

We have not officially supported multi arch images for those versions.

Multi arch is supported starting with
provisioner 2.0
attacher 3.0
snapshotter 3.0
resizer 1.0

[stuartpb]

Maybe it's a coincidence, then, but I'm pretty sure I was using those images successfully for about a week at least before this change hit.

[dims]

@msau42 GoogleContainerTools/distroless#622

[msau42]

This should be addressed with the latest sidecar releases and head.

/lclose

[pohly]

/close

[k8s-ci-robot]

@pohly: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.