Error importing kube-poroxy.tar for ARM64 with containerd

[hakman]

What happened:

Since switching the kOps for ARM64 periodic tests to containerd container runtime, they are failing, not being able to create the cluster:
https://testgrid.k8s.io/kops-misc#kops-aws-arm64-conformance

kOps logs from these tests show that there are issues importing the kube-proxy.image image:

I0113 09:16:05.009436    8756 load_image.go:159] running command ctr --namespace k8s.io images import
/var/cache/nodeup/sha256:1c7dc3f79c105657b765092b69b55d14ad530fe3db7d95cd664191f10bd60816_https___storage_googleapis_com_kubernetes-release_release_v1_20_1_bin_linux_arm64_kube-proxy_tar
W0113 09:16:06.571666    8756 executor.go:131] error running task "LoadImage.0" (9m58s remaining to succeed):
error loading docker image with 'ctr --namespace k8s.io images import
/var/cache/nodeup/sha256:1c7dc3f79c105657b765092b69b55d14ad530fe3db7d95cd664191f10bd60816_https___storage_googleapis_com_kubernetes-release_release_v1_20_1_bin_linux_arm64_kube-proxy_tar':
exit status 1: unpacking k8s.gcr.io/kube-proxy-arm64:v1.20.1 (sha256:6f6f747fa89f88a076fa6534946e392f0629a4c55f2f7413d855281d3af90e50)...
ctr: content digest sha256:e969937ccd0f155a9cc6e79c3d433679fb85853a3adbe6280dbba18a11202a2d: not found

containerd logs show similar thing:

time="2021-01-13T09:09:14.057352011Z" level=error msg="CreateContainer within sandbox
\"e87c5d542ae2bb38266df4c4a65351a7a6b63c08b961fdb828bc979aba3e5ba1\"
for &ContainerMetadata{Name:kube-proxy,Attempt:0,} failed"
error="rpc error: code = NotFound desc = failed to create containerd container:
error unpacking image: failed to resolve rootfs: content digest
sha256:9dee21b6b85b275630ede8fe219f62d68388c32a0fba11c05f2cff4fa38adfb6: not found"

Extracting the contents of the kube-proxy.tar file can be noticed that the configuration JSON contains has the architecture set to "amd64":

$ cat  9dee21b6b85b275630ede8fe219f62d68388c32a0fba11c05f2cff4fa38adfb6.json | json_pp
{
   "architecture" : "amd64",
...

How to reproduce it (as minimally and precisely as possible):

1. Start an instance with Ubuntu 20.04 and ARM64 CPU in AWS
2. Install containerd 1.4.3 from https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/arm64/
3. Download https://storage.googleapis.com/kubernetes-release/release/v1.20.1/bin/linux/arm64/kube-proxy.tar
4. Try to import the image: sudo ctr images import kube-proxy.tar

Anything else we need to know?:

Issue was opened here because the problem seems to come from the debian-iptables image used as a base for kube-proxy.

Changing the architecture to "arm64" inside the config JSON makes the import succeed.

Environment:

• Cloud provider or hardware configuration: AWS with m6g.large instance type
• OS (e.g: cat /etc/os-release): Ubuntu 20.04 Focal ARM64
• Kernel (e.g. uname -a): n/a
• Others:

[hakman]

/cc @BenTheElder

[hakman]

Seems this will be fixed by #1839

[hakman]

xRef: kubernetes/kubernetes#98229