Support containerd container runtime on darwin/linux arm with preload=false #10402

ilya-zuyev · 2021-02-08T19:15:41Z

Currently minikube is failed to start with --container-runtime=containerd --preload=false. It works fine if --preload=true thouth

minikube start --driver=docker --preload=false --container-runtime=containerd -v=8 --alsologtostderr |& tee start.log
[...]

W0208 19:32:27.810932 3580642 logs.go:135] Found kubelet problem: Feb 08 19:31:56 minikube kubelet[5559]: E0208 19:31:56.765691    5559 pod_workers.go:191] Error syncing pod c31fe6a5afdd142cf3450ac972274b36 ("etcd-minikube_kub
e-system(c31fe6a5afdd142cf3450ac972274b36)"), skipping: failed to "StartContainer" for "etcd" with CreateContainerError: "failed to create containerd container: error unpacking image: failed to resolve rootfs: content digest s
ha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28: not found"
W0208 19:32:27.821905 3580642 logs.go:135] Found kubelet problem: Feb 08 19:32:05 minikube kubelet[5559]: E0208 19:32:05.925319    5559 pod_workers.go:191] Error syncing pod c767dbeb9ddd2d01964c2fc02c621c4e ("kube-apiserver-mi
nikube_kube-system(c767dbeb9ddd2d01964c2fc02c621c4e)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 20s restarting failed container=kube-apiserver pod=kube-apiserver-minikube_kube
-system(c767dbeb9ddd2d01964c2fc02c621c4e)"
W0208 19:32:27.827154 3580642 logs.go:135] Found kubelet problem: Feb 08 19:32:07 minikube kubelet[5559]: E0208 19:32:07.774430    5559 pod_workers.go:191] Error syncing pod c31fe6a5afdd142cf3450ac972274b36 ("etcd-minikube_kub
e-system(c31fe6a5afdd142cf3450ac972274b36)"), skipping: failed to "StartContainer" for "etcd" with CreateContainerError: "failed to create containerd container: error unpacking image: failed to resolve rootfs: content digest s
ha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28: not found"
W0208 19:32:27.830690 3580642 logs.go:135] Found kubelet problem: Feb 08 19:32:11 minikube kubelet[5559]: E0208 19:32:11.096390    5559 pod_workers.go:191] Error syncing pod c767dbeb9ddd2d01964c2fc02c621c4e ("kube-apiserver-mi
nikube_kube-system(c767dbeb9ddd2d01964c2fc02c621c4e)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 20s restarting failed container=kube-apiserver pod=kube-apiserver-minikube_kube
-system(c767dbeb9ddd2d01964c2fc02c621c4e)"
W0208 19:32:27.837225 3580642 logs.go:135] Found kubelet problem: Feb 08 19:32:16 minikube kubelet[5559]: E0208 19:32:16.930143    5559 pod_workers.go:191] Error syncing pod 57b8c22dbe6410e4bd36cf14b0f8bdc7 ("kube-controller-m
anager-minikube_kube-system(57b8c22dbe6410e4bd36cf14b0f8bdc7)"), skipping: failed to "StartContainer" for "kube-controller-manager" with CrashLoopBackOff: "back-off 20s restarting failed container=kube-controller-manager pod=k
ube-controller-manager-minikube_kube-system(57b8c22dbe6410e4bd36cf14b0f8bdc7)"
W0208 19:32:27.840193 3580642 logs.go:135] Found kubelet problem: Feb 08 19:32:19 minikube kubelet[5559]: E0208 19:32:19.538328    5559 pod_workers.go:191] Error syncing pod 57b8c22dbe6410e4bd36cf14b0f8bdc7 ("kube-controller-m
anager-minikube_kube-system(57b8c22dbe6410e4bd36cf14b0f8bdc7)"), skipping: failed to "StartContainer" for "kube-controller-manager" with CrashLoopBackOff: "back-off 20s restarting failed container=kube-controller-manager pod=k
ube-controller-manager-minikube_kube-system(57b8c22dbe6410e4bd36cf14b0f8bdc7)"
W0208 19:32:27.843476 3580642 logs.go:135] Found kubelet problem: Feb 08 19:32:22 minikube kubelet[5559]: E0208 19:32:22.760726    5559 pod_workers.go:191] Error syncing pod c767dbeb9ddd2d01964c2fc02c621c4e ("kube-apiserver-mi
nikube_kube-system(c767dbeb9ddd2d01964c2fc02c621c4e)"), skipping: failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 20s restarting failed container=kube-apiserver pod=kube-apiserver-minikube_kube
-system(c767dbeb9ddd2d01964c2fc02c621c4e)"
W0208 19:32:27.846628 3580642 logs.go:135] Found kubelet problem: Feb 08 19:32:23 minikube kubelet[5559]: E0208 19:32:23.765975    5559 pod_workers.go:191] Error syncing pod c31fe6a5afdd142cf3450ac972274b36 ("etcd-minikube_kub
e-system(c31fe6a5afdd142cf3450ac972274b36)"), skipping: failed to "StartContainer" for "etcd" with CreateContainerError: "failed to create containerd container: error unpacking image: failed to resolve rootfs: content digest s
ha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28: not found"
I0208 19:32:27.851257 3580642 logs.go:120] Gathering logs for dmesg ...
I0208 19:32:27.851273 3580642 ssh_runner.go:149] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0208 19:32:27.858659 3580642 command_runner.go:123] > [Feb 8 16:17] kauditd_printk_skb: 7 callbacks suppressed
I0208 19:32:27.858723 3580642 logs.go:120] Gathering logs for describe nodes ...
I0208 19:32:27.858738 3580642 ssh_runner.go:149] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
I0208 19:32:27.920915 3580642 command_runner.go:123] ! The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0208 19:32:27.920953 3580642 logs.go:127] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/min
ikube/binaries/v1.20.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:

The text was updated successfully, but these errors were encountered:

afbjorklund · 2021-02-08T19:19:36Z

What is the problem here, is it broken ? It is supposed to fall back to cache, if there is no available preload

ilya-zuyev · 2021-02-08T19:23:06Z

/assign

ilya-zuyev · 2021-02-08T19:35:10Z

What is the problem here, is it broken ? It is supposed to fall back to cache, if there is no available preload

Updated the description. It seems, the issue is related to failed to create containerd container: error unpacking image: failed to resolve rootfs: content digest sha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28: not found"

ilya-zuyev · 2021-02-26T23:26:16Z

It looks like we have an issue with etcd image manifest

on ARM host:

ubuntu@ip-172-31-39-30:~/src/z/minikube$ out/minikube delete --all --purge
* Successfully deleted all profiles
* Successfully purged minikube directory located at - [/home/ubuntu/.minikube]

ubuntu@ip-172-31-39-30:~/src/z/minikube$ minikube start --driver=docker --container-runtime=containerd --preload=false -v=5 --alsologtostderr  >start.log 2>&1
[ hangs for 5 minutes and fails ]

If we check the cached etcd image:


ubuntu@ip-172-31-39-30:~/src/minikube$ tar xfv ~/.minikube/cache/images/k8s.gcr.io/etcd* --wildcards 'sha*' -O | jq | head
sha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28
{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "ExposedPorts": {
ubuntu@ip-172-31-39-30:~/src/minikube$

note "amd64" arch on arm machine

The content of image layers is correct though:

ubuntu@ip-172-31-39-30:~$ mkdir -p ~/tmp/image
ubuntu@ip-172-31-39-30:~$ tar xf ~/.minikube/cache/images/k8s.gcr.io/etcd* -C ~/tmp/image
ubuntu@ip-172-31-39-30:~$ ls ~/tmp/image
3c187ad2b50e732c3d1aceb28f94c6358711e99b85ddd6c976b0d882c7cb24df.tar.gz  aedf218301234b573821148738d44bf450f16e33d7f33b671c7a068f74d8b10b.tar.gz  sha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28
4000adbbc3eb1099e3a263c418f7c1d7def1fa2de0ae00ba48245cda9389c823.tar.gz  c01a58061c3de2ebe6e61222afcf73ec28cfa2f64788bcf6beed9e06ced5d70a.tar.gz
7e982895c1fce6598d3ae953ac54ceac9f9846ff1ae4101d8f885b6400a8d92a.tar.gz  manifest.json
ubuntu@ip-172-31-39-30:~$ tar zxf ~/tmp/image/7e982895c1fce6598d3ae953ac54ceac9f9846ff1ae4101d8f885b6400a8d92a.tar.gz -C ~/tmp/image                                                         
ubuntu@ip-172-31-39-30:~$ file ~/tmp/image/usr/local/bin/etcd
/home/ubuntu/tmp/image/usr/local/bin/etcd: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=C2gD-OhOm0EaEE6O5P-G/E0Rtpt8guBmaGHRO19gQ/OcfznmXfak8UQlpC_k77/c5upPp32vabmtzAPWE5y, not stripped
ubuntu@ip-172-31-39-30:~$

Let's look inside the started kicbase container:

ubuntu@ip-172-31-39-30:~/src/minikube$ m ssh 

# inside the kicbase container 

docker@minikube:~$ sudo crictl ps  -a
CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID
aaaf9782dd452       3a1a2b528610a       About a minute ago   Exited              kube-controller-manager   6                   3f73d18167afc
a881df6c16519       95d99817fc335       About a minute ago   Exited              kube-apiserver            6                   f842148972b17
d9e7d474c7e50       60d957e44ec8a       9 minutes ago        Running             kube-scheduler            0                   30e300f7726ee
docker@minikube:~$ 

docker@minikube:~$ sudo ctr -n k8s.io images  ls "name~=etcd"
ERRO[0000] failed resolving platform for image k8s.gcr.io/etcd:3.4.13-0  error="content digest sha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28: not found"
REF                      TYPE                                                 DIGEST                                                                  SIZE      PLATFORMS LABELS                          
k8s.gcr.io/etcd:3.4.13-0 application/vnd.docker.distribution.manifest.v2+json sha256:e532480280867cc44644ca16c64c812c7555ecd4b5ab37a7d3080fb3a611440b 128.3 MiB -         io.cri-containerd.image=managed 
docker@minikube:~$ 


docker@minikube:~$ sudo ctr -n k8s.io run k8s.gcr.io/etcd:3.4.13-0  foo
ctr: content digest sha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28: not found
docker@minikube:~$

Note '-' PLATFORM. It looks like the image was imported by ctr image import incorrectly and kubeadm couldn't create a etcd deployment

ilya-zuyev · 2021-03-02T19:45:33Z

It looks like the problem here is config of k8s.gcr.io/v2/etcd/manifests/3.4.13-0 image for linux/arm64 platform. For some reason it's amd64, not arm64

Tried to fetch it manually:

➜  ~ curl -L  k8s.gcr.io/v2/etcd/manifests/3.4.13-0 
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1372,
         "digest": "sha256:bd4d2c9a19be8a492bc79df53eee199fd04b415e9993eb69f7718052602a147a",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1372,
         "digest": "sha256:4cadaf5f37998d038861c66215809eee8316c7604934e03dd86015a0f6704cd3",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1372,
         "digest": "sha256:cab4dc43598ed9b5c6a524dfea1abf4772a5e164c330451e5ca2635a95675aa8",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1373,
         "digest": "sha256:03d9db03b83a5991ede76dd53e19a1e46e6bf3c498ae8ace2e1be8f3d1d03ad6",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1372,
         "digest": "sha256:dd9baad37b716232a737cd781cf65ceb9f7c541264cdde9ad3870d84e8644a49",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}%  


➜  ~ curl -L  k8s.gcr.io/v2/etcd/manifests/sha256:cab4dc43598ed9b5c6a524dfea1abf4772a5e164c330451e5ca2635a95675aa8
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 2929,
      "digest": "sha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 639412,
         "digest": "sha256:4000adbbc3eb1099e3a263c418f7c1d7def1fa2de0ae00ba48245cda9389c823"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 962465,
         "digest": "sha256:c01a58061c3de2ebe6e61222afcf73ec28cfa2f64788bcf6beed9e06ced5d70a"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 122578700,
         "digest": "sha256:7e982895c1fce6598d3ae953ac54ceac9f9846ff1ae4101d8f885b6400a8d92a"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1090470,
         "digest": "sha256:3c187ad2b50e732c3d1aceb28f94c6358711e99b85ddd6c976b0d882c7cb24df"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 9254541,
         "digest": "sha256:aedf218301234b573821148738d44bf450f16e33d7f33b671c7a068f74d8b10b"
      }
   ]
}% 





➜  ~ curl -L  k8s.gcr.io/v2/etcd/blobs/sha256:05b738aa1bc6355db8a2ee8639f3631b908286e43f584a3d2ee0c472de033c28 | jq

{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "ExposedPorts": {
      "2379/tcp": {},
      "2380/tcp": {},
      "4001/tcp": {},
      "7001/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt",
      "ETCD_UNSUPPORTED_ARCH=arm64"
    ],
    "Cmd": null,
    "Image": "sha256:3cd4c7a04b04c476e23baac9d8b497467f13e5c6f86b16ebda9ae8edbb0186ed",
    "Volumes": null,
    "WorkingDir": "/",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": null
  },
  "container": "9ed40c528a69f165979a3e257649bfc174f947ac28189577913449b5edfd9e89",
  "container_config": {
    "Hostname": "9ed40c528a69",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "ExposedPorts": {
      "2379/tcp": {},
      "2380/tcp": {},
      "4001/tcp": {},
      "7001/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt",
      "ETCD_UNSUPPORTED_ARCH=arm64"
    ],
    "Cmd": [
      "/bin/sh",
      "-c",
      "#(nop) ",
      "ENV ETCD_UNSUPPORTED_ARCH=arm64"
    ],
    "Image": "sha256:3cd4c7a04b04c476e23baac9d8b497467f13e5c6f86b16ebda9ae8edbb0186ed",
    "Volumes": null,
    "WorkingDir": "/",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {}
  },
  "created": "2020-08-27T13:54:20.043362298Z",
  "docker_version": "19.03.8",
  "history": [
    {
      "created": "1970-01-01T00:00:00Z",
      "author": "Bazel",
      "created_by": "bazel build ..."
    },
    {
      "created": "2020-08-27T13:47:31.271664261Z",
      "created_by": "/bin/sh -c #(nop) WORKDIR /",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T13:54:14.492776476Z",
      "created_by": "/bin/sh -c #(nop) COPY file:cfbf17b03c2adf1716c6e76edbee67a5be7c6241e0a379eac5e7c3ce4475cb1a in /bin/ "
    },
    {
      "created": "2020-08-27T13:54:14.615687579Z",
      "created_by": "/bin/sh -c #(nop)  EXPOSE 2379 2380 4001 7001",
      "empty_layer": true
    },
    {
      "created": "2020-08-27T13:54:18.267262647Z",
      "created_by": "/bin/sh -c #(nop) COPY multi:849e1b626bf1eda94958dc7277f882657285bf5e0f495c93121fd04612e095ab in /usr/local/bin/ "
    },
    {
      "created": "2020-08-27T13:54:19.480162205Z",
      "created_by": "/bin/sh -c #(nop) COPY file:fd5b8ee929cc7a7a318cde74df2e33dfb0ecad224b5d92c511fba659cc3cb37e in /bin/ "
    },
    {
      "created": "2020-08-27T13:54:19.849020987Z",
      "created_by": "/bin/sh -c #(nop) COPY multi:4a4c1a72453581fdd4a183848e1993781030db8bc02e7029fb25c838d9be5523 in /usr/local/bin/ "
    },
    {
      "created": "2020-08-27T13:54:20.043362298Z",
      "created_by": "/bin/sh -c #(nop)  ENV ETCD_UNSUPPORTED_ARCH=arm64",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:d72a74c56330b347f7d18b64d2effd93edd695fde25dc301d52c37efbcf4844e",
      "sha256:d984922c069982565ab59a43b431d4f8ee93d1336303f3cc86c0fe1b633e8265",
      "sha256:3d7007202b6363b032ca53a67e6eaff14e91edb2dc89cccc193d5b02f7a4cdb5",
      "sha256:ae19a9659aebb1e039edd9f140c81dae224df09e8b73e28774f9a554272cceb8",
      "sha256:fbf70b84758aff28e16a0e7df3748794fba5abc64af7e60c2581205a991c6385"
    ]
  }
}

Note "architecture": "amd64", line

ilya-zuyev · 2021-03-02T19:59:37Z

kubernetes/kubernetes#99656

ilya-zuyev self-assigned this Feb 8, 2021

ilya-zuyev added arch/arm64 kind/bug Categorizes issue or PR as related to a bug. labels Feb 8, 2021

afbjorklund added co/runtime/containerd co/runtime/crio CRIO related issues labels Feb 8, 2021

ilya-zuyev added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Feb 8, 2021

ilya-zuyev added this to the v1.18.0 milestone Feb 8, 2021

ilya-zuyev changed the title ~~Support non-docker (containerd + cri-o) container runtimes on darwin/linux arm~~ Support containerd container runtimes on darwin/linux arm Feb 8, 2021

ilya-zuyev changed the title ~~Support containerd container runtimes on darwin/linux arm~~ Support containerd container runtime on darwin/linux arm Feb 8, 2021

afbjorklund removed the co/runtime/crio CRIO related issues label Feb 14, 2021

ilya-zuyev mentioned this issue Feb 25, 2021

[WIP] Preload CNI images #10609

Closed

medyagh changed the title ~~Support containerd container runtime on darwin/linux arm~~ Support containerd container runtime on darwin/linux arm with preload=false Feb 25, 2021

medyagh modified the milestones: v1.18.0, v.1.19.0-candidate Feb 26, 2021

ilya-zuyev mentioned this issue Feb 26, 2021

arm64: Fix incorrect image arch in the manifest for etcd and other kube images #10642

Merged

ilya-zuyev mentioned this issue Mar 18, 2021

arm64 + docker driver: --container-runtime=containerd: ctr: content digest sha256:x: not found (reproducible) #10852

Closed

medyagh closed this as completed in #10642 Mar 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support containerd container runtime on darwin/linux arm with preload=false #10402

Support containerd container runtime on darwin/linux arm with preload=false #10402

ilya-zuyev commented Feb 8, 2021 •

edited

Loading

afbjorklund commented Feb 8, 2021

ilya-zuyev commented Feb 8, 2021

ilya-zuyev commented Feb 8, 2021

ilya-zuyev commented Feb 26, 2021

ilya-zuyev commented Mar 2, 2021 •

edited

Loading

ilya-zuyev commented Mar 2, 2021

Support containerd container runtime on darwin/linux arm with preload=false #10402

Support containerd container runtime on darwin/linux arm with preload=false #10402

Comments

ilya-zuyev commented Feb 8, 2021 • edited Loading

afbjorklund commented Feb 8, 2021

ilya-zuyev commented Feb 8, 2021

ilya-zuyev commented Feb 8, 2021

ilya-zuyev commented Feb 26, 2021

ilya-zuyev commented Mar 2, 2021 • edited Loading

ilya-zuyev commented Mar 2, 2021

ilya-zuyev commented Feb 8, 2021 •

edited

Loading

ilya-zuyev commented Mar 2, 2021 •

edited

Loading