docker run error 'exec format error' post upgrade on Apple silicon M2 #7168

habasse-traore · 2024-02-04T15:47:30Z

Description

I'm getting an error running 2 images post upgrade to the latest version 4.27.1 of docker-desktop for mac
The container fails with "exec format error"
Also docker-desktop fails when trying to push an image to docker hub and I have to reset to factory settings before getting it to work again.

Reproduce

Platform: Macbook Pro M2
OS: Sonoma 14.2.1
Docker Desktop: 4.27.1
cmd:

docker pull gcr.io/k8s-minikube/minikube-ingress-dns:0.0.2
docker run gcr.io/k8s-minikube/minikube-ingress-dns:0.0.2
Output: "exec /usr/local/bin/docker-entrypoint.sh: exec format error"
docker pull docker.io/upmcenterprises/registry-creds:1.10
docker run docker.io/upmcenterprises/registry-creds:1.10
Output: "exec /registry-creds: exec format error"

Expected behavior

docker run should run the container. The same images run on docker-desktop 4.26.1

docker version

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.2
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Thu Feb  1 00:18:45 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.27.1 (136059)
 Engine:
  Version:          25.0.2
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       fce6e0c
  Built:            Thu Feb  1 00:23:21 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    25.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /Users/habasse.traore/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.3-desktop.1
    Path:     /Users/habasse.traore/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.22
    Path:     /Users/habasse.traore/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/habasse.traore/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/habasse.traore/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/habasse.traore/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /Users/habasse.traore/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/habasse.traore/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/habasse.traore/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/habasse.traore/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/habasse.traore/.docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 25.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 23.44GiB
 Name: docker-desktop
 ID: a1ec940b-3186-488e-80d1-f6879134a154
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

EFBFC4C6-AD42-4470-8971-A009AF56D571/20240204153215

Additional Info

I'm spinning these containers has part of a local minikube dev environment
the 2 containers images falling are required to enable ingress-dns and registry-creds service for the cluster

geek-kb · 2024-02-05T15:07:00Z

I have the same issue.
It started today after updating Docker desktop to the latest version on MacOS M3 Pro Sonoma 14.1 .
Please assist.

kryszce · 2024-02-06T08:37:14Z

I have exactly the same problem after upgrading to the latest docker version
Version 4.27.1 (136059)
OS Sonoma 14.3

Could someone please advise on how to get this working?

fozcode · 2024-02-06T10:23:27Z

Same with me after upgrade to 4.27:

Pulling image "docker.io/bitnami/postgresql:15.1.0-debian-11-r12"
Successfully pulled image "docker.io/bitnami/postgresql:15.1.0-debian-11-r12" in 12.162s (26.656s including waiting)
exec /opt/bitnami/scripts/postgresql/entrypoint.sh: exec format error

Downgrading to 4.26 has fixed it and the image runs again. Download link: https://docs.docker.com/desktop/release-notes/#4261

fozcode · 2024-02-07T09:27:30Z

In my case the image is only available for amd64 link

OS/ARCH
linux/amd64

I have Rosetta emulation disabled. So presumably 4.26 was performing some other kind of amd64 emulation and 4.27 is not. I'm going to update my image to a newer version that provides both amd64 and arm64.

rafaelavd · 2024-02-07T11:16:29Z

anyone can point out what was the breaking change that caused this?
Thanks

mdkent · 2024-02-07T20:05:02Z

We've got multiple people hitting this here.

Platform: Apple M2 Max
OS: 14.3 (23D56)
Docker Desktop 4.27.1 (136059)

Pretty easy to reproduce with Rosetta enabled:

mk-mbp3 3.3.0 ~ docker run --platform linux/amd64 hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:4bd78111b6914a99dbc560e6a20eab57ff6655aea4a80c50b0c5491968cbc2e6
Status: Downloaded newer image for hello-world:latest
exec /hello: exec format error

with Rosetta disabled hello-world runs, but I'm getting weird crashes like

       qemu: uncaught target signal 11 (Segmentation fault) - core dumped
       Segmentation fault

in other apps.

Downgrading to 4.26.1 fixed it as well.

adityasamant25 · 2024-02-08T16:45:02Z

I can confirm this as well.
Upgrading to 4.27.1 has broken x86 image emulation on Mac.

Downgrading to 4.26.1 works, so I'll be staying on this version until the bug is fixed on 4.27.x

dgageot · 2024-02-08T19:09:29Z

Hi everyone, I can't reproduce on 4.27.2 that was just released but I think it might be a transient error. Could you try this new version and tell me if it fixes your issue?

mmarchini · 2024-02-09T00:32:50Z

@dgageot still happening consistently on 4.27.2 on macOS 14.3.1. I can use docker run --platform linux/amd64 ubuntu:jammy uname -a to reproduce the error. It also doesn't look device specific since I've had other engineers reporting the same issue today. Could it be at all related to #7172? Both are related to architecture emulation and both started happening with 4.27.0.

jglogan · 2024-02-09T03:31:57Z

If you're using any kind of kubernetes node image in your linuxkit VM, it could be because the new node images have some systemd fun that messes up the binfmt handlers. Since those aren't namespaced, they'll mess up emulation handling for all your containers.

See:

v0.21.0: kind container makes changes to binfmt configutation kubernetes-sigs/kind#3510
Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images kubernetes/minikube#17700

Workaround for kind, until a fix arrives (PR is done), is to use the --image option to select your node image, and use one of the supported images from the v0.20.0 release notes. Downside of this is that you'll lose the leaky container fixes that landed with 0.21.0, so definitely get away from those images once there are revised images for 0.21.0.

Workaround for minikube is to use the --base-image option to do the same, but here you can select any new image from https://console.cloud.google.com/gcr/images/k8s-minikube/global/kicbase-builds that works for you. Think you might have to wait for the 1.33 release of minikube to get a fixed image by default.

dgageot · 2024-02-09T08:04:42Z

Thanks a lot @jglogan, that makes a lot of sense. I couldn't figure out how people's binfmt could be messed up that much!

Everyone:

Those of you with the qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault error, you should follow issue #7172. tl;dr the qemu update we shipped with 4.27 seems to have an issue.

Those of you with the exec format error, please can you look at @jglogan's comment and tell me if that applies to you?

fozcode · 2024-02-09T10:19:22Z

Those of you with the exec format error, please can you look at @jglogan's and tell me if that applies to you?

My use case is with minikube so I think the answer is yes. Thanks for figuring this out!

alexwilson1 · 2024-02-09T13:50:18Z

Just tried minikube start --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703092832-17830 on Docker 4.27.2 (M2) and same issue exec /bin/sh: exec format error (after removing minikube completely)

Also tried gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1707386837-18127 but to no avail

habasse-traore · 2024-02-09T17:55:08Z

Upgrading to 4.27.2 seems to have revolved the issue for me
Platform: Apple M2 Max
OS: Sonoma 14.2.1
Docker Desktop 4.27.2 (137060)
Docker-Desktop General Config

My use case is with minikube and followed @jglogan comments running
minikube start --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703092832-17830 --driver docker
minikube starts now without any issues. The previously failing container images ingress-dns and registry-creds service are now running without exec format error

Ran an additional validation by running docker run --platform linux/amd64 hello-world and that also seems to run fine

jglogan · 2024-02-09T18:20:15Z

@alexwilson1 try ensuring that minikube is not running, and then restart DD entirely.

When you run a minikube that uses the broken node images, emulation is not going to work anywhere until your linuxkit VM gets rebooted.

And if you have minikube running when you restart and your node containers have an auto start policy, the handlers will be removed as soon as DD fires up and starts those containers.

jglogan · 2024-02-09T18:24:56Z

@fozcode No problem! TBH I didn't figure it out, a colleague did. Just passing along what they found!

livvius · 2024-02-09T18:46:40Z

Upgrading to 4.27.2 seems to have revolved the issue for me Platform: Apple M2 Max OS: Sonoma 14.2.1 Docker Desktop 4.27.2 (137060) Docker-Desktop General Config

My use case is with minikube and followed @jglogan comments running minikube start --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703092832-17830 --driver docker minikube starts now without any issues. The previously failing container images ingress-dns and registry-creds service are now running without exec format error

Ran an additional validation by running docker run --platform linux/amd64 hello-world and that also seems to run fine

Ingress-dns seems working fine now, but it seems not completly fixed I'm using Percona MongoDB Operator and has the same issue

alexwilson1 · 2024-02-10T10:24:22Z

@alexwilson1 try ensuring that minikube is not running, and then restart DD entirely.

When you run a minikube that uses the broken node images, emulation is not going to work anywhere until your linuxkit VM gets rebooted.

And if you have minikube running when you restart and your node containers have an auto start policy, the handlers will be removed as soon as DD fires up and starts those containers.

Thanks! I removed all containers, images, volumes from DD, did a docker system prune, removed dd completely then installed 4.7.2 and launched with minikube start --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703092832-17830 and now it works

eko · 2024-02-11T06:57:36Z

I have the same issue here on macOS Sonoma 14.3.1 and Docker Desktop 4.27.2 (Rosetta emulation enabled):

$ docker run --platform linux/amd64 hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:4bd78111b6914a99dbc560e6a20eab57ff6655aea4a80c50b0c5491968cbc2e6
Status: Downloaded newer image for hello-world:latest
exec /hello: exec format error

jgmartinez · 2024-02-12T09:54:57Z

I'm also facing the same issue. Mac M3 Max, MacOS Sonoma 14.3.1, Docker Desktop 4.27.2 with Rosetta enabled.

I fully reinstalled Docker Desktop and it worked for a while, but the issue came back.

~ docker run --platform linux/amd64 -it ubuntu:latest bash
exec /usr/bin/bash: exec format error

dgageot · 2024-02-13T09:35:46Z

I'm going to close this issue. It's not a Docker Desktop issue per se. It's caused by minikube/kind messing up with Docker Desktop's qemu/rosetta configuration.

This comment gives a solution. Basically, Minikube needs to be started with a newer image that was fixed to not mess with Docker Desktop.

biscout42 · 2024-02-19T15:46:58Z

Same issue here, Mac M2 Pro, MacOS Sonoma 14.3.1, Docker Desktop 4.27.2 with Rosetta enabled.

Not related to minikube. Following command fails on DD 4.27.2

~ docker run --platform linux/amd64 -it ubuntu:latest bash
exec /usr/bin/bash: exec format error

And it works with DD 4.26.1

jlmoya · 2024-02-19T22:07:39Z

Same as @biscout42 , After upgrading to 4.27.x all AMD images fail to start with "exec format error" on Mac M2 with Sonoma 14.3.1 and have Rosetta enabled and it has nothing to do with minikube. Tested with 4.27.0, 4.27.1 and 4.27.2 all fail with the same "exec format error" message. Falling back to 4.26.1 works flawlessly.

biscout42 · 2024-02-20T08:02:03Z

@dgageot , would you suggest to create a new issue or re-open this one? As I see the topic was also about docker w/o minikube (only image name)

docker pull gcr.io/k8s-minikube/minikube-ingress-dns:0.0.2
docker run gcr.io/k8s-minikube/minikube-ingress-dns:0.0.2

vietvudanh · 2024-02-20T09:21:08Z

I don't think this problem caused by minikube, I have deleted, using new image for minikube, even remove minikube all together but still got the error. Others stated that they do not use minikube too.

dgageot · 2024-02-20T09:36:56Z

@biscout42 @jlmoya @vietvudanh Please open another issue if you really think it's not related to minikube/k8s/kind.
I have to admit I have doubts since at least two of you have either installed minikube or ran minikube images.
On the new issue, please share your diagnostics so that I can try to understand. Thanks!

biscout42 · 2024-02-20T09:59:57Z

Double checked again, and imho @dgageot , you have a point. If I remove minikube via

brew remove minikube

and then clean and restart DD

docker system prune -a

I can run following command w/o errors on Docker Desktop 4.27.2 (137060)

docker run --platform linux/amd64 -it ubuntu:latest bash

biscout42 · 2024-02-20T10:16:10Z

FYI: kubernetes/minikube#18217

vietvudanh · 2024-02-20T10:34:27Z

Oh ok, my bad. I didn't prune everything after remove minikube.

I did what @biscout42 suggest in his comment and now it looks fine.

Thank you @dgageot and @biscout42.

habasse-traore added the needs-triage label Feb 4, 2024

docker-robot bot added the version/4.27.1 label Feb 4, 2024

dgageot self-assigned this Feb 8, 2024

dgageot added status/triage area/emulation and removed needs-triage labels Feb 8, 2024

dgageot closed this as completed Feb 13, 2024

dgageot removed their assignment Feb 13, 2024

biscout42 mentioned this issue Feb 20, 2024

docker run error 'exec format error' on Apple silicon M2 on docker desktop 4.27.2 kubernetes/minikube#18217

Closed

ytsarev mentioned this issue Jun 6, 2024

deploy-full-local-setup: external-dns pod exec format error with Docker Desktop > 4.26.1 k8gb-io/k8gb#1588

Closed

ConnorNelson mentioned this issue Sep 5, 2024

Infrastructure: Fix MacOS support pwncollege/dojo#558

Open

devlikepro mentioned this issue Sep 13, 2024

[BUG][WEBJS] - Interface/api/sendText reply_to message, click the message abnormally in the official app or web terminal devlikeapro/waha#542

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker run error 'exec format error' post upgrade on Apple silicon M2 #7168

docker run error 'exec format error' post upgrade on Apple silicon M2 #7168

habasse-traore commented Feb 4, 2024

geek-kb commented Feb 5, 2024

kryszce commented Feb 6, 2024

fozcode commented Feb 6, 2024 •

edited

Loading

fozcode commented Feb 7, 2024 •

edited

Loading

rafaelavd commented Feb 7, 2024

mdkent commented Feb 7, 2024

adityasamant25 commented Feb 8, 2024

dgageot commented Feb 8, 2024

mmarchini commented Feb 9, 2024

jglogan commented Feb 9, 2024 •

edited

Loading

dgageot commented Feb 9, 2024

fozcode commented Feb 9, 2024

alexwilson1 commented Feb 9, 2024 •

edited

Loading

habasse-traore commented Feb 9, 2024 •

edited

Loading

jglogan commented Feb 9, 2024 •

edited

Loading

jglogan commented Feb 9, 2024

livvius commented Feb 9, 2024 •

edited

Loading

alexwilson1 commented Feb 10, 2024

eko commented Feb 11, 2024

jgmartinez commented Feb 12, 2024

dgageot commented Feb 13, 2024

biscout42 commented Feb 19, 2024

jlmoya commented Feb 19, 2024 •

edited

Loading

biscout42 commented Feb 20, 2024 •

edited

Loading

vietvudanh commented Feb 20, 2024

dgageot commented Feb 20, 2024

biscout42 commented Feb 20, 2024

biscout42 commented Feb 20, 2024

vietvudanh commented Feb 20, 2024

docker run error 'exec format error' post upgrade on Apple silicon M2 #7168

docker run error 'exec format error' post upgrade on Apple silicon M2 #7168

Comments

habasse-traore commented Feb 4, 2024

Description

Reproduce

Expected behavior

docker version

docker info

Diagnostics ID

Additional Info

geek-kb commented Feb 5, 2024

kryszce commented Feb 6, 2024

fozcode commented Feb 6, 2024 • edited Loading

fozcode commented Feb 7, 2024 • edited Loading

rafaelavd commented Feb 7, 2024

mdkent commented Feb 7, 2024

adityasamant25 commented Feb 8, 2024

dgageot commented Feb 8, 2024

mmarchini commented Feb 9, 2024

jglogan commented Feb 9, 2024 • edited Loading

dgageot commented Feb 9, 2024

fozcode commented Feb 9, 2024

alexwilson1 commented Feb 9, 2024 • edited Loading

habasse-traore commented Feb 9, 2024 • edited Loading

jglogan commented Feb 9, 2024 • edited Loading

jglogan commented Feb 9, 2024

livvius commented Feb 9, 2024 • edited Loading

alexwilson1 commented Feb 10, 2024

eko commented Feb 11, 2024

jgmartinez commented Feb 12, 2024

dgageot commented Feb 13, 2024

biscout42 commented Feb 19, 2024

jlmoya commented Feb 19, 2024 • edited Loading

biscout42 commented Feb 20, 2024 • edited Loading

vietvudanh commented Feb 20, 2024

dgageot commented Feb 20, 2024

biscout42 commented Feb 20, 2024

biscout42 commented Feb 20, 2024

vietvudanh commented Feb 20, 2024

fozcode commented Feb 6, 2024 •

edited

Loading

fozcode commented Feb 7, 2024 •

edited

Loading

jglogan commented Feb 9, 2024 •

edited

Loading

alexwilson1 commented Feb 9, 2024 •

edited

Loading

habasse-traore commented Feb 9, 2024 •

edited

Loading

jglogan commented Feb 9, 2024 •

edited

Loading

livvius commented Feb 9, 2024 •

edited

Loading

jlmoya commented Feb 19, 2024 •

edited

Loading

biscout42 commented Feb 20, 2024 •

edited

Loading