Skip to content

Latest 3.10-slim-bullseye fails on Macbook M1 with Rosetta emulation #792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fabienheureux opened this issue Jan 25, 2023 · 7 comments · Fixed by #794
Closed

Latest 3.10-slim-bullseye fails on Macbook M1 with Rosetta emulation #792

fabienheureux opened this issue Jan 25, 2023 · 7 comments · Fixed by #794

Comments

@fabienheureux
Copy link

Description

I am using Python image with amd64 platform flag on an arm64 host with a recent experimental feature from Docker : Rosetta emulation.
Recents changes #785 and #790 appear to have broken latest 3.10-slim-bullseye tag

Steps to reproduce

  1. Activate "Rosetta for x86/amd64 emulation on Apple Silicon" (available from Docker desktop version 4.16)
  2. Get the Dockerfile written below
  3. docker build .
# Tag used : python:3.10-slim-bullseye
# Last week digest (prior to #785 and #790) : this is working fine
FROM --platform=linux/amd64 python@sha256:3ec369a5db9693672a61f0631a980e66eb6d161f206afa7c9c18237adac466f5 
RUN python --version

# Latest digest : this one breaks for me at the moment
FROM --platform=linux/amd64 python@sha256:3ec369a5db9693672a61f0631a980e66eb6d161f206afa7c9c18237adac466f5
RUN python --version

Output from docker build

 [+] Building 1.6s (5/5) FINISHED                                                                                                                                                                                                                
 => [internal] load build definition from Dockerfile                                                                                                                                                                                       0.0s
 => => transferring dockerfile: 342B                                                                                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/python@sha256:e04bc5390a13e6e910980715f1156552fbd3deddf7668e1892575989f9504c70                                                                                                          1.2s
 => CACHED [stage-1 1/2] FROM docker.io/library/python@sha256:e04bc5390a13e6e910980715f1156552fbd3deddf7668e1892575989f9504c70                                                                                                             0.0s
 => ERROR [stage-1 2/2] RUN python --version                                                                                                                                                                                               0.3s
------
 > [stage-1 2/2] RUN python --version:
#5 0.278 rosetta error: failed to open elf at 
#5 0.279 Trace/breakpoint trap
------
executor failed running [/bin/sh -c python --version]: exit code: 133

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.0)
  compose: Docker Compose (Docker Inc., v2.15.1)
  dev: Docker Dev Environments (Docker Inc., v0.0.5)
  extension: Manages Docker extensions (Docker Inc., v0.2.17)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.23.0)

Server:
Containers: 56
Running: 0
Paused: 0
Stopped: 56
Images: 139
Server Version: 20.10.22
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
seccomp
Profile: default
cgroupns
Kernel Version: 5.15.49-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 5
Total Memory: 7.667GiB
Name: docker-desktop
ID: S7XT:BZCS:DMNP:2TAA:PFBH:7PU4:CKGN:MNJW:CWHQ:U6RS:VLO2:UXQI
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5000
127.0.0.0/8
Live Restore Enabled: false

@tianon
Copy link
Member

tianon commented Jan 25, 2023

We're setting rpath correctly (and very conservatively), so this is definitely going to be a bug in Rosetta's parsing of the ELF binary. 😞

@jimnicholls
Copy link
Contributor

This is not just 3.10-slim-bullseye. It happens for alpine, non-slim and different Python versions.

docker run --rm -it --platform=linux/amd64 python:3.11-alpine
Digest: sha256:ca1298a74c3495b6bc539044352301fa371f6e10b0e085f93d2a4ef3e4c3fa6c
Status: Downloaded newer image for python:3.11-alpine
rosetta error: failed to open elf at ¿ELF

The ¿ is actually code point 007F.

Sometimes I get a seg fault instead of a rosetta error.

docker run --rm -it --platform=linux/amd64 python:3.8 /bin/bash
Digest: sha256:0bdd43369c583eb82a809fbe6908d6ec721b7a29f7d9dce427d70924baf0ab7b
Status: Downloaded newer image for python:3.8
root@0dd6d9e2e463:/# python3
Segmentation fault

@tianon
Copy link
Member

tianon commented Jan 26, 2023

I can't reproduce with QEMU's emulation, so this has to be a bug in Rosetta -- does anyone know where would be an appropriate place to raise/file that?

@rik
Copy link

rik commented Jan 27, 2023

Before sending feedback to Apple, I think there's one more test to do: execute an x86_64 binary with similar modifications directly on an ARM-based Mac, without Docker.

If that test passes then it may be a Docker issue or a Virtualization.framework issue. If that test fails, it is a Rosetta issue.

(This is not my area of expertise at all so I have no idea how to do that)

How to send Apple feedback

@jimnicholls
Copy link
Contributor

jimnicholls commented Jan 27, 2023

Nice idea @rik. Unfortunately macOS doesn't use ELF. It uses the Mach-O format1.

Since macOS Ventura, users running virtual machines with Linux as a guest operating system, can make use of Rosetta 2 to run x86-64 code compiled for Linux, within the virtual machine. Rosetta 2 works as a runtime binary, which is required to be installed on the guest operating system. 2

It turns out what I was seeing in the Rosetta error message was the first four bytes of an ELF, which are the ELF magic number 0x7F E L F. 3.

Footnotes

  1. https://en.wikipedia.org/wiki/Mach-O

  2. https://en.wikipedia.org/wiki/Rosetta_(software)#Rosetta_2

  3. https://en.wikipedia.org/wiki/Executable_and_Linkable_Format#File_header

@jimnicholls
Copy link
Contributor

jimnicholls commented Jan 27, 2023

Using patchelf to set and then remove the rpath triggers this issue. patchelf seems to be make substantial changes to the ELF, particularly the program headers. Even hello world breaks after patchelf.

Dockerfile

# syntax = docker/dockerfile:1

FROM debian:bullseye

SHELL [ "/bin/bash", "-c", "-x" ]

RUN <<END_RUN
    apt-get update
    apt-get install -y --no-install-recommends gcc libc6-dev patchelf elfutils
    rm -rf /var/lib/apt/lists/*
END_RUN

RUN <<END_RUN
    cat <<END_OF_C >hello.c
#include <stdio.h>
int main() {
    printf("Hello world\n");
}
END_OF_C
    gcc -o hello hello.c
    ./hello
    patchelf --print-rpath hello
    patchelf --set-rpath '$ORIGIN/../lib' --output patched-hello hello
    patchelf --print-rpath patched-hello
    patchelf --remove-rpath patched-hello
    patchelf --print-rpath patched-hello
    diff -u <(eu-readelf -h -d hello) <(eu-readelf -h -d patched-hello)
    eu-readelf -l hello
    eu-readelf -l patched-hello
    ./hello
    ./patched-hello
END_RUN

docker buildx build --platform=linux/amd64 --progress=plain .

#10 [3/3] RUN <<END_RUN (cat <<END_OF_C >hello.c...)
#10 0.145 + cat
#10 0.160 + gcc -o hello hello.c
#10 0.400 + ./hello
#10 0.410 Hello world
#10 0.410 + patchelf --print-rpath hello
#10 0.424
#10 0.426 + patchelf --set-rpath '$ORIGIN/../lib' --output patched-hello hello
#10 0.441 + patchelf --print-rpath patched-hello
#10 0.457 $ORIGIN/../lib
#10 0.457 + patchelf --remove-rpath patched-hello
#10 0.471 + patchelf --print-rpath patched-hello
#10 0.484
#10 0.487 + diff -u /dev/fd/63 /dev/fd/62
#10 0.487 ++ eu-readelf -h -d hello
#10 0.488 ++ eu-readelf -h -d patched-hello
#10 0.514 --- /dev/fd/63	2023-01-27 15:54:00.352150006 +0000
#10 0.514 +++ /dev/fd/62	2023-01-27 15:54:00.353150006 +0000
#10 0.514 @@ -14,14 +14,14 @@
#10 0.514    Flags:
#10 0.514    Size of this header:               64 (bytes)
#10 0.514    Size of program header entries:    56 (bytes)
#10 0.514 -  Number of program headers entries: 11
#10 0.514 +  Number of program headers entries: 13
#10 0.514    Size of section header entries:    64 (bytes)
#10 0.514    Number of section headers entries: 30
#10 0.514 -  Section header string table index: 29
#10 0.514 +  Section header string table index: 22
#10 0.514
#10 0.514
#10 0.514 -Dynamic segment contains 30 entries:
#10 0.514 - Addr: 0x0000000000003df8  Offset: 0x002df8  Link to section: [ 6] '.dynstr'
#10 0.514 +Dynamic segment contains 31 entries:
#10 0.514 + Addr: 0x0000000000005000  Offset: 0x005000  Link to section: [24] '.dynstr'
#10 0.514    Type              Value
#10 0.514    NEEDED            Shared library: [libc.so.6]
#10 0.514    INIT              0x0000000000001000
#10 0.514 @@ -30,10 +30,10 @@
#10 0.514    INIT_ARRAYSZ      8 (bytes)
#10 0.514    FINI_ARRAY        0x0000000000003df0
#10 0.514    FINI_ARRAYSZ      8 (bytes)
#10 0.514 -  GNU_HASH          0x0000000000000308
#10 0.514 -  STRTAB            0x00000000000003d8
#10 0.514 -  SYMTAB            0x0000000000000330
#10 0.514 -  STRSZ             130 (bytes)
#10 0.514 +  GNU_HASH          0x0000000000005330
#10 0.514 +  STRTAB            0x00000000000051f0
#10 0.514 +  SYMTAB            0x0000000000005288
#10 0.514 +  STRSZ             145 (bytes)
#10 0.514    SYMENT            24 (bytes)
#10 0.514    DEBUG
#10 0.514    PLTGOT            0x0000000000004000
#10 0.514 @@ -51,5 +51,6 @@
#10 0.514    NULL
#10 0.514    NULL
#10 0.514    NULL
#10 0.514 +  NULL
#10 0.514    NULL
#10 0.514    NULL
#10 0.515 + eu-readelf -l hello
#10 0.532 Program Headers:
#10 0.532   Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
#10 0.532   PHDR           0x000040 0x0000000000000040 0x0000000000000040 0x000268 0x000268 R   0x8
#10 0.532   INTERP         0x0002a8 0x00000000000002a8 0x00000000000002a8 0x00001c 0x00001c R   0x1
#10 0.532 	[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
#10 0.532   LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x000560 0x000560 R   0x1000
#10 0.532   LOAD           0x001000 0x0000000000001000 0x0000000000001000 0x0001bd 0x0001bd R E 0x1000
#10 0.532   LOAD           0x002000 0x0000000000002000 0x0000000000002000 0x000158 0x000158 R   0x1000
#10 0.532   LOAD           0x002de8 0x0000000000003de8 0x0000000000003de8 0x000248 0x000250 RW  0x1000
#10 0.532   DYNAMIC        0x002df8 0x0000000000003df8 0x0000000000003df8 0x0001e0 0x0001e0 RW  0x8
#10 0.532   NOTE           0x0002c4 0x00000000000002c4 0x00000000000002c4 0x000044 0x000044 R   0x4
#10 0.532   GNU_EH_FRAME   0x002010 0x0000000000002010 0x0000000000002010 0x00003c 0x00003c R   0x4
#10 0.532   GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW  0x10
#10 0.532   GNU_RELRO      0x002de8 0x0000000000003de8 0x0000000000003de8 0x000218 0x000218 R   0x1
#10 0.532
#10 0.532  Section to Segment mapping:
#10 0.532   Segment Sections...
#10 0.532    00
#10 0.532    01      [RO: .interp]
#10 0.532    02      [RO: .interp .note.gnu.build-id .note.ABI-tag .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt]
#10 0.532    03      [RO: .init .plt .plt.got .text .fini]
#10 0.532    04      [RO: .rodata .eh_frame_hdr .eh_frame]
#10 0.532    05      [RELRO: .init_array .fini_array .dynamic .got] .got.plt .data .bss
#10 0.532    06      [RELRO: .dynamic]
#10 0.532    07      [RO: .note.gnu.build-id .note.ABI-tag]
#10 0.532    08      [RO: .eh_frame_hdr]
#10 0.532    09
#10 0.532    10      [RELRO: .init_array .fini_array .dynamic .got]
#10 0.532 + eu-readelf -l patched-hello
#10 0.550 Program Headers:
#10 0.550   Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
#10 0.550   PHDR           0x000040 0x0000000000000040 0x0000000000000040 0x0002d8 0x0002d8 R   0x8
#10 0.550   LOAD           0x000000 0x0000000000000000 0x0000000000000000 0x000560 0x000560 R   0x1000
#10 0.550   GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW  0x10
#10 0.550   LOAD           0x001000 0x0000000000001000 0x0000000000001000 0x0001bd 0x0001bd R E 0x1000
#10 0.550   LOAD           0x002000 0x0000000000002000 0x0000000000002000 0x000158 0x000158 R   0x1000
#10 0.550   GNU_EH_FRAME   0x002010 0x0000000000002010 0x0000000000002010 0x00003c 0x00003c R   0x4
#10 0.550   LOAD           0x002de8 0x0000000000003de8 0x0000000000003de8 0x000248 0x000250 RW  0x1000
#10 0.550   GNU_RELRO      0x002de8 0x0000000000003de8 0x0000000000003de8 0x000218 0x000218 R   0x1
#10 0.550   DYNAMIC        0x005000 0x0000000000005000 0x0000000000005000 0x0001f0 0x0001f0 RW  0x8
#10 0.550   LOAD           0x005000 0x0000000000005000 0x0000000000005000 0x0003c0 0x0003c0 RW  0x1000
#10 0.550   INTERP         0x005358 0x0000000000005358 0x0000000000005358 0x00001c 0x00001c R   0x1
#10 0.550 	[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
#10 0.550   NOTE           0x005378 0x0000000000005378 0x0000000000005378 0x000020 0x000020 R   0x4
#10 0.550   NOTE           0x005398 0x0000000000005398 0x0000000000005398 0x000024 0x000024 R   0x4
#10 0.550
#10 0.550  Section to Segment mapping:
#10 0.550   Segment Sections...
#10 0.550    00
#10 0.550    01      [RO: .gnu.version .gnu.version_r .rela.dyn .rela.plt]
#10 0.550    02
#10 0.550    03      [RO: .init .plt .plt.got .text .fini]
#10 0.550    04      [RO: .rodata .eh_frame_hdr .eh_frame]
#10 0.550    05      [RO: .eh_frame_hdr]
#10 0.550    06      [RELRO: .init_array .fini_array .got] .got.plt .data .bss
#10 0.550    07      [RELRO: .init_array .fini_array .got]
#10 0.550    08      .dynamic
#10 0.550    09      .dynamic .dynstr .dynsym .gnu.hash .interp .note.ABI-tag .note.gnu.build-id
#10 0.550    10      .interp
#10 0.550    11      .note.ABI-tag
#10 0.550    12      .note.gnu.build-id
#10 0.551 + ./hello
#10 0.562 Hello world
#10 0.562 + ./patched-hello
#10 ERROR: executor failed running [/bin/bash -c -x     cat <<END_OF_C >hello.c
#include <stdio.h>
int main() {
    printf("Hello world\n");
}
END_OF_C
    gcc -o hello hello.c
    ./hello
    patchelf --print-rpath hello
    patchelf --set-rpath '$ORIGIN/../lib' --output patched-hello hello
    patchelf --print-rpath patched-hello
    patchelf --remove-rpath patched-hello
    patchelf --print-rpath patched-hello
    diff -u <(eu-readelf -h -d hello) <(eu-readelf -h -d patched-hello)
    eu-readelf -l hello
    eu-readelf -l patched-hello
    ./hello
    ./patched-hello
]: exit code: 139

@rik
Copy link

rik commented Feb 4, 2023

@jimnicholls Thank you very much for working on this. I've just confirmed it fixed the issue for me with the latest tag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants