moby/buildkit v0.20.0 causes libc-bin segfaults (`qemu: uncaught target signal 11`)

[EslamElHusseiny]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

I'm building a docker image using buildx and moby/buildkit for multiplatform images and I'm seeing a segfault error while building the image while installing libc-bin

#23 15.42 Processing triggers for libc-bin (2.31-13+deb11u3) ...
#23 15.48 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#23 15.62 Segmentation fault (core dumped)
#23 15.63 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#23 15.73 Segmentation fault (core dumped)
#23 15.73 dpkg: error processing package libc-bin (--configure):
#23 15.73  installed libc-bin package post-installation script subprocess returned error exit status 139
#23 15.75 Errors were encountered while processing:
#23 15.75  libc-bin
#23 15.85 E: Sub-process /usr/bin/dpkg returned an error code (1)
#23 ERROR: process "/dev/.buildkit_qemu_emulator /bin/sh -c apt install -y curl" did not complete successfully: exit code: 100
------
 > [stage-1 8/9] RUN apt install -y curl:
#23 15.35 Setting up publicsuffix (20220811.1734-0+deb11u1) ...
#23 15.37 Setting up libldap-2.4-2:arm64 (2.4.57+dfsg-3+deb11u1) ...
#23 15.39 Setting up libcurl4:arm64 (7.74.0-1.3+deb11u14) ...
#23 15.41 Setting up curl (7.74.0-1.3+deb11u14) ...
#23 15.42 Processing triggers for libc-bin (2.31-13+deb11u3) ...
#23 15.48 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#23 15.62 Segmentation fault (core dumped)
#23 15.63 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#23 15.73 Segmentation fault (core dumped)
Sub-process /usr/bin/dpkg returned an error code (1)
------
Dockerfile:18
--------------------
  16 |     RUN apt update
  17 |     # RUN apt install -y wget
  18 | >>> RUN apt install -y curl
  19 |     # Download the Datadog tracing jar to /app/dd-java-agent.jar
  20 |     RUN mkdir -p /opt/datadog/ && \
--------------------
ERROR: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c apt install -y curl" did not complete successfully: exit code: 100

Reproduction

docker buildx build for a dockerfile that installs wget or curl using a base image of debian bullseye

Version information

 docker version 
Client: Docker Engine - Community
 Version:           24.0.2
 API version:       1.41 (downgraded from 1.43)
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:52:13 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:53 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

moby/buildkit image tag buildx-stable-1 which maps to v0.20.0

[tonistiigi]

This seems to be issue in qemu-9.2.0 . It is recommended to install your own emulators in kernel instead of relying on fallback from buildkit. Some options for that are https://github.com/tonistiigi/binfmt https://github.com/docker/setup-qemu-action . https://hub.docker.com/r/tonistiigi/binfmt has some hints on qemu version selection.

[hegerdes]

I had the same Issue!

While digging down and creating a reproducible build env I tested qemu-v9.2.2⁠, qemu-v8.1.5⁠ and qemu-v7.0.0⁠. Buildkit v0.20.0 all errored with a libc-segfault while buildkit v0.18.2 works fine.

I will continue to test more variations

[luka5]

For the record:

I was facing the same issue today when running apt-get install ... in a Dockerfile.

• Build architecture: amd64
• Building Dockerfile from Ubuntu 22.04
  • Target architecture: amd64 ✔
  • Target architecture: arm64 ❌
• Building Dockerfile from Ubuntu 24.04
  • Target architecture: amd64 ✔
  • Target architecture: arm64 ✔

I did switch from moby v0.20.0 back to v0.19.0. It works again. I did not test qemu-v9.2.2 or newer.

Looking forward for the newer release. Thanks for your good work!

[ggjulio]

I confirm, same issue with an image based on ubuntu 22.04 (envoy) + buildx v0.20.1. ( moby/buildkit:buildx-stable-1@sha256:2c59b0a95f5b2dc103814d69f695a61f131e75f3150ab58ea8afecd75e3d1f9a )

• https://github.com/envoyproxy/envoy/blob/release/v1.30/ci/Dockerfile-envoy#L2

[hegerdes]

I can confirm the obversions from @luka5. Only affected version is buildkit v0.20.0, everything below works just fine.

Another resolution is to change the order of setup-qemu-action and setup-buildx-action. If setup-qemu-action is called first, it even works with buildkit v0.20.0:

steps:
  - name: Checkout code
    uses: actions/checkout@v4
  - name: Set up QEMU
    uses: docker/setup-qemu-action@v3
  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v3

[tonistiigi]

#5808