Add (experimental) buildx support to allow simple multiarch builds

[marklagendijk]

The issue
Creating multiarch images is so hard, that only very big projects do it. Even those projects often use an outdated approach: special tag based instead of manifest based.

Possible solution 1
Add (experimental) support for buildx on Docker Hub. Adding it already as experimental feature has the following advantages:

1. Many people will use it. So lots of feedback.
2. For many cases it will just work. So lots of images will already start to be published as multiarch.
3. Adding support for the eventual, stable solution will be easier.

If I understand things correctly this could work with the existing setup, using strategy 1: "Using the QEMU emulation support in the kernel".

Possible solution 2
Create examples on how to use buildx with the current Docker Hub (downloading and using it manually, via hooks).

Background story
As a developer / devops I love Docker. Although one needs to learn a new set of knowledge about how Docker and related tools work, a whole class of other issues is abstracted away.
When I wanted to put a new open source project on Docker Hub, I was pleasantly surprised by automated Docker Hub builds:

1. I didn't know it existed.
2. When I used it, I was amazed by how easy it was to setup.
3. I saw that all the options one would normally want, where right there, and easy to use.

Then I looked for settings about multiarch. They weren't there. I Googled on how to achieve this. I found that it was complicated to solve, and required quite some extra knowledge and setup.

It made me sad.

I really appreciate Docker and all the work being done to make it awesome. I understand that some things (like multiarch) are very complex, and I appreciate that you are working on hiding the complexity from the users.

[jmb12686]

I am also interested in getting the "buildx" feature added to Docker Hub Automated Builds. Right now I am using 'hooks' to override the build phase, manually create manifests, etc in Docker Hub Automated Builds. I would prefer to use buildx, but not sure if it's available in the Docker Hub build environment?

[jmb12686]

Looks like 'buildx' plugin can be enabled and used on Docker Hub Automated Builds, but upon inspection during a pre_build hook, it appears that the supported platforms only includes linux/amd64, linux/386. It seems the Docker Hub Build Server is on version 18.03.x-ee, and would need updated to 19.03+ if I am not mistaken. This seems to be something that the Docker Inc team would need to update, since I manually update what I can in my hook.

See Docker Hub build logs, after executing the pre_build hook in my repo here:
Client:
Debug Mode: false
Plugins:
template: Use templates to quickly create new services or applications (Docker Inc., v0.1.5)
app: Docker Application (Docker Inc., v0.8.0)
assemble: assemble is a high-level build tool (Docker Inc., v0.36.0)
buildx: Build with BuildKit (Docker Inc., v0.3.0-5-g5b97415-tp-docker)
cluster: Manage Docker clusters (Docker Inc., v1.1.0-8c33de7)
registry: Manage Docker registries (Docker Inc., 0.1.0)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 18.03.1-ee-3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-1060-aws
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.675GiB
Name: ip-10-66-191-57
ID: ZILC:EWFH:3J2N:GEER:KIVG:JPWW:VQGS:NJN7:LIU3:66ON:BRVK:FCTE
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 31
Goroutines: 42
System Time: 2019-09-11T14:37:48.340098508Z
EventsListeners: 0
Username: jmb12686
Registry: https://index.docker.io/v1/
Labels:
com.docker.security.apparmor=enabled
com.docker.security.seccomp=enabled
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Unable to find image 'multiarch/qemu-user-static:register' locally
register: Pulling from multiarch/qemu-user-static
ee153a04d683: Pulling fs layer
84cc22363d00: Pulling fs layer
96aa3d7b9b30: Pulling fs layer
8d669bf48302: Pulling fs layer
8d669bf48302: Waiting
ee153a04d683: Verifying Checksum
ee153a04d683: Download complete
84cc22363d00: Verifying Checksum
84cc22363d00: Download complete
96aa3d7b9b30: Verifying Checksum
96aa3d7b9b30: Download complete
8d669bf48302: Verifying Checksum
8d669bf48302: Download complete
ee153a04d683: Pull complete
84cc22363d00: Pull complete
96aa3d7b9b30: Pull complete
8d669bf48302: Pull complete
Digest: sha256:c77eb2da3597aa370f07ef970e2e0adf155172eb9d3c40e43d97aa43eef6b0c9
Status: Downloaded newer image for multiarch/qemu-user-static:register
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
################## start buildx tasks ##################
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
default * docker-container
default default inactive
################## Create new buildx builder ##################
mybuilder
################## Use new buildx builder ##################
################## Inspect buildx builder ##################
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 12.6s done
#1 creating container buildx_buildkit_mybuilder0
#1 creating container buildx_buildkit_mybuilder0 1.7s done
#1 DONE 14.3s
Name: mybuilder
Driver: docker-container
Nodes:
Name: mybuilder0
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64, linux/386
################## done with pre_build ##################

[marklagendijk]

I have not yet been able to get multiarch builds working locally. I haven't found comprehensive instructions, yet. At least the following is needed:

• Enable experimental feature support in the Docker daemon: /etc/docker/daemon.json: { "experimental": true }
• Enable experimental feature support in the Docker CLI: ~/.docker/config.json: { "experimental": "enabled" }. Alternatively set the environment variable: DOCKER_CLI_EXPERIMENTAL=enabled.
• Have QEMU emulation setup properly:
  • sudo apt install qemu-user-static. Apparently version 1:2.11 is not new enough, but 1:2.12 is. See this issue.
• Checking that QEMU support works: docker run --rm arm64v8/alpine ls
• Create a builder with docker buildx create --name mybuilder --use
• Checking that the builder supports alternative platforms: buildx inspect --bootstrap

I got stuck at setting up QEMU emulation support, when trying this locally, because I got the older version. At this moment I'm not sure where the limitation with Docker Hub lies, and whether this limitation can be overcome.

[ruimarinho]

I'm also looking for this. Right now I'm trying to understand if Docker 19.03+ has already been made available on Docker Hub, but the platform is very, very slow.

[kaisawind]

the docker version is 18.03.1 on Docker Hub.It does not support buildx. I think the first thing is to upgrade docker to 19.03+.By the way, an other to build multiarch images is to use multiarch/qemu-user-static with docker hub hook.But this is not good.

KernelVersion: 4.4.0-1060-aws
Arch: amd64
BuildTime: 2018-08-30T18:42:30.000000000+00:00
ApiVersion: 1.37
Platform: {u'Name': u''}
Version: 18.03.1-ee-3
MinAPIVersion: 1.12
GitCommit: b9a5c95
Os: linux
GoVersion: go1.10.2

multiarch/qemu-user-static with docker hub hook

.
├── amd64
│   └── Dockerfile
└── arm64
    ├── Dockerfile
    └── hooks
        └── pre_build

pre_build

docker run --rm --privileged multiarch/qemu-user-static:register --reset

arm64/Dockerfile

FROM multiarch/qemu-user-static:x86_64-aarch64 as qemu
FROM arm64v8/golang:alpine AS builder
COPY --from=qemu /usr/bin/qemu-aarch64-static /usr/bin
RUN apk add upx make git
RUN make
FROM arm64v8/alpine
COPY --from=builder /home/bin/main /

[hwdsl2]

Docker Hub automated builds seem to now run Docker version 19.03.8. This could potentially enable the use of buildx for multiarch builds.

[timbru31]

I’ve messed around with it last weekend, but couldn’t get the buildx worker to support something different than amd64 or i386. I’ve even installed qemu manually and not via apt

[scyto]

@timbru31 what approach did you experiment with? (hooks are way beyond my skill and overly complex for basic multiarch builds. I have used it locally on docker desktop and it works brilliantly).

I assume for docker hub to use buildx it would:

• have experimental be enabled
• have a new buildX instance created:

docker buildx create --name multiarchbuilder
docker buildx use multiarchbuilder
docker buildx inspect --bootstrap

then after this one needs to execute ones docker buildx command line?

did you manage to run these commands in dockerhub somehow?

[timbru31]

@scyto I've used hooks to execute the required bash commands first (apt-get update, install qemu, enable experimental support). And yes those commands did run successfully, but no other arch support was present.

I've switched to GitHub Actions in the meanwhile, it's 100000% faster, too. (example: https://github.com/timbru31/docker-java-node/blob/master/.github/workflows/buildx.yml)

[arkodg]

folks, you can now build with buildkit on Hub by setting DOCKER_BUILDKIT=1 in the env setting. Here's more info https://docs.docker.com/docker-hub/builds/#build-images-with-buildkit
Multiarch builds in on our roadmap as well - docker/roadmap#109

[marklagendijk]

@arkodg would multiarch builds already work with this?

[arkodg]

@marklagendijk, not supported yet, but its part of the roadmap

[sebdanielsson]

folks, you can now build with buildkit on Hub by setting DOCKER_BUILDKIT=1 in the env setting. Here's more info https://docs.docker.com/docker-hub/builds/#build-images-with-buildkit
Multiarch builds in on our roadmap as well - docker/roadmap#109

If I would like to build for both amd64 and arm64, how would I do that? I've added a BUILD ENVIRONMENT VARIABLE on Docker Hub.

DOCKER_BUILDKIT=1

How do I specify which architectures it should build for?

[github-actions]

We are clearing up our old issues and your ticket has been open for 6 months with no activity. Remove stale label or comment or this will be closed in 15 days.

[sebdanielsson]

Any updates on this? A list of known blockers?

[ColinM9991]

Any updates on this? A list of known blockers?

It's unlikely you'll hear anything back. This wouldn't be the first issue to go stale since they rarely address any of these issues.

I'm relatively new to Docker but have learned that the best way to handle an issue is to simply "deal with it". Support is non existent.

[NiklasBr]

Any updates on this? A list of known blockers?