docker buildx inspect mybuilder --bootstrap doesn't have other platforms

[npu21]

docker buildx inspect mybuilder --bootstrap doesn't have other platforms.
Why there's no Platforms like linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6?
here's snapshot:

root@b6siq9fppm-master-1:~# docker buildx inspect mybuilder --bootstrap
Name: mybuilder
Driver: docker-container
Nodes:
Name: mybuilder0
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64, linux/386

---

root@b6siq9fppm-master-1:~# docker info
Client:
Debug Mode: false
Plugins:
app: Docker Application (Docker Inc., v0.8.0)
buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)
Server:
Containers: 52
Running: 48
Paused: 0
Stopped: 4
Images: 27
Server Version: 19.03.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-45-generic
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.789GiB
Name: inspurtest12-b6siq9fppm-master-1
ID: OOQI:B75G:AO3Z:OAJI:IUEK:LYAL:CZZY:BKOH:SA4P:OSIA:7WUI:X2LO
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
registry.inspurcloud.cn
10.150.0.0/16
127.0.0.0/8
Live Restore Enabled: true
WARNING: No swap limit support

---

root@b6siq9fppm-master-1:~# cat /proc/version
Linux version 4.15.0-45-generic (buildd@lgw01-amd64-031) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019

---

docker buildx version
github.com/docker/buildx v0.3.1-tp-docker 6db68d0

---

ls -al /proc/sys/fs/binfmt_misc/
total 0
drwxr-xr-x 2 root root 0 Dec 11 11:43 .
dr-xr-xr-x 1 root root 0 Dec 11 11:43 ..
-rw-r--r-- 1 root root 0 Dec 11 11:43 python2.7
-rw-r--r-- 1 root root 0 Dec 11 11:43 python3.6
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-aarch64
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-alpha
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-arm
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-armeb
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-cris
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-m68k
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-microblaze
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-mips
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-mips64
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-mips64el
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-mipsel
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-ppc
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-ppc64
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-ppc64abi32
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-ppc64le
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-s390x
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-sh4
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-sh4eb
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-sparc
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-sparc32plus
-rw-r--r-- 1 root root 0 Dec 11 11:43 qemu-sparc64
--w------- 1 root root 0 Dec 11 11:43 register
-rw-r--r-- 1 root root 0 Dec 11 11:43 status

Thanks for any help in advance！

[npu21]

@tonistiigi need your help

[tonistiigi]

moby/buildkit#550

Either there was something wrong with installing qemu-user or you didn't restart/recreate the builder instance after reinstalling. Try something like docker run --rm arm64v8/alpine ls and if it works you just need to recreate or restart the builder instance. If it doesn't then qemu support isn't correctly installed.

[ArturKlauser]

I'm pretty sure that

Operating System: Ubuntu 18.04.2 LTS

does not register the qemu interpreters with update-binfmts ... --fix-binary because the binfmt-support package which contains update-binfmts is too old and does not understand that flag yet. You can check that with cat /proc/sys/fs/binfmt_misc/qemu-aarch64. If flags does not contain F then it's not registered with --fix-binary.

Since your kernel is new enough (>4.8) you could reregister your qemu interpreters with binfmt_misc by hand. One option for that is running

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

which reregisters all the image's qemu interpreters on the host kernel with the --fix-binary flag. The host doesn't actually need the qemu-user-static package installed at all in this case. Note though that you'll need to do this after every reboot since it doesn't stick.

If you have the choice, the easiest solution would be to install a newer version of the OS (e.g. 19.04 should work), or at least install a newer version of binfmt-support (>= 2.1.7) and then reinstall qemu-user-static.

[ArturKlauser]

To expand on my previous answer, and for reference for future users finding this thread while searching for answers to get buildx to work for them, I've written up a post detailing the constraints in the Linux environment that have to be met in order to get multi-architecture buildx to work.

[qlyoung]

@ArturKlauser - awesome post, thank you for that.

Hopefully this isn't too far off the subject of this issue, but while I've got the "basic" set of other platforms available in my builder:

running linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

I'm not seeing the others I've got installed, such as e.g. "vanilla" ppc, which seems to be registered correctly:

# cat /proc/sys/fs/binfmt_misc/qemu-ppc
enabled
interpreter /usr/bin/qemu-ppc-static
flags: OCF
offset 0
magic 7f454c4601020100000000000000000000020014
mask fffffffffffffffcfffffffffffffffffffefff

I've tried restarting docker and creating a new builder but they aren't showing up. What have I done wrong?

Edit:
On Ubuntu 20.04, qemu-user-static includes all of the more exotic emulators registered with F by default and I'm still seeing the same behavior with a new builder.

[ArturKlauser]

@qlyoung - The list of platforms you see in your builder is the intersection of the platforms for which you have installed QEMU support, and the list of platforms that buildx supports. I think the list you gave

linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

includes almost all platforms supported by buildx. The only ones that seem missing are linux/arm/v5 and linux/mips64le. So even though you might have QEMU support for other, additional platforms, buildx isn't going to start to use them.

• https://github.com/docker-library/official-images#architectures-other-than-amd64
• https://github.com/docker-library/official-images#multiple-architectures
• https://github.com/docker-library/bashbrew/blob/v0.1.0/vendor/github.com/docker-library/go-dockerlibrary/architecture/oci-platform.go#L14-L26