multi arch support

[crazy-max]

• buildx
• compose
• containerd
• credential-helpers
• docker-cli
• docker-engine
• scan

[crazy-max]

Looks like the base go image (golang:bullseye) being used tamper go env 😐 which breaks rpm packages:

#24 0.236 + go env
#24 0.238 GO111MODULE=""
#24 0.238 GOARCH="s390x"
#24 0.238 GOBIN=""
#24 0.238 GOCACHE="/root/.cache/go-build"
#24 0.238 GOENV="/root/.config/go/env"
#24 0.238 GOEXE=""
#24 0.238 GOEXPERIMENT=""
#24 0.238 GOFLAGS=""
#24 0.238 GOHOSTARCH="s390x"
#24 0.238 GOHOSTOS="linux"
#24 0.238 GOINSECURE=""
#24 0.238 GOMODCACHE="/go/pkg/mod"
#24 0.238 GONOPROXY=""
#24 0.238 GONOSUMDB=""
#24 0.238 GOOS="linux"
#24 0.238 GOPATH="/go"
#24 0.238 GOPRIVATE=""
#24 0.238 GOPROXY="https://proxy.golang.org|direct"
#24 0.238 GOROOT="/usr/local/go"
#24 0.238 GOSUMDB="sum.golang.org"
#24 0.238 GOTMPDIR=""
#24 0.238 GOTOOLDIR="/usr/local/go/pkg/tool/linux_s390x"
#24 0.238 GOVCS=""
#24 0.238 GOVERSION="go1.18.5"
#24 0.238 GCCGO="gccgo"
#24 0.238 AR="ar"
#24 0.238 CC="s390x-linux-gnu-gcc"
#24 0.238 CXX="g++"
#24 0.238 CGO_ENABLED="1"
#24 0.238 GOMOD="/root/rpmbuild/BUILD/src/docker-credential-helpers/go.mod"
#24 0.238 GOWORK=""
#24 0.238 CGO_CFLAGS="-g -O2"
#24 0.238 CGO_CPPFLAGS=""
#24 0.238 CGO_CXXFLAGS="-g -O2"
#24 0.238 CGO_FFLAGS="-g -O2"
#24 0.238 CGO_LDFLAGS="-g -O2"
#24 0.238 PKG_CONFIG="pkg-config"
...
...
#24 1.160 # github.com/docker/docker-credential-helpers/secretservice
#24 1.160 cgo: C compiler "s390x-linux-gnu-gcc" not found: exec: "s390x-linux-gnu-gcc": executable file not found in $PATH
#24 1.161 make: *** [Makefile:20: build-secretservice] Error 2
#24 1.161 error: Bad exit status from /var/tmp/rpm-tmp.levagD (%build)
#24 1.161
#24 1.161 RPM build warnings:
#24 1.161     Missing build-id in /root/rpmbuild/BUILDROOT/docker-credential-pass-0.7.0-1.fc35.s390x/usr/bin/docker-credential-pass
#24 1.161
#24 1.161 RPM build errors:
#24 1.161     Bad exit status from /var/tmp/rpm-tmp.levagD (%build)

Repro with docker run:

$ docker run --rm -t --platform linux/s390x golang:bullseye go env
Unable to find image 'golang:bullseye' locally
bullseye: Pulling from library/golang
cf58bfabf9fb: Pull complete
ce515b66a74a: Pull complete
36bf6be09afa: Pull complete
ff4c0b442ebb: Pull complete
de9b6d274b98: Pull complete
85cd147201cd: Pull complete
e1b78db19f94: Pull complete
Digest: sha256:d92ddd8ad9d960c67dc34cffc2ed7b0ef399be2549505bf5ef94a7f4ca016a05
Status: Downloaded newer image for golang:bullseye
GO111MODULE=""
GOARCH="s390x"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_s390x"
GOVCS=""
GOVERSION="go1.19.1"
GCCGO="gccgo"
AR="ar"
CC="s390x-linux-gnu-gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build671129115=/tmp/go-build -gno-record-gcc-switches"

Looks good with other platforms. I guess that's because for s390x go release is not downloaded from official website but being bootstrapped from sources?

Edit: Hum doesn't look like it: https://github.com/docker-library/golang/blob/2a6ba509d511d58eeb14b81888133745bfe7ef5b/1.18/bullseye/Dockerfile#L56.

Maybe you have an idea @tianon? cc @thaJeztah

[tianon]

Sorry, not sure I understand what's the failure/problem is? The go env in the s390x version of golang:bullseye doesn't look all that much different from the one on amd64 except that CC is set to a more specific (but still technically correct) value? 😅

--- /dev/fd/63	2022-09-23 12:42:00.406387513 -0700
+++ /dev/fd/62	2022-09-23 12:42:00.406387513 -0700
@@ -1,12 +1,12 @@
 GO111MODULE=""
-GOARCH="amd64"
+GOARCH="s390x"
 GOBIN=""
 GOCACHE="/root/.cache/go-build"
 GOENV="/root/.config/go/env"
 GOEXE=""
 GOEXPERIMENT=""
 GOFLAGS=""
-GOHOSTARCH="amd64"
+GOHOSTARCH="s390x"
 GOHOSTOS="linux"
 GOINSECURE=""
 GOMODCACHE="/go/pkg/mod"
@@ -19,13 +19,12 @@
 GOROOT="/usr/local/go"
 GOSUMDB="sum.golang.org"
 GOTMPDIR=""
-GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
+GOTOOLDIR="/usr/local/go/pkg/tool/linux_s390x"
 GOVCS=""
 GOVERSION="go1.19.1"
 GCCGO="gccgo"
-GOAMD64="v1"
 AR="ar"
-CC="gcc"
+CC="s390x-linux-gnu-gcc"
 CXX="g++"
 CGO_ENABLED="1"
 GOMOD="/dev/null"
@@ -36,4 +35,4 @@
 CGO_FFLAGS="-g -O2"
 CGO_LDFLAGS="-g -O2"
 PKG_CONFIG="pkg-config"
-GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3686874489=/tmp/go-build -gno-record-gcc-switches"
+GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4014933557=/tmp/go-build -gno-record-gcc-switches"

(https://packages.debian.org/search?searchon=contents&keywords=s390x-linux-gnu-gcc)

[crazy-max]

Thanks for looking into it @tianon

Sorry, not sure I understand what's the failure/problem is?

The issue is the following when building on native s390x platform (rhel dist) with rpmbuild:

#24 1.160 cgo: C compiler "s390x-linux-gnu-gcc" not found: exec: "s390x-linux-gnu-gcc": executable file not found in $PATH

except that CC is set to a more specific (but still technically correct) value? 😅

Imo it should be CC=gcc like CXX set to g++ by default no? It doesn't seem to be an issue from the golang image though but the official tarball https://dl.google.com/go/go1.18.6.linux-s390x.tar.gz.

Looks to be already reported 👀: golang/go#16443 (comment)

[crazy-max]

Looks to work now with this hack:

if ! command "$(go env CC)" &> /dev/null; then
  go env -w CC=gcc
fi

But don't really like it 😅

[tianon]

golang/go#53383 (comment) is a more recent (and still open!) discussion of the same 👀

[crazy-max]

golang/go#53383 (comment) is a more recent (and still open!) discussion of the same 👀

Ah yes indeed spot on!

[thaJeztah]

golang/go#53383 (comment) is a more recent (and still open!) discussion of the same 👀

Ah yes indeed spot on!

So IIUC, the problem is that the Golang team uses cross-compiling, and because of that their cross-compile toolchain is baked into the code at compile time?

[crazy-max]

So IIUC, the problem is that the Golang team uses cross-compiling, and because of that their cross-compile toolchain is baked into the code at compile time?

@thaJeztah Yes