cmd/link: relocation error on ppc64le musl with cgo and CGO_CFLAGS=-Os

[nmeum]

This is a follow-up for #51787.

What version of Go are you using (go version)?

$ go version
go version go1.18.1 linux/ppc64le

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/home/buildozer/.cache/go-build"
GOENV="/home/buildozer/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/buildozer/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/buildozer/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_ppc64le"
GOVCS=""
GOVERSION="go1.18.1"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1359095760=/tmp/go-build -gno-record-gcc-switches"

What did you do?

This is issue we encountered when packaging Go software for Alpine Linux Edge ppc64le. On all other architectures, supported by Alpine, the affected Go software builds fine but on ppc64le it fails with a relocation error (which makes me think that this is related to #51787).

This issue only seems to appear with -Os in the CGO_CFLAGS environment variable (keep in mind though many wrappers around go build, e.g. Makefiles, set CGO_FLAGS=$CFLAGS). Without setting CGO_CFLAGS the software builds fine.

For example:

$ git clone https://github.com/maximbaz/yubikey-touch-detector
$ cd yubikey-touch-detector
$ CGO_CFLAGS=-Os go build -v

I have not yet been able to produce a minimal example but this seems to affect a lot of software #51787 (comment) has more examples. Let me know if you need me to provide more information.

What did you expect to see?

A successful build.

What did you see instead?

os/user
net
crypto/x509
github.com/godbus/dbus/v5
crypto/tls
github.com/esiqveland/notify
github.com/coreos/go-systemd/v22/activation
github.com/maximbaz/yubikey-touch-detector/notifier
github.com/maximbaz/yubikey-touch-detector/detector
github.com/maximbaz/yubikey-touch-detector
# github.com/maximbaz/yubikey-touch-detector
net(.text): relocation target _savegpr0_28 not defined
net(.text): relocation target _restgpr0_28 not defined
net(.text): relocation target _savegpr0_29 not defined
net(.text): relocation target _restgpr0_29 not defined
runtime/cgo(.text): relocation target _restgpr0_30 not defined
runtime/cgo(.text): relocation target _restgpr0_31 not defined
runtime/cgo(.text): relocation target _savegpr0_23 not defined
runtime/cgo(.text): relocation target _restgpr0_23 not defined
runtime/cgo(.text): relocation target _restgpr0_30 not defined
runtime/cgo(.text): relocation target _savegpr0_29 not defined
runtime/cgo(.text): relocation target _restgpr0_29 not defined
runtime/cgo(.text): relocation target _savegpr0_25 not defined
runtime/cgo(.text): relocation target _restgpr0_25 not defined
runtime/cgo(.text): relocation target _savegpr0_28 not defined
runtime/cgo(.text): relocation target _restgpr0_28 not defined
runtime/cgo(.text): relocation target _restgpr0_31 not defined
os/user(.text): relocation target _restgpr0_30 not defined
os/user(.text): relocation target _savegpr0_29 not defined
os/user(.text): relocation target _restgpr0_29 not defined

---

CC: @pmur, @cherrymui

[cherrymui]

Without setting CGO_CFLAGS the software builds fine.

Interesting. Presumably this causes the C compiler to emit symbols (like _savegpr0_28) that the Go linker doesn't currently handle.

A workaround is to try go build -ldflags=-linkmode=external. Would that be an option?

On #51787 (comment) you also mentioned "unexpected trampoline for shared or dynamic linking" errors. When/how could that happen? Could you provide the command line for that one? Thanks.

[nmeum]

A workaround is to try go build -ldflags=-linkmode=external. Would that be an option?

That does seem to work, yes.

On #51787 (comment) you also mentioned "unexpected trampoline for shared or dynamic linking" errors. When/how could that happen? Could you provide the command line for that one? Thanks.

I debugged this a bit further and since it seems that different flags are needed to reproduce the two I decided to create two separate issues. I hope that's ok. The other issue is #52337.

[pmur]

This is a PPC64 (and likely -Os) only quirk. binutils ld generates these symbols. Also, I noticed the project in question uses a makefile which explicitly requests external linking. @nmeum is alpine using the project supplied makefile?

@Cherry should I be matching this behavior in the internal linker? These functions are part of the PPC64 ELF ABIs.

[cherrymui]

I think it is okay to require external linking for some C compiler flags. However, it would be nice to support ones that are system default. How complex that would be to add the support? Thanks.

[nmeum]

This is a PPC64 (and likely -Os) only quirk. binutils ld generates these symbols. Also, I noticed the project in question uses a makefile which explicitly requests external linking. @nmeum is alpine using the project supplied makefile?

Yep, Alpine is using CFLAGS=-Os by default and uses the Makefile provided by the project which sets CGO_CFLAGS=$CFLAGS. It is not just this particular project though, we seem to have quite a few Go packages in Alpine which sets CGO_CFLAGS in a similar manner.

[pmur]

I think it is feasible support generating those ABI functions when linking internally. I'll make a patch.

[gopherbot]

Change https://go.dev/cl/400796 mentions this issue: cmd/link: generate ABI save/restore functions if needed

[gopherbot]

Change https://go.dev/cl/400796 mentions this issue: cmd/link: generate PPC64 ABI save/restore functions if needed

[gopherbot]

Change https://go.dev/cl/401114 mentions this issue: cmd/link: generate ABI register save/restore functions if needed