cmd/link: unexpected trampoline error on ppc64le musl with -buildmode=pie

[nmeum]

This is (probably) related to #52336 and #51787.

What version of Go are you using (go version)?

$ go version
go version go1.18.1 linux/ppc64le

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="ppc64le"
GOBIN=""
GOCACHE="/home/buildozer/.cache/go-build"
GOENV="/home/buildozer/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="ppc64le"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/buildozer/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/buildozer/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_ppc64le"
GOVCS=""
GOVERSION="go1.18.1"
GCCGO="gccgo"
GOPPC64="power8"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1359095760=/tmp/go-build -gno-record-gcc-switches"

What did you do?

This is issue we encountered when packaging Go software for Alpine Linux Edge ppc64le. On all other architectures, supported by Alpine, the affected Go software builds fine but on ppc64le it fails with a "unexpected trampoline for shared or dynamic linking" error (see also #52336 and #51787).

This issue can only be reproduced with -buildmode=pie.

One software where this can be reproduced is vault (I have not yet been able to create a minimal example):

$ git clone https://github.com/hashicorp/vault
$ cd vault
$ go build -buildmode=pie -o test

See also #51787 (comment)

What did you expect to see?

A successful build.

What did you see instead?

The following error message:

k8s.io/api/extensions/v1beta1.(*ScaleStatus).MarshalToSizedBuffer: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*ScaleStatus).MarshalToSizedBuffer: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*ScaleStatus).MarshalToSizedBuffer: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*ScaleStatus).MarshalToSizedBuffer: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DeploymentRollback).Size: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DeploymentRollback).Size: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*ScaleStatus).Size: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*ScaleStatus).Size: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*AllowedCSIDriver).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*AllowedFlexVolume).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*AllowedHostPath).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSet).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSetCondition).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSetCondition).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSetCondition).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSetCondition).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSetCondition).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSetList).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSetSpec).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSetSpec).String: unexpected trampoline for shared or dynamic linking
k8s.io/api/extensions/v1beta1.(*DaemonSetSpec).String: unexpected trampoline for shared or dynamic linking

---

CC: @pmur, @cherrymui

[cherrymui]

@pmur
https://cs.opensource.google/go/go/+/master:src/cmd/link/internal/ppc64/asm.go;l=823 seems to refer to early return at https://cs.opensource.google/go/go/+/master:src/cmd/link/internal/ppc64/asm.go;l=768 , which only returns when external linking. Now that we have internal PIE linking, maybe that needs revisit. Do we need a different trampoline for PIE?

[pmur]

Yes, we should be generating TOC-relative trampolines when internal linking PIE. I will prepare a patch.

[gopherbot]

Change https://go.dev/cl/400234 mentions this issue: cmd/link: use TOC-relative trampolines on PPC64 when needed

[sergiodj]

I'm working on backporting the patch that fixes this issue to golang 1.18.2, but I'm experiencing some problems. Maybe someone will have an idea here.

FWIW, I'm working on backporting this patch into Ubuntu's golang for its Kinetic series (development release), because the golang currently shipped is affected by this bug on ppc64le (which is ultimately impacting other software that use PIE during build).

After backporting this patch, the bug seems to be resolved for those software and I can get a build to complete; however, I start experiencing problems when I try to build golang 1.18.2 with itself (containing the patch mentioned above):

$ (export PATH="$PWD/bin:$PATH"; cd src; eval "$(go tool dist env)"; go tool dist test -v -run='^go_test:cmd/link$')

##### Testing packages.
# go tool dist test -run=^go_test:cmd/link$
--- FAIL: TestTrampoline (0.85s)
    link_test.go:677: executable failed to run (pie): signal: segmentation fault (core dumped)
    link_test.go:680: unexpected output (pie):
--- FAIL: TestTrampolineCgo (2.67s)
    link_test.go:754: executable failed to run (pie): signal: segmentation fault (core dumped)
    link_test.go:757: unexpected output (pie):
FAIL
FAIL    cmd/link        2.847s
FAIL
# go tool dist test -run=^go_test:cmd/link$
go tool dist: Failed: exit status 1

I'm trying to understand what's going on bug my golang-fu is not great.

I cloned this repository and analysed the commits that differ between HEAD and go1.18.2 inside the src/cmd/link/ directory, to check if something else caught my attention. There is commit 12420f9, but it doesn't seem to be related to the error I'm seeing.

Maybe I'm missing something here. FWIW, I'm not compiling golang with PIE. I'm happy to provide more info if needed, and/or to file a new bug if that's deemed necessary. Thanks in advance.

[pmur]

Hi @sergiodj, I'll check to make sure there aren't any soft-dependencies which also need backported. That test is one which forces trampoline generation while using -buildmode=pie. Maybe this is enough justification for a backport?

@gopherbot please consider this for backporting to 1.18.

[gopherbot]

Backport issue(s) opened: #53107 (for 1.18).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[sergiodj]

Hey @pmur, thanks for looking into this. I will try to block some time during the day today to continue the investigation as well, and will let you know if I find something.

[pmur]

Ah, you will need to backport https://go-review.googlesource.com/c/go/+/400574 too. That should unblock you. It is a soft-dependency. @cherrymui provided the justification for the backport is OK, is your change OK to include in a backport?

[cherrymui]

@pmur yeah, I think so. Thanks for looking into it.

[sergiodj]

@pmur Aha, thank you for the pointer. Indeed, after backporting the patch you mentioned I can now build golang itself. Thanks!