cmd/link: issues with Apple's new linker in Xcode 15 beta

[cherrymui]

NOTE -- please read

If you see lines starting with ld: warning but not other outputs, they are just warnings, not errors. You should still get a working binary, if there is no other output. Please only report if you see a link error, or a broken binary, or a warning that is not mentioned in the "warnings" section below. Thanks.

See #61229 (comment) for more notes about malformed LC_DYSYMTAB warning. Thanks.

---

In Xcode 15 beta it comes with Apple's new linker, ld-prime. As of beta 3 it is enabled by default. I did some experiments and found some issues for Go (cgo) with the new Apple linker. Some of the issues are reported in #61190. I'm filing a new issue to track all related faulures.

warnings

Apparently the new linker is more picky. It issues some warnings that the old linker (ld64) doesn't. This includes

ld: warning: -bind_at_load is deprecated on macOS

ld: warning: search path '/usr/local/lib' not found

ld: warning: ignoring duplicate library '-lm'

ld: warning: '.../go.o' has malformed LC_DYSYMTAB, expected 92 undefined symbols to start at index 15983, found 102 undefined symbol

These are just warnings and don't affect the correctness of the build. I'm preparing CLs for fix/work around them.

offset/addend being dropped

Mostly for c-archive, c-shared, and plugin build mode, for some relocations it seems the offset/addend being dropped and the relocation is resolved to the beginning of the section. This may cause failures like SIGILL. E.g.

% go test cmd/cgo/internal/testcarchive                 
--- FAIL: TestInstall (1.10s)
    carchive_test.go:487: [go tool cgo -objdir /var/folders/tf/nsc4z0p112vc_drrjn299ds80000gn/T/carchive_test3584575482/_obj472038228 -exportheader p.h p/p.go]
    carchive_test.go:489: [go install -buildmode=c-archive ./libgo]
    carchive_test.go:489: [clang -fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/tf/nsc4z0p112vc_drrjn299ds80000gn/T/go-build2775077316=/tmp/go-build -gno-record-gcc-switches -fno-common -I /var/folders/tf/nsc4z0p112vc_drrjn299ds80000gn/T/carchive_test3584575482/pkg/darwin_arm64_shared/testcarchive -o ./testp1 main.c main_unix.c /var/folders/tf/nsc4z0p112vc_drrjn299ds80000gn/T/carchive_test3584575482/pkg/darwin_arm64_shared/testcarchive/libgo.a]
    carchive_test.go:489: 
    carchive_test.go:489: signal: illegal instruction
...

#60694 is the same failure mode.

debug info issues

The new linker doesn't generate stab symbols for our functions, causing debug info being lost. Failure look like

--- FAIL: TestDWARF (0.00s)
    --- FAIL: TestDWARF/testprogcgo (3.85s)
        dwarf_test.go:171: ErrUnknownPC
FAIL
FAIL	cmd/link	8.822s

There is also issues with combining DWARF into executable/shared object. Currently it is affecting c-shared build mode. ld-prime rejects the shared object if we add a DWARF segment to it.

non-PIE build doesn't work (Intel Mac only)

The new linker silently ignores the -no_pie flag and generates a PIE binary, which won't work because we assumed some addresses are not relocated. CL https://go.dev/cl/461697 changes the default to PIE and CL https://go.dev/cl/511355 contains a workaround if we are asked to build a non-PIE binary.

fixes/workarounds

CL https://golang.org/cl/505415 and the stack, as well as other CLs linked in this issue, contain tentative fixes and workarounds that address some of the issues. I'll also work with Apple to have them addressed on their side.

I'm hoping o have them addressed before Apple's new linker is finally released.

[ismail]

I also have an ld assertion failure (reported to Apple as FB12550006):

❯ go install -ldflags "-s -w" github.com/rclone/rclone@master
# github.com/rclone/rclone
/opt/homebrew/Cellar/go/1.20.5/libexec/pkg/tool/darwin_arm64/link: running cc failed: exit status 1
ld: warning: -bind_at_load is deprecated on macOS
ld: warning: '/private/var/folders/68/1pjj14514sl3vj7lbt69lst40000gn/T/go-link-876362130/go.o' has malformed LC_DYSYMTAB, expected 152 undefined symbols to start at index 168240, found 170 undefined symbols starting at index 135
0  0x104a50380  __assert_rtn + 72
1  0x104a099bc  ___ZN2ld16LayoutExecutable27writeContentWithoutLinkEditENSt3__14spanIhLm18446744073709551615EEEy_block_invoke + 9552
2  0x184981950  _dispatch_client_callout2 + 20
3  0x184994ba0  _dispatch_apply_invoke + 176
4  0x184981910  _dispatch_client_callout + 20
5  0x1849933cc  _dispatch_root_queue_drain + 864
6  0x184993a04  _dispatch_worker_thread2 + 156
7  0x184b2b0d8  _pthread_wqthread + 228
ld: Assertion failed: (addr + content.size() <= sectionEndAddr), function writeContentWithoutLinkEdit_block_invoke, file Layout.cpp, line 5689.
clang: error: linker command failed with exit code 1 (use -v to see invocation)

❯ go version
go version go1.20.5 darwin/arm64

❯ clang -v
Apple clang version 15.0.0 (clang-1500.0.34.3)
Target: arm64-apple-darwin23.0.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[cherrymui]

@ismail Thanks for the report. The CL stack mentioned above (https://golang.org/cl/505415 and dependencies) also avoids the assertion failure.

[gopherbot]

Change https://go.dev/cl/503935 mentions this issue: cmd/link: always use symbol-targeted relocations on Mach-O

[gopherbot]

Change https://go.dev/cl/503538 mentions this issue: cmd/link: fix up more zero-sized local symbols on darwin dynamic linking

[gopherbot]

Change https://go.dev/cl/502616 mentions this issue: cmd/link: use symbol-targeted relocation for initializers on Mach-O

[gopherbot]

Change https://go.dev/cl/502617 mentions this issue: cmd/link: don't generate DYSYMTAB when external linking on Mach-O

[gopherbot]

Change https://go.dev/cl/503539 mentions this issue: cmd/link: sort HOSTOBJ and UNDEFEXT symbols with undefined symbols in Mach-O symbol table

[gopherbot]

Change https://go.dev/cl/505415 mentions this issue: cmd/link: work around issues with ld-prime

[gopherbot]

Change https://go.dev/cl/508696 mentions this issue: cmd/link: suppress some warnings for ld-prime

[gopherbot]

Change https://go.dev/cl/508697 mentions this issue: cmd/cgo/internal/test: don't pass -lm on darwin

[cherrymui]

Another failure mode: (AMD64 Mac only, not ARM64) as of Xcode 15 beta 4 (dyld-1009.6 BUILD 20:09:19 Jul 7 2023) non-PIE binary linked with ld-prime crashes at startup.

runtime: pcHeader: magic= 0xfffffff1 pad1= 0 pad2= 0 minLC= 1 ptrSize= 8 pcHeader.textStart= 0x0 text= 0x10b865000 pluginpath= 
fatal error: invalid function symbol table
runtime: panic before malloc heap initialized

runtime stack:
runtime.throw({0x10b8d04de?, 0x7ff7b469e7a0?})
	/Users/cherryyz/src/go/src/runtime/panic.go:1077 +0x5c fp=0x7ff7b469e420 sp=0x7ff7b469e3f0 pc=0x10b88ffdc
runtime.moduledataverify1(0x7ff7b469e560?)
	/Users/cherryyz/src/go/src/runtime/symtab.go:533 +0x816 fp=0x7ff7b469e540 sp=0x7ff7b469e420 pc=0x10b8aa376
runtime.moduledataverify(...)
	/Users/cherryyz/src/go/src/runtime/symtab.go:519
runtime.schedinit()
	/Users/cherryyz/src/go/src/runtime/proc.go:724 +0x34 fp=0x7ff7b469e580 sp=0x7ff7b469e540 pc=0x10b8939b4
runtime.rt0_go()
	/Users/cherryyz/src/go/src/runtime/asm_amd64.s:349 +0x11e fp=0x7ff7b469e588 sp=0x7ff7b469e580 pc=0x10b8b859e

And the CLs above are not enough to fix it. I'm looking into it.
Possibly related to the crash in #61190, not investigated yet.

(PIE binaries are fine. Also fine on ARM64 as it is always PIE.)

[piusalfred]

I am on MacOS Sonoma beta M1,Xcode 15 beta. I use go.120. When I try to install gopls this is what I get

# golang.org/x/tools/cmd/goimports
ld: warning: '/private/var/folders/jc/lq1rvy214tq4643flbh887xr0000gn/T/go-link-1689581912/go.o' has malformed LC_DYSYMTAB, expected 75 undefined symbols to start at index 6656, found 84 undefined symbols starting at index 53

[gopherbot]

Change https://go.dev/cl/461697 mentions this issue: cmd/go: default to PIE linking on darwin/amd64

[gopherbot]

Change https://go.dev/cl/511355 mentions this issue: cmd/link: force ld64 when building non-PIE binary on macOS

[cherrymui]

@piusalfred that is just a warning, not an error. You should still get a binary. Does the binary run? If not, see #61190, and post on that issue if you have any update. Thanks!

[cherrymui]

The non-PIE binary crash (#61229 (comment)) is due to ld-prime silently ignores -no_pie flag and generates a PIE binary instead. But that doesn't work because in exe buildmode we assumed the binary is not relocated and so generated non-relocated addresses, which would not be correct if the binary is actually relocated.

The CLs above handle it: 1. switch to PIE by default, 2. force ld64 in exe buildmode.

(#61190 is not this. Still need to investigate.)

[cobo-louis]

/usr/local/go/pkg/tool/darwin_arm64/link: running clang failed: exit status 1
ld: warning: '/private/var/folders/7n/lt8klzz95mj0kn2fx631dt740000gp/T/go-link-2991864128/go.o' has malformed LC_DYSYMTAB, expected 125 undefined symbols to start at index 56533, found 206 undefined symbols starting at index 142
0 0x100ccee8a __assert_rtn + 64
1 0x100c7693c ___ZN2ld16LayoutExecutable27writeContentWithoutLinkEditENSt3__14spanIhLm18446744073709551615EEEy_block_invoke + 10316
2 0x7ff802dd6066 _dispatch_client_callout2 + 8
3 0x7ff802de918f _dispatch_apply_invoke_and_wait + 213
4 0x7ff802de8692 _dispatch_apply_with_attr_f + 1207
5 0x7ff802de8847 dispatch_apply + 45
6 0x100c76cf9 void mapReduce<ld::Atom const*, mach_o::Error>(std::__1::span<ld::Atom const*, 18446744073709551615ul>, unsigned long, void (unsigned long, mach_o::Error&, std::__1::span<ld::Atom const*, 18446744073709551615ul>) block_pointer, void (std::__1::span<mach_o::Error, 18446744073709551615ul>) block_pointer) + 745
7 0x100c7e218 ld::LayoutExecutable::writeToFile(char const*) + 22792
8 0x100c1d686 main + 12838
ld: Assertion failed: (0 && "Kind::arm64_adrp_ldr missing extra info"), function applyFixup, file Fixup.cpp, line 743.
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)

[cherrymui]

@cobo-louis could you share the program you are building and the steps to reproduce the failure? Then we can report it to Apple. This is a bug in Apple's new linker. Thanks.

[cherrymui]

The warning on the race C object (the syso file) is due to this code in LLVM TSAN:
https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/sanitizer_common/sanitizer_redefine_builtins.h#L20-L22

asm("memcpy = __sanitizer_internal_memcpy");
asm("memmove = __sanitizer_internal_memmove");
asm("memset = __sanitizer_internal_memset");

Apparently, on Darwin, that caused the object to include weird symbols like (objdump -t output)

00000000000000f3         *UND* memcpy
0000000000006331         *UND* memmove
0000000000002b25         *UND* memset

They are "undefined" symbols but have values (unlike all other UND symbols). And apparently the new Apple linker doesn't like them and warns.

If I comment those three lines off, it no longer generates those symbols and no more warnings from the linker.

On Darwin, at assembler level, all C symbols are prefixed by _. So those lines are probably already not doing anything useful on Darwin. Perhaps those lines should be

asm("_memcpy = ___sanitizer_internal_memcpy");
asm("_memmove = ___sanitizer_internal_memmove");
asm("_memset = ___sanitizer_internal_memset");

on Darwin?
That also makes the weird symbols go away, as well as the warning.
cc @dvyukov (These symbols are from https://reviews.llvm.org/D151152 . You probably have some context. Thanks!)

[gabkov]

Hi,

I am trying to install a cli package but i receive the following error and the package is not installed:

➜  ~ go install github.com/ethereum/go-ethereum/cmd/abigen@latest
# github.com/ethereum/go-ethereum/cmd/abigen
ld: warning: '/private/var/folders/nh/78bdkps97rv6pdws74j1xy0w0000gn/T/go-link-1841817092/go.o' has malformed LC_DYSYMTAB, expected 121 undefined symbols to start at index 30647, found 165 undefined symbols starting at index 96

I am not sure how this is xcode related however I used xcode before. Any help appreciated.

[nebiros]

Hi,

I am trying to install a cli package but i receive the following error and the package is not installed:

➜  ~ go install github.com/ethereum/go-ethereum/cmd/abigen@latest
# github.com/ethereum/go-ethereum/cmd/abigen
ld: warning: '/private/var/folders/nh/78bdkps97rv6pdws74j1xy0w0000gn/T/go-link-1841817092/go.o' has malformed LC_DYSYMTAB, expected 121 undefined symbols to start at index 30647, found 165 undefined symbols starting at index 96

I am not sure how this is xcode related however I used xcode before. Any help appreciated.

Try adding that env var: #61229 (comment) before running go install

[cherrymui]

@gabkov your build should have succeeded and the binary installed. As mentioned above, it is just a warning, not an error.

That said, which version of Go are you using? I would not expect a warning like this for this particular build with recent versions of Go.

[tkgalk]

Still seeing those warnings on 1.22 installed today on a Sonoma 14.3.1 M1.

[gabkov]

Hi,
I am trying to install a cli package but i receive the following error and the package is not installed:

➜  ~ go install github.com/ethereum/go-ethereum/cmd/abigen@latest
# github.com/ethereum/go-ethereum/cmd/abigen
ld: warning: '/private/var/folders/nh/78bdkps97rv6pdws74j1xy0w0000gn/T/go-link-1841817092/go.o' has malformed LC_DYSYMTAB, expected 121 undefined symbols to start at index 30647, found 165 undefined symbols starting at index 96

I am not sure how this is xcode related however I used xcode before. Any help appreciated.

Try adding that env var: #61229 (comment) before running go install

Thank you for the suggestion @nebiros, but unfortunately setting the env var only hid the warning and the package was still not installed.

@gabkov your build should have succeeded and the binary installed. As mentioned above, it is just a warning, not an error.

That said, which version of Go are you using? I would not expect a warning like this for this particular build with recent versions of Go.

@cherrymui i was using 1.20.6, but today i upgraded to 1.22.0 and the warning is gone and i was able to install the package as well. Not sure if it's just a local issue or related to the actual go version. Thanks anyway.

[wendorf]

I've been able to reproduce the "malformed LC_DYSYMTAB" warning on my Macbook Pro M2 Max running Sonoma 14.3.1 and go 1.22.1. I do not see the warning at all when using go 1.21.8.

The best reproducible test case I've been able to come up with is running tests on https://github.com/DataDog/zstd , ref 869dae002e5efb372a0b09cd7d99390ca2089cc1:

❯ go test -race .
# github.com/DataDog/zstd.test
ld: warning: '/private/var/folders/mf/75msnn554t940_ky9yrfzfcw0000gn/T/go-link-98474360/000055.o' has malformed LC_DYSYMTAB, expected 98 undefined symbols to start at index 1626, found 95 undefined symbols starting at index 1626
ok  	github.com/DataDog/zstd	(cached)

When I fully remove the C code and edit the Go code to not use the C package, the warning is not present. Unfortunately, I'm unable to figure out what about the C code is causing this problem.

[cherrymui]

@wendorf as mentioned above #61229 (comment) , the warning is for the C code in the race detector (LLVM TSAN) itself. The only difference that user C code made is that it changes the default link mode. Without user C code, the Go linker by default does all the linking itself, without using the C linker (Apple's linker). When user C code is present, it uses the C linker by default. You can try passing -ldflags=-linkmode=internal to force using the Go linker to do the entire linking (although it is not guaranteed to be able to handle all C objects).

[cherrymui]

A note about the malformed LC_DYSYMTAB warning: take a look at the object file name that is warned on:

• if it is .../go.o, it is the object of the Go code. This is already fixed in recent versions of Go 1.20.x, 1.21.x, and 1.22.x, and should not appear with an up-to-date version of Go.
• if it is .../00NNNN.o with some 6-digit number, it is a C object.
  • if you pass -race and you're using Go 1.22.x, it is probably the race detector's C code, as mentioned above cmd/link: issues with Apple's new linker in Xcode 15 beta #61229 (comment)
  • otherwise it is from user C code

The warning on the race detector C object is not really a bug in Go. Maybe we can fix in the upstream TSAN (also see #61229 (comment)). To work around, you could try

• passing -ldflags=-extldflags=-Wl,-ld_classic to force the old Apple linker
• passing -ldflags=-linkmode=internal to force using Go linker to do the whole linking (although it is not guaranteed to be able to handle all C objects)
• also, an IDE should NOT treat warning as error by default.

Thanks.

[h31]

Hello @nebiros,

I reproduced your problem with the linker warning highlighted as an error in GoLand. The reason why the warning is displayed in red is that the linker message is being written to stderr. This is a highlighting problem and it shouldn't affect the build process in any way.
I've created an issue about this in our bug tracker: GO-16873.

[tkgalk]

The reason why the warning is displayed in red is that the linker message is being written to stderr.

That would explain why GoLand marked those tests as failed. I don't think this is a nice behaviour, warnings shouldn't be posted to stderr, it's reasonable for other applications to assume anything on stderr is an error.

[h31]

That would explain why GoLand marked those tests as failed.

I think this is a different problem, GoLand shouldn't mark tests as failed due to extra output. Can you please describe how to reproduce this issue? Here or in the comments to GO-16873.

[nebiros]

Hello @nebiros,

I reproduced your problem with the linker warning highlighted as an error in GoLand. The reason why the warning is displayed in red is that the linker message is being written to stderr. This is a highlighting problem and it shouldn't affect the build process in any way. I've created an issue about this in our bug tracker: GO-16873.

@h31 I know it doesn't affect the build process, but I wasn't talking about that, I was talking about testing and running unit tests:#61229 (comment), those warnings mark tests as failed, so it breaks all tests