ld.lld: error: relocation R_386_PC32 cannot be used against symbol __zig_probe_stack

[leonidborisenko]

Environment

• Linux with x86_64 kernel and i386 glibc userspace
• Zig snapshot from zig-linux-x86_64-0.8.0-dev.1104+5e81b048a.tar.xz tarball

$ zig version
0.8.0-dev.1104+5e81b048a

Minimal reproducible example

// dummy.zig
export fn dummy() c_int {
    return 0;
}

$ zig build-lib dummy.zig -dynamic -target i386-native

zig build-lib output (with `ld.lld` error)

ld.lld: error: relocation R_386_PC32 cannot be used against symbol __zig_probe_stack; recompile with -fPIC
>>> defined in /tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/libcompiler_rt.a(/tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/compiler_rt.o)
>>> referenced by fs.zig:684 (/tmp/zig/zig-linux-x86_64-0.8.0-dev.1104+5e81b048a/lib/std/fs.zig:684)
>>>               zig-cache/o/dbc73e4dde353bae2e8456cf743e6f42/dummy.o:(std.fs.Dir.openFile)

ld.lld: error: relocation R_386_PC32 cannot be used against symbol __zig_probe_stack; recompile with -fPIC
>>> defined in /tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/libcompiler_rt.a(/tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/compiler_rt.o)
>>> referenced by os.zig:4504 (/tmp/zig/zig-linux-x86_64-0.8.0-dev.1104+5e81b048a/lib/std/os.zig:4504)
>>>               zig-cache/o/dbc73e4dde353bae2e8456cf743e6f42/dummy.o:(std.os.toPosixPath)

ld.lld: error: relocation R_386_PC32 cannot be used against symbol __zig_probe_stack; recompile with -fPIC
>>> defined in /tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/libcompiler_rt.a(/tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/compiler_rt.o)
>>> referenced by debug.zig:1040 (/tmp/zig/zig-linux-x86_64-0.8.0-dev.1104+5e81b048a/lib/std/debug.zig:1040)
>>>               zig-cache/o/dbc73e4dde353bae2e8456cf743e6f42/dummy.o:(std.debug.printLineFromFileAnyOs)
error: LLDReportedFailure

Note

It happens only in -O Debug. It doesn't happen in all other -O modes.

Workaround

• Just add -z notext to build-lib command line arguments, like:

zig build-lib dummy.zig -dynamic -target i386-native -z notext

• This option could be set in build.zig. As suggested in ld.lld: error: relocation R_386_PC32 cannot be used against symbol __zig_probe_stack #7935 (comment):

  you can set lib.link_z_notext = true; in your build.zig

See pull request #10056 (and merged commit 2cdffc9).

Zig 0.8.1 didn't include this commit, following downloadable releases should allow this workaround. I've tested it with 0.9.0-dev.1561+5ebdc8c46 (released on 2021-11-03).

Workaround (for Zig versions released before 29-Oct-2021)

Execute `zig ld.lld` manually with added `-z notext`

1. Find ld.lld invocation arguments.

$ zig build-lib dummy.zig -dynamic -target i386-native --verbose-link
ld.lld -error-limit=0 --gc-sections -m elf_i386 -shared -o libdummy.so -soname libdummy.so zig-cache/o/dbc73e4dde353bae2e8456cf743e6f42/dummy.o /tmp/zig/cache/zig/o/44e6d86d0e734b882fd45d9658d515d8/libc.a /tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/libcompiler_rt.a
ld.lld: error: relocation R_386_PC32 cannot be used against symbol __zig_probe_stack; recompile with -fPIC
>>> defined in /tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/libcompiler_rt.a(/tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/compiler_rt.o)
>>> referenced by fs.zig:684 (/tmp/zig/zig-linux-x86_64-0.8.0-dev.1104+5e81b048a/lib/std/fs.zig:684)
>>>               zig-cache/o/dbc73e4dde353bae2e8456cf743e6f42/dummy.o:(std.fs.Dir.openFile)

ld.lld: error: relocation R_386_PC32 cannot be used against symbol __zig_probe_stack; recompile with -fPIC
>>> defined in /tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/libcompiler_rt.a(/tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/compiler_rt.o)
>>> referenced by os.zig:4504 (/tmp/zig/zig-linux-x86_64-0.8.0-dev.1104+5e81b048a/lib/std/os.zig:4504)
>>>               zig-cache/o/dbc73e4dde353bae2e8456cf743e6f42/dummy.o:(std.os.toPosixPath)

ld.lld: error: relocation R_386_PC32 cannot be used against symbol __zig_probe_stack; recompile with -fPIC
>>> defined in /tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/libcompiler_rt.a(/tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/compiler_rt.o)
>>> referenced by debug.zig:1040 (/tmp/zig/zig-linux-x86_64-0.8.0-dev.1104+5e81b048a/lib/std/debug.zig:1040)
>>>               zig-cache/o/dbc73e4dde353bae2e8456cf743e6f42/dummy.o:(std.debug.printLineFromFileAnyOs)
error: LLDReportedFailure

2. Execute zig ld.lld manually with added -z notext.

$ zig ld.lld -error-limit=0 --gc-sections -m elf_i386 -shared -o libdummy.so -soname libdummy.so zig-cache/o/dbc73e4dde353bae2e8456cf743e6f42/dummy.o /tmp/zig/cache/zig/o/44e6d86d0e734b882fd45d9658d515d8/libc.a /tmp/zig/cache/zig/o/24934007c007b2d5b7135d2223ee0adf/libcompiler_rt.a -z notext

3. libdummy.so will be produced.

Conclusions

That begs a question: is it or is it not possible to pass freeform extra arguments to internal zig ld.lld invocation simply by using additional zig build-lib options and/or by using appropriate function in build.zig? (I didn't find a way to do it.)

Also I don't know and can't argue whether passing -z notext is a safe or right solution. Just reporting that it works.

Relevant links

-z notext

• reviews.llvm.org / [ELF] - Implemented -znotext
• can't create dynamic relocation R_386_32 with LLD ClangBuiltLinux/linux#579
• Unable to install openssl on FreeBSD 12.1 32 bit openssl/openssl#11305 (comment) and FreeBSD Bugzilla – Attachment #197583: OpenSSL 1.1 LLD fix for bug #231459

TEXTREL

• Android Changes / Text Relocations (Enforced for API level >= 23)
• Gentoo Hardened/Textrels Guide

[LemonBoy]

The R_386_PC32 relocation smells fishy, R_386_PLT32 should be used instead as this is a position-independent code and that's the only external symbol that's not going trough the PLT.

LLVM's code emitting the stack-probe call defaults to CALLpcrel32 (and CALL64pcrel32 for x86-64).

cc @MaskRay

[MaskRay]

I suspect zig sets dso_local on function declarations for -fpic mode.

[LemonBoy]

AFAICS not really, at least not explicitly.

[LemonBoy]

This problem can be reproduced with this small snippet of LLVM IR:

target triple = "i386-unknown-linux-none"

declare void @ext()

define void @dummy() #0 {
Entry:
  %trigger = alloca [4097 x i8], align 1
  call void @ext()
  ret void
}

attributes #0 = { "probe-stack"="probe" }

When compiled with --relocation-model=pic the call to ext is routed trough the PLT (R_386_PLT32) as expected while the stack probe remains PC-relative. This makes me think the stack probe symbol is not a common external symbol and must not be routed trough the PLT (That's reasonable as this this is a stack probe after all).

[ikskuh]

I have encountered this bug today when working on android-x86

[silversquirl]

This seems pretty critical. Pretty much disallows using Zig for building shared libraries.

The suggested workaround isn't practical to use since there's no way to pass lld flags from build.zig, and even compiling in release doesn't always work.

Here's a simple example that triggers this bug in ReleaseSafe mode:

const std = @import("std");
export fn hello() void {
    std.debug.print("Hello, world!\n", .{});
}

Compile with zig build-lib -dynamic -target i386-native -lc -OReleaseSafe hello.zig (note: linking libc is the thing that makes this example break in ReleaseSafe - if you don't link libc it only breaks in Debug)

[leonidborisenko]

The suggested workaround isn't practical to use since there's no way to pass lld flags from build.zig, and even compiling in release doesn't always work.

Here's a simple example that triggers this bug in ReleaseSafe mode:

Current master allows adding -z notext to build-lib command line arguments, like:

zig build-lib dummy.zig -dynamic -target i386-native -z notext

See pull request #10056 (and merged commit 2cdffc9). I've tested it with 0.9.0-dev.1561+5ebdc8c46 (released on 2021-11-03).

Both my example (from issue report) and yours (from quoted comment, with exactly the same command line arguments as you provided, except added -z notext) could be built just fine and afterwards produced shared libraries work (for me) as expected.

I've updated my original report with this workaround.

Still, I want to stress, that:

Also I don't know and can't argue whether passing -z notext is a safe or right solution. Just reporting that it works.

[bayo-code]

What is the status of this? Building shared libraries for android doesn't seem to work

[mlugg]

@Hardebayho, in case you never found out, you can set lib.link_z_notext = true; in your build.zig as a workaround.

[bayo-code]

@mlugg Yeah, I did find the flag. Thanks

[ikskuh]

@Hardebayho, in case you never found out, you can set lib.link_z_notext = true; in your build.zig as a workaround.

Thanks for that, let's try integrating it into the Android SDK!

[silbinarywolf]

Just an FYI for anyone building x86 code for Android and using the workaround mentioned above:
lib.link_z_notext = true;

Then you'll likely hit the following issue when loading your code/APK at runtime:

Text Relocations Enforced for API Level 23
See: https://android.googlesource.com/platform/bionic/+/refs/tags/ndk-r14/android-changes-for-ndk-developers.md

For context, Android 6.0 is at API Level 23.