Cannot cross-compile aarch64-musl on x86_64 linux

[asonix]

Cross compiling fails with the linker being unable to find __addtf3, __multf3 and __subtf3

code:

asonix@asonix-boxes-16-04:~/hello-world-rust$ cat src/main.rs 
fn main() {
    println!("Hello, world!");
}

I'm compiling the default hello-world application, and it works fine for aarch64-unknown-linux-gnu in addition to working fine for arm-unknown-linux-musleabihf. I installed musl the same way for both aarch64 and for armhf, so I'm not sure what the issue would be.

Furthermore, I can compile a simple hello-world application written in C with the musl standard library just fine.

Note: I have encountered this issue both in Arch Linux with the 4.14.4 kernel and Ubuntu 16.04 with the 4.4.0 kernel.

version info for the output below, although this happens with nightly and stable:

asonix@asonix-boxes-16-04:~/hello-world-rust$ rustc --version --verbose
rustc 1.22.1 (05e2e1c41 2017-11-22)
binary: rustc
commit-hash: 05e2e1c41414e8fc73d0f267ea8dab1a3eeeaa99
commit-date: 2017-11-22
host: x86_64-unknown-linux-gnu
release: 1.22.1
LLVM version: 4.0

output:

asonix@asonix-boxes-16-04:~/hello-world-rust$ cargo build --target=aarch64-unknown-linux-musl --verbose
   Compiling hello-world v0.1.0 (file:///home/asonix/hello-world-rust)
     Running `rustc --crate-name hello_world src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=df195ab9189e537e -C extra-filename=-df195ab9189e537e --out-dir /home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps --target aarch64-unknown-linux-musl -C ar=aarch64-linux-gnu-ar -C linker=aarch64-linux-gnu-musl-gcc -L dependency=/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps -L dependency=/home/asonix/hello-world-rust/target/debug/deps`
error: linking with `aarch64-linux-gnu-musl-gcc` failed: exit code: 1
  |
  = note: "aarch64-linux-gnu-musl-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-nostdlib" "-Wl,--eh-frame-hdr" "-Wl,-(" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/crt1.o" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/crti.o" "-L" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps/hello_world-df195ab9189e537e.hello_world0.rust-cgu.o" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps/hello_world-df195ab9189e537e.hello_world1.rust-cgu.o" "-o" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps/hello_world-df195ab9189e537e" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps/hello_world-df195ab9189e537e.crate.allocator.rust-cgu.o" "-Wl,--gc-sections" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "-L" "/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps" "-L" "/home/asonix/hello-world-rust/target/debug/deps" "-L" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liballoc_system-751808ba756769d5.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/librand-3d7b10e850a67e89.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libpanic_unwind-8cb97051d8238386.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liballoc-78c21267a2dc15c1.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd_unicode-0e1b544c94586415.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libunwind-25cc9b024a02d330.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-d42e80cee81b06ce.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libcore-0c5e3d6c117f8c44.rlib" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-bd7cc5ada1e908e0.rlib" "-static" "-Wl,-Bdynamic" "/home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/crtn.o" "-Wl,-)"
  = note: /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib(std-827ddea93328ae2d.std0.rust-cgu.o)(.debug_info+0x71d3): R_AARCH64_ABS64 used with TLS symbol _ZN3std10sys_common11thread_info11THREAD_INFO7__getit5__KEY17hffb73c8d664d75eeE
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib(std-827ddea93328ae2d.std0.rust-cgu.o)(.debug_info+0x185a7): R_AARCH64_ABS64 used with TLS symbol _ZN3std2io5stdio12LOCAL_STDOUT7__getit5__KEY17hc4c662569625ecfaE
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib(std-827ddea93328ae2d.std0.rust-cgu.o)(.debug_info+0x24159): R_AARCH64_ABS64 used with TLS symbol _ZN3std9panicking12LOCAL_STDERR7__getit5__KEY17h933b3f2f97fe548cE
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-827ddea93328ae2d.rlib(std-827ddea93328ae2d.std0.rust-cgu.o)(.debug_info+0x24191): R_AARCH64_ABS64 used with TLS symbol _ZN3std9panicking18update_panic_count11PANIC_COUNT7__getit5__KEY17h5de4a458c39aaf87E
          /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-d42e80cee81b06ce.rlib(vfprintf.o): In function `fmt_fp':
          vfprintf.c:(.text.fmt_fp+0x178): undefined reference to `__addtf3'
          vfprintf.c:(.text.fmt_fp+0x20c): undefined reference to `__multf3'
          vfprintf.c:(.text.fmt_fp+0x240): undefined reference to `__subtf3'
          vfprintf.c:(.text.fmt_fp+0x250): undefined reference to `__addtf3'
          vfprintf.c:(.text.fmt_fp+0x278): undefined reference to `__addtf3'
          vfprintf.c:(.text.fmt_fp+0x284): undefined reference to `__subtf3'
          vfprintf.c:(.text.fmt_fp+0x33c): undefined reference to `__subtf3'
          vfprintf.c:(.text.fmt_fp+0x344): undefined reference to `__multf3'
          vfprintf.c:(.text.fmt_fp+0x4e8): undefined reference to `__multf3'
          vfprintf.c:(.text.fmt_fp+0x548): undefined reference to `__subtf3'
          vfprintf.c:(.text.fmt_fp+0x550): undefined reference to `__multf3'
          vfprintf.c:(.text.fmt_fp+0x828): undefined reference to `__addtf3'
          /home/asonix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-d42e80cee81b06ce.rlib(frexpl.o): In function `frexpl':
          frexpl.c:(.text.frexpl+0x38): undefined reference to `__multf3'
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

error: Could not compile `hello-world`.

Caused by:
  process didn't exit successfully: `rustc --crate-name hello_world src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=df195ab9189e537e -C extra-filename=-df195ab9189e537e --out-dir /home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps --target aarch64-unknown-linux-musl -C ar=aarch64-linux-gnu-ar -C linker=aarch64-linux-gnu-musl-gcc -L dependency=/home/asonix/hello-world-rust/target/aarch64-unknown-linux-musl/debug/deps -L dependency=/home/asonix/hello-world-rust/target/debug/deps` (exit code: 101)

[parched]

Yeah this is a known issue which needs to be fixed in the compiler_builtins crate. See rust-lang/compiler-builtins#201

[asonix]

I guess my followup is why is aarch64-unknown-linux-musl an allowed target in Rust if f128 doesn't exist in the language and linking anything doesn't work.

[parched]

Good question, it's not entirely clear to me if it's actually compiler_builtins responsibility to provide these. For now you should be fine if you also link libgcc.

[ciofeca]

Is there any workaround?

Say, some combination of #[cfg(link...)] and extern fn __addtf3(_a: i128, _b: i128) -> i128 { 1 }

[ghost]

It would be greatly appreciated if anyone could provide any ideas for any kind of temporary workaround for this until it is properly fixed. I don't care if certain functionality (such as 128-bit integers or whatever) doesn't work. Being able to build static musl executables for aarch64 would be very useful to me.

[parched]

Does linking libgcc not work?

[ciofeca]

There are some use-cases for 64-bit statically linking with musl: for example, SailfishX sports a 64-bit Linux kernel and 32-bit only apps/libs (no 64-bit ld-linux*.so installed). A 64-bit statically linked executable would work out of the box.

[Amanieu]

In case anyone is blocked on this, here's a workaround:

cargo rustc --target aarch64-unknown-linux-musl -- -C link-arg=-lgcc

[Amanieu]

This should be reopened due to rust-lang/libc#1045

[killercup]

@bbqsrc just mentioned this as only-working-with-workaround (cargo rustc --target aarch64-unknown-linux-musl --bin $cratename -- -C link-arg=-lgcc).

Reopening as requested by @Amanieu because of reverted fix PR

[ciofeca]

I was anyway able to cross-compile to aarch64 musl target using this .cargo/config (thanks for the workaround!):

[target.aarch64-unknown-linux-musl]
linker = "aarch64-linux-gnu-gcc"
rustflags = [ "-C", "target-feature=+crt-static", "-C", "link-arg=-lgcc" ]

[qm3ster]

@ciofeca's solution worked for me on Windows
I still had to mklink ar.exe aarch64-linux-musl-ar.exe for the moment, not sure how to get gcc to ask for the right one.

[clarfonthey]

So, what exactly is needed to fix this without the -lgcc?

[12101111]

So, what exactly is needed to fix this without the -lgcc?

__addtf3 is quadruple precision add, __multf3 is quadruple precision multiplication and __subtf3is quadruple precision substruction.

the symbol __addtf3 is equal to this C function:

long double __addtf3 (long double a, long double b){
  return a+b;
}

libgcc.a from gcc and libclang_rt.builtins-aarch64.a from llvm compiler-rt provide
the implement of __addtf3, __multf3 and __subtf3.

In rust we use compiler-builtins (writing in rust) to replace llvm compiler-rt (writing in c and llvm asm)

quadruple precision, and long double in C, means 128bit float number, but
rust don't have f128 type, so compiler-builtins don't support those *tf3

A temporary fix is linking to libgcc.a or libclang_rt.builtins-aarch64.a, but permanent fix won't exist until rust have f128

[AlexTMjugador]

The compiler-builtins crate now has an implementation for AArch64 long double intrinsics, so this issue is solved in most cases.

However, because these intrinsics are implemented using C functions coming from LLVM's compiler-rt, using -Z build-std in combination with -Z build-std-features may not enable the compiler-builtins-c std feature, causing this issue to users of -Z build-std and -Z build-std-features in the latest Rust builds.

Therefore, users of -Z build-std -Z build-std-features may still need to work around this issue by either:

• Linking with the GCC intrinsics implementation (-C link-arg=-lgcc), as mentioned above.
• Explicitly build std with the compiler-builtins-c feature, by setting -Z build-std-features=<other-features>,compiler-builtins-c and setting up the RUST_COMPILER_RT_ROOT environment variable to the path to the compiler-rt source, similarly to how the compiler-builtins CI workflow does it:

curl -L -o code.tar.gz https://github.com/rust-lang/llvm-project/archive/rustc/17.0-2023-09-19.tar.gz
tar xzf code.tar.gz --strip-components 1 llvm-project-rustc-17.0-2023-09-19/compiler-rt
RUST_COMPILER_RT_ROOT="$PWD/compiler-rt" cargo build --release <...> -Z build-std -Z build-std-features=<features>,compiler-builtins-c

The second option is more complex to set up, but it should produce outputs closer to the usual pre-built Rust std binaries.