musl/amd64: relocation R_X86_64_32 against hidden symbol can not be used when making a shared object

[lucab]

Trying to build a project which depends on ring (current release, 0.9.7) for musl-amd64 target fails at linktime with a relocation error.

The environment is an up-to-date debian stretch (next stable), with musl 1.1.16, ld.bfd 2.28 and rustc 1.17.0.

This can be reproduced by building any of the examples from this wip branch:

$ cargo build --target x86_64-unknown-linux-musl --release --example trace
[...]
error: linking with `cc` failed: exit code: 1
  |
[...]
  = note: /usr/bin/ld: dkregistry-rs/target/x86_64-unknown-linux-musl/release/deps/libring-4ab2a2cec4828006.rlib(e_aes.o): relocation R_X86_64_32 against hidden symbol `GFp_AES_encrypt' can not be used when making a shared object
          /usr/bin/ld: dkregistry-rs/target/x86_64-unknown-linux-musl/release/deps/libring-4ab2a2cec4828006.rlib(curve25519.o): relocation R_X86_64_32 against `.rodata.kMongomeryBasePoint.3027' can not be used when making a shared object; recompile with -fPIC
          /usr/bin/ld: dkregistry-rs/target/x86_64-unknown-linux-musl/release/deps/libring-4ab2a2cec4828006.rlib(ecp_nistz256.o): relocation R_X86_64_32 against `.rodata.GFp_nistz256_precomputed' can not be used when making a shared object; recompile with -fPIC
          /usr/bin/ld: dkregistry-rs/target/x86_64-unknown-linux-musl/release/deps/libring-4ab2a2cec4828006.rlib(gfp_p384.o): relocation R_X86_64_32 against `.rodata.Q' can not be used when making a shared object; recompile with -fPIC
          /usr/bin/ld: dkregistry-rs/target/x86_64-unknown-linux-musl/release/deps/libring-4ab2a2cec4828006.rlib(gcm.o): relocation R_X86_64_32 against `.rodata.ZEROS.2281' can not be used when making a shared object; recompile with -fPIC
          /usr/bin/ld: final link failed: Nonrepresentable section on output
          collect2: error: ld returned 1 exit status

[lucab]

I've spent a bit more time looking into this, and after several wrong leads I now think that this due to rustc static-linking assumption about the musl target, resulting in an hardcoded -static when shelling out to musl-gcc. Thus, objects are currently compiled as:

running "musl-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-m64" "-static" "-I" "include" "-std=c1x" "-Wbad-function-cast" "-Wmissing-prototypes" "-Wnested-externs" "-Wstrict-prototypes" "-fdata-sections" "-ffunction-sections" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-declarations" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-Wno-cast-align" "-fstack-protector" "-g3" "-Werror" "-U_FORTIFY_SOURCE" "-D_XOPEN_SOURCE=700" "-c" "-otarget/x86_64-unknown-linux-musl/debug/build/ring-87a98ec525fe48e6/out/gcm.o" "crypto/modes/gcm.c"

I'm currently playing with just appending a -shared flag, which may be enough to override the earlier static flag and result in proper linkage.

[briansmith]

@lucab I tried building your project's ups/hyper-rustls branch like this:

 cargo build --target x86_64-unknown-linux-musl --release --example trace

It worked fine with no changes to ring or gcc-rs. Your wip branch has been deleted so I'm not sure if it is different. LMK how I can reproduce the problem.

[briansmith]

I am still skeptical of the approach of using -shared. The error message is “relocation R_X86_64_32 against hidden symbol `xxx' can not be used when making a shared object.” But, why is the linker trying to make a shared object in this configuration? Aren't you trying to statically link everything?

[lucab]

Meh, a typo slipped in the branch name (s/wip/ups/) I updated the initial comment. master is using a forked ring repo with the fix in #549, so that it doesn't fail.

I understand you are still skeptical and I'm sorry I don't have the full knowledge to answer you. My current understanding is that statically linking the final binary results in combining together objects from multiple rlibs and my own binary object, but some combinations are not valid. But your questions are on spot, and the final binary is indeed weird:

$ ldd target/x86_64-unknown-linux-musl/release/examples/trace
 statically linked

$ file target/x86_64-unknown-linux-musl/release/examples/trace
 ELF 64-bit LSB shared object,
 x86-64,
 version 1 (GNU/Linux),
 dynamically linked,
 interpreter /lib64/ld-linux-x86-64.so.2,
 BuildID[sha1]=1a0fc33ab4bbd055bdd15438ae680e6f13feaa7b,
 not stripped

I'm now waiting for the next release of gcc-rs with rust-lang/cc-rs#167 to make some more experiments.

[lucab]

Indeed, it looks like something fishy was going on with this cargo cross-building. In particular, it looks like even with --target x86_64-unknown-linux-musl cargo/rustc are going through my host cc instead of musl-gcc. This resulted in the host ld adding -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie when linking and turning the final executable into a dynamic artifact with no external linked-libs, but with a loader set. I didn't realize this, as lld was reporting static linking.

I tried reverting to vanilla ring and just tweaking my cross-linker options locally in .cargo/config, and the following snippet fixed it (without further changes in ring or gcc-rs):

[target.x86_64-unknown-linux-musl]
linker = "musl-gcc"

I think this is the proper fix instead of #549, so I'm going to close that PR and just stick to a local config override. This bug can be closed as well, or it can be addressed by a small doc section like "In case of cross-compilation issue, make sure that the proper cross-toolchain binaries are configured and used", somewhere.

[lucab]

It looks like I'm not the first one stepping on this: rust-lang/rust#40049 (comment), rust-lang/cargo#4133.

[briansmith]

Thanks lucab. I agree that the errors can be very confusing if you don't override the linker. Thanks for filling in the missing details!