change musl linker to /lib/ld-musl-x86_64.so.1 or support -dynamic-linker option

[vishvananda]

I've been attempting to build libraries for the x86_64-unknown-linux-musl target. I've managed to get static linking working. Dynamic linking is very close. Consider the following steps.

1. build libseccomp with musl:

    git clone https://github.com/seccomp/libseccomp
    cd libseccomp
    ./autogen.sh
    CPPFLAGS="-I/usr/include/x86_64-linux-musl" ./configure --enable-static
    make

2. build a project that includes the libseccomp-sys crate and include the new seccomp:

    RUSTFLAGS="-L native=/path/to/libseccomp/src/.libs -l static=seccomp"
    cargo build --verbose --target  x86_64-unknown-linux-musl

3. attempt to build with a dynamic library (ignore for the moment that this pulls in libc, still working on that issue):

    RUSTFLAGS="-L native=/path/to/libseccomp/src/.libs"
    cargo build --verbose --target  x86_64-unknown-linux-musl
    # using something from libseccomp segfaults because it tries to use the default loader
    LD_LIBRARY_PATH=/path/to/libseccomp/src/.libs ldd railcar
        linux-vdso.so.1 =>  (0x00007ffd5d5fe000)
        libseccomp.so.0 => /path/to/libseccomp/src/.libs/libseccomp.so.0 (0x00007f9b4e9ee000)
        /lib64/ld-linux-x86-64.so.2 (0x0000558a2ea4f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9b4e618000)

4. replace the dynamic linker with the musl one:

    patchelf --set-interpreter /lib/ld-musl-x86_64.so.1 $BINARY
    patchelf --remove-needed ld-linux-x86-64.so.2 $BINARY
    LD_LIBRARY_PATH=/path/to/libseccomp/src/.libs ./$BINARY
    # runs without error
    LD_LIBRARY_PATH=/path/to/libseccomp/src/.libs ldd railcar
        linux-vdso.so.1 =>  (0x00007fffea727000)
        libseccomp.so.0 => /path/to/libseccomp/src/.libs/libseccomp.so.0 (0x00007f267f87d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f267f4a7000)
        /lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x0000555f3380c000)

5. try to build binary with the right interpreter

    RUSTFLAGS="-L native=/path/to/libseccomp/src/.libs -C link-arg=-dynamic-linker=/lib/ld-musl-x86_64.so.1"
    cargo build --verbose --target  x86_64-unknown-linux-musl
    patchelf --print-interpreter $BINARY
        /lib64/ld-linux-x86-64.so.2

Any ideas why the -dynamic-linker arg is getting ignored when building my binary?

[vishvananda]

Ok I've managed to build libseccomp.so without bringing in libc:

CC=musl-gcc CPPFLAGS="-idirafter/usr/include/x86_64-linux-gnu -idirafter/usr/include" ./configure --enable-static

This .so is not readable by ldd but i can see what it wants using the interpreter directly

/lib/ld-musl-x86_64.so.1 --list src/.libs/libseccomp.so
    /lib/ld-musl-x86_64.so.1 (0x5606ecaf3000)
    libc.so => /lib/ld-musl-x86_64.so.1 (0x5606ecaf3000)

After building the binary with the new version I cannot run it, nor ldd it

sudo LD_LIBRARY_PATH=/home/vishvananda/libseccomp/src/.libs ./$BINARY
./$BINARY: error while loading shared libraries: /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header

but I can run it with the musl ld:

LD_LIBRARY_PATH=/path/to/libseccomp/src/.libs /lib/ld-musl-x86_64.so.1 ./$BINARY
# success!

and it runs just fine if i force the new interpreter:

patchelf --set-interpreter /lib/ld-musl-x86_64.so.1 $BINARY
LD_LIBRARY_PATH=/path/to/libseccomp/src/.libs ./$BINARY
# success!

Essentially I have reached the same conclusion, which is If I want to dynamically link to musl libraries, I have to be able to change the dynamic-interpreter somehow. I think this comes down to one of the following:

1) supporting the -dynamic-linker LDFLAG
2) defaulting the dynamic linker to the musl-specific ld for musl targetted builds

[vishvananda]

A bit more progress: I've found the invocation to actually pass link-arg to the linker:

RUSTFLAGS="-L native=/path/to/libseccomp/src/.libs -C link-arg=-dynamic-linker -C link-arg=/lib/ld-musl-x86_64.so.1"

Unfortunately, while this is adding a dependency on /lib/ld-musl-x86_64.so.1, it isn't properly setting it for the interpreter. I'm not sure what is missing to make it actually set the interpreter properly.

[vishvananda]

Looks like the people building on alpine ran into a similar problem here: #31322

That said, I don't see anything there that would allow me to change the interpreter that rustc is sticking into my elf header for the binaries it produces for the musl target.

[alexcrichton]

Thanks for the report! Some things I might recommend:

• Using RUSTFLAGS to link static libraries seems highly susceptible to breakage. I'd highly recommend using a build script instead which should be able to hook everything up correctly.
• Have you tried using the feature intended for customizing CRT linkage? You should be able to do so for musl with RUSTFLAGS=-Ctarget-feature=-crt-static.
• I'd make sure musl-gcc is being used but it looks like you're doing so already.
• I'd make sure to compile all native code with musl-gcc as well.

[vishvananda]

Thanks for taking a look at this.

1. yes a build script seems like a better choice, I was just using RUSTFLAGS for testing. Can I set the proper options up in the build script for my binary, or do I need to add a build script to the libseccomp-sys crate to pull the static link information from features or environment variables?
2. Adding -C target-feature=-crt-static -Z unstable-options without dynamic-linker fails to find libc
3. Adding crt-static with dynamic-linker builds properly, but the file cannot be launched by either ld-linux or ld-musl (ld-musl prints: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by ./$BINARY))
4. Is there a way besides specifying the target to make sure the rust build happens with musl-gcc?

I think the proper solution here is to get rust to set the .interp header properly when writing out the ELF file. I have not yet figured out what component is responsible for generating the elf, but I suspect that if -dynamic-linker is specified then it should be set in the header like ld does. The fact dynamic-linker causes a dependency to be added but not included in the elf header makes me think that rust is doing some post-processing before writing out the ELF and not setting .interp. Any tips for where I should look for that?

[alexcrichton]

Typically when you compile for musl but the resulting binary wants ld-linux-* it means that something went wrong along the way (e.g. cross compilers weren't set up or something like that). I don't think that -dynamic-linker is the best solution here (as it's likely just a symptom, not the real problem)

Do you have some logs I could perhaps take a look at and help with?

[codyps]

x86_64-unknown-linux-musl defaults to using cc as it's linker. Try adding -C linker=musl-gcc to RUSTFLAGS.

[vishvananda]

@jmesmon that is exactly what I just discovered. I had assumed it was defaulting to musl-gcc. It woks fine without passing in dynamic-library with -C linker=musl-gcc.

I guess the default wasn't changed becasue for static linking cc works just fine. It should probably be documented somewhere that dynamic linking requires changing the linker.

Thanks for the help @alexcrichton and @jmesmon

Feel free to close this if there are no further comments.

[alexcrichton]

Ok, glad it worked out!

[iav]

What if cross-compilation to another platform? On x86_64-unknown-linux-gnu for armv7-unknown-linux-musleabihf? What replacement for RUSTFLAGS="-L native=/path/to/libseccomp/src/.libs -C link-arg=-dynamic-linker -C link-arg=/lib/ld-musl-x86_64.so.1" should be used?

[Arcttiicc]

So i am making a own server with some other people and when we finally were done this popped up. dont know what it is, dont know what it does, just need some help on how to fix it.

[Pterodactyl Daemon] Server marked as STARTING
[Pterodactyl Daemon] Checking size of server data directory...
[Pterodactyl Daemon] Disk Usage: 10037M / 51200M
[Pterodactyl Daemon] Ensuring correct ownership of files.
[Pterodactyl Daemon] Running server preflight.
[Pterodactyl Daemon] Starting server container.
:/home/container$ /home/container/alpine/opt/cfx-server/ld-musl-x86_64.so.1 --library-path /home/container/alpine/usr/lib/v8/:/home/container/alpine/lib/:/home/container/alpine/usr/lib/ -- /home/container/alpine/opt/cfx-server/FXServer +set citizen_dir /home/container/alpine/opt/cfx-server/citizen/ +set sv_licenseKey i4ywcdoe6k8s3q5np9a9jgapat8v0ex8 +set sv_maxplayers 32 +exec server.cfg
/entrypoint.sh: eval: line 1: /home/container/alpine/opt/cfx-server/ld-musl-x86_64.so.1: Permission denied
[Pterodactyl Daemon] Server marked as OFF
[Pterodactyl Daemon] ---------- Detected server process in a crashed state! ----------
[Pterodactyl Daemon] Exit Code: 126
[Pterodactyl Daemon] Out of Memory: false
[Pterodactyl Daemon] Error Response:
[Pterodactyl Daemon] Aborting automatic reboot due to crash within the last 60 seconds.