Tracking issue for musl host toolchain

[mati865]

TODO:

• Installable host toolchain (Musl host toolchain #58575)
  • Rustup support (Add x86_64-unknown-linux-musl build rustup#1882)
• Tests passing as the host ( Make tests compatible with musl host  #60474)
• Transition musl targets to dynamically link libc by default
  • MCP approving this: Update the existing musl targets to be dynamically linked. compiler-team#422
• static-pie support static-pie for musl target (remove crt1.o) #53968 (done in Enabling static-pie for musl #70740)
• Update to musl 1.2 x86_64-unknown-linux-musl: backport musl bug fixes or use new 1.2 version? #91178

For people looking how to use as it dynamic target:

RUSTFLAGS="-C target-feature=-crt-static" cargo build

[wangbj]

It seems x86_64-unknown-linux-musl target doesn't honor --enable-default-pie, a different behavior compare to x86_64-unknown-linux-gnu. Is this a known issue?

[mati865]

@wangbj I don't think Rust supports --enable-default-pie for any target. Could you provide more details?

[wangbj]

hmm, the problem seems caused by musl-gcc wrapper instead of rustc:

details: https://gist.github.com/wangbj/ead5fa8c96418de7c9050bc354fbf353

--enable-default-pie is a gcc configure flag. sorry for the noise. (shortened the text to a gist).

[Cogitri]

FWIW, Void has been hacking around this for some time: https://github.com/void-linux/void-packages/blob/742a7d53eb0b373bf8dad3b51db3380c5cdf8dc1/srcpkgs/rust/patches/musl-dont-use-crt-static.patch and https://github.com/void-linux/void-packages/blob/742a7d53eb0b373bf8dad3b51db3380c5cdf8dc1/srcpkgs/rust/patches/link-musl-dynamically.patch

I'm currently working on this for Alpine too (BTW, maybe I can ping you on the PR for this @mati865 so you can take a look at it?).

[mati865]

I knew about Alpine and Void. IMO Alpine approach is better because <arch>-unknown-linux-musl remain static by default (it behaves like upstream) and new target <arch>-alpine-linux-musl links dynamically by default.

I had looked at that PR and libc changes will be incompatible with upstream rust behaviour. Official Rust packages when statically linking to musl create executables that all fully static (run basically everywhere), that patch will create static executables that depend on musl runtime (use have to install musl to be able to run them).

[Cogitri]

Yup, but at least on Void we've come to terms that the Rust in the repos is only really meant for packages in the repos, while users interested in rust development should use upstream Rust via rustip.

[smaeul]

I had looked at that PR and libc changes will be incompatible with upstream rust behaviour. Official Rust packages when statically linking to musl create executables that all fully static (run basically everywhere), that patch will create static executables that depend on musl runtime (use have to install musl to be able to run them).

That doesn't make sense. Static executables by definition don't depend on having musl installed. Are you referring to rust-lang/libc#1327? That change would require having a musl-targeting toolchain installed for rustc to link (static) musl-targeting binaries. But any static binaries compiled by rustc would still be totally independent and require no musl installation.

[mati865]

@smaeul I'm referring to https://github.com/alpinelinux/aports/blob/21c1983ce828ead1afc9e80aa77ede330c662182/community/rust/link-musl-dynamically.patch

[smaeul]

@mati865 Okay, so that patch is misnamed. It doesn't have anything to do with static versus dynamic linking. It makes the same change as rust-lang/libc#1327 -- using the C toolchain's copy of libc.a instead of bundling it inside liblibc.rlib. And my comment above applies to it as well.

Alpine also ships custom targets that use the C toolchain's crt*.o instead of rustc shipping a second copy. Since on Alpine targets rustc's copy of crt*.o are not used, this patch stops rustc from making the copy. (This change is unrelated to the liblibc change.)

[mati865]

Thanks for the explanation.

[thedrow]

#53968 also seems to be related but it's not included in the OP's checklist.
Am I correct?

[mati865]

Updated

[thedrow]

#61328 also seems important.

[mati865]

I don't see how #61328 relates to this, there is no issue when using target-feature=-crt-static.
It applies to musl toolchain in general but it has nothing to do with using musl toolchain natively.

[gnzlbg]

In rust-lang/libc#1327, a #[cfg(target_env = "dyn_musl")] or similar was proposed to support both statically and dynamically linked musl environments. I don't know if that would be a good way to make progress here though.

How do other run-times deal with this ? Is it possible to statically link glibc and others into Rust binaries ? Maybe we need a #[cfg(target_env_linkage = "static/dynamic/...")] system instead of duplicating target_envs per target.

[mati865]

AFAIK musl is the only target which you can link statically and dynamically at same time.

Duplicating target_env is easy but for me it sounds like a bad workaround because user would have to install 2 toolchains that have exactly the same libs and the only difference comes to target-feature=-crt-static.
With static nobundle it would be third 99% identical toolchain.

Ideally they should use single shared sysroot because all that is required can be handled at runtime.
I think Rust is not capable of it yet so this would have to go through RFC first.

[smaeul]

How do other run-times deal with this ? Is it possible to statically link glibc and others into Rust binaries ?

The two relevant target attributes are crt_static_respected and crt_static_allows_dylibs.

For glibc (and all other non-musl unix-like platforms), crt_static_respected is false, so the -C target-feature=+crt-static option is ignored by the backend and still dynamically links libc, and therefore dynamic native libraries can still be linked. So the only effect of the option is that it matches
#[cfg(target_feature = "crt-static")].

For MSVC, crt_static_respected and crt_static_allows_dylibs are both true, so -C target-feature=+crt-static only affects the C runtime, and other (rust or native) libraries can still be loaded as DLLs. This is because on Windows, there's not really such a thing as a "static binary".

musl is in the unique situation of crt_static_respected being true and crt_static_allows_dylibs being false. Since musl uses ELF (and this would apply to glibc too if it supported static linking), linking libc statically requires all other libraries to also be linked statically. For native libraries, this means #[link(kind = "static")] or #[link(kind = "static-nobundle)]".

rustc currently handles this limitation correctly for rust libraries and direct native dependencies, but not for native libraries that are transitive dependencies of a dependent crate.

I had a patch (#55566) that "fixed" this by changing the semantics of #[link] with no kind in the +crt-static and not crt_static_allows_dylibs case. But the default kind is documented to be dylib, so that change was unacceptable. Instead, we need a patch to produce an error, similar to the non-transitive case here.

Maybe we need a #[cfg(target_env_linkage = "static/dynamic/...")] system instead of duplicating target_envs per target.

We already have that: #[cfg(target_feature = "crt-static")].

---

The problem in rust-lang/libc#1327 is that we try to support cross-compilation to musl without having the proper sysroot installed. The whole musl_root system is a giant hack to support that, and it breaks in the native case, where we want to use the libc.a that is actually present on the system (and updated by the package manager, etc.). Distributions that ship a native musl rust have to add a patch similar to the one in that PR (specifically e.g. this one for Adélie and Gentoo).

[Cogitri]

I had a patch (#55566) that "fixed" this

Fwiw Alpine Linux ships a similar patch (although we don't use crt-static for pur distro packages). Good to know what's the status on that, thanks.

https://github.com/alpinelinux/aports/blob/master/community/rust/musl-fix-static-linking.patch

Distributions that ship a native musl rust have to add a patch similar to the one in that PR (specifically e.g. this one for Adélie and Gentoo).

Void Linux and Alpine Linux carry a similiar patch: https://github.com/alpinelinux/aports/blob/master/community/rust/link-musl-dynamically.patch

[gnzlbg]

The problem in rust-lang/libc#1327 is that we try to support cross-compilation to musl without having the proper sysroot installed.

Many production users rely on this feature, so we are kind of constrained on not breaking this. If the only way to achieve that is have a second set of targets, then that's the way it is.

[smaeul]

Many production users rely on this feature, so we are kind of constrained on not breaking this. If the only way to achieve that is have a second set of targets, then that's the way it is.

What about shipping libc.a alongside rust, like we already do for crt*.o? Then we could link it static-nobundle from liblibc.rlib instead of bundling it inside. And then if we only add a -L for that directory to the linker args when cross-compiling, it would properly use the system libc.a for the native case as well. (And distro packagers could simply rm rust's copy of libc.a instead of patching).

[gnzlbg]

I think I would prefer something that also works great for distro packages. While they could tune things when packaging, a distro that uses musl as its stdlib, will have users that might install rust via rustup, and they will want their "default" rust target to dynamically link the system musl, instead of doing something else, right?

[msrd0]

Another problem I noticed trying to compile/run the rust compiler on a musl host like Alpine Linux is that using the jemallocator for the compiler itself does not seem to work. Given that rust is known to run slower with musl's default allocator than with jemalloc, I tried enabling jemalloc = true in rustc's config file (rustc version 1.48). However, probably due to the rust compiler only referencing jemalloc, and the jemalloc-sys crate forcing a prefix on musl targets, this seems to have no effect.

[Cogitri]

FWIW Rust recently got a new allocator with musl 1.2 (mallocng), so it might be worth to benchmark that again :)
We've been avoiding jemalloc whereever possible on Alpine Linux and even patched it out of some packages, IIRC it doesn't work too well on musl, but maybe that has changed since we last tried using it.

[msrd0]

I don't know much about jemalloc, I was just playing around to see if it improved stuff and had to realize that the compiler didn't seem to use it. So I don't know if the performance issues I'm seeing are caused by the allocator, but other rust projects like ripgrep have seen the allocator change to cause performance regressions on musl targets (and that one is also not patched on alpine).

Unfortunately, alpine:edge with musl 1.2.2_pre6-r0 and rust 1.47 from the alpine repos is consistently 5 to 10 seconds slower than alpine with musl 1.1.24-r10 and rust 1.47 compiled with a barely modified APKBUILD from aports (that is, enabling clippy and rustfmt tools). So it does not seem like the new allocator changed a great deal. Also, alpine is up to 30% slower compared to the official rust docker image when running cargo build for gotham. My docker host system is ArchLinux in case it matters (which performs the best but that's an unfair comparison).

[haraldh]

Opened a new issue against the musl version used in building #91178

[yshui]

is it possible to have dylib/cdylib with +crt-static?

[bjorn3]

Musl doesn't support that as the dynamic linker is part of libc and it doesn't support multiple copies of libc in a single process like on basically every Unix. The only mainstream system where that is possible is Windows as there the dynamic linker is part of the kernel and libc is not the system abi and multiple libc copies can co-exist in a single process.

[yshui]

i believe it would be possible to create "statically linked" shared library. i.e. all libc symbols are resolved at link time and the shared library is completely self-contained that it neither exposes nor references any libc symbols. we would also need the equivalent of -fno-semantic-interposition etc., and other things i might have missed, but i think it is doable at least in theory.

it might cause some confusion since two symbols ostensibly have the same name can actually be completely unrelated to each other, (which, btw, already happens in rust today if my project happens to use different versions of the same crate), but the shared library should otherwise work.

[bjorn3]

There are various pieces of global state in libc as well as some process state like the tls register for which each libc instance assumes exclusive access.

[yshui]

Ah right, yeah I forgot about that. That definitely makes this unsupportable from Rust's perspective. There goes my dreams :'(

OTOH it's probably not that bad? I'd imagine process state and tls register setup generally happen before main and probably isn't going to be touched again when a shared library is loaded. I can probably still make it work as a hack.

[bjorn3]

TLS registering also happens on every dlopen and pthread_create, so don't use either from within your statically linked dylib. And you better make sure that the host program uses musl as libc as errno is stored at a fixed offset from the TLS register, which other libc's may use for other purposes.