Use lld by default on x64 Ubuntu 20.04 LTS

[Gankra]

This is a metabug, constraining the unbound scope of #39915.

What is lld

A linker that's part of the llvm project, which is desirable for two reasons:

• it's very friendly to cross compilation (hence its emphasis for embedded targets)
• it's very fast (often runs in half the time as Gold -- linking can take several minutes for big projects (rustc, servo, etc.) and linking can be a huge % of the compile with incremental builds, so halving this runtime is a Big Deal.)

Rust currently ships its own copy of lld which it calls rust-lld. This is used by default to link bare-metal targets and wasm.

Goal

The goal of this metabug is to use rust-lld by default on a major x64 linux distro. I have arbitrarily chosen Ubuntu 20.04 LTS as our target. With all likelihood, this will incidentally get it working/enabled on many other linux distros, BSDs, and other similar platforms, but I am purposefully constraining the scope to one distro for the sake of focusing the effort.

My understanding is that the ELF backend is quite well maintained. Quoting lld's own landing page:

We are currently working closely with the FreeBSD project to make LLD default system linker in future versions of the operating system, so we are serious about addressing compatibility issues. As of February 2017, LLD is able to link the entire FreeBSD/amd64 base system including the kernel. With a few work-in-progress patches it can link approximately 95% of the ports collection on AMD64. For the details, see FreeBSD quarterly status report.

Hopefully this has continued to improve, and using lld will be a slam dunk. Although I specify that we should use rust-lld, it may be necessary/desirable to detect and use system copies of lld.

Blocking Issues

• Support feeding rust-lld into gcc (and clang) #71519 - support making gcc use rust-lld
• Coverage is not generated when using lld as linker #71233 - coverage information not generated when using lld

[cuviper]

As a distro maintainer (Fedora and RHEL), I would definitely want this to use the system lld, at least when we're talking about the system rustc that I compile.

[est31]

There is no such thing as a Ubuntu 20.04 LTS target platform (unlike wasm or the bare embedded targets you mention), but restricting it to x86_64-unknown-linux-gnu would be a good idea.

[Gankra]

yes, doing a good job with Ubuntu will likely get a lot of x86_64-unknown-linux-gnu working. but i'd rather get one specific (major) distro working first, since we're talking about integrating with system toolchains here.

[est31]

@Gankra I assumed you meant a target based check because the two other cases you mention (wasm and bare metal) are target based. Now I'm even more confused. What do you want to do? A host based check? As in: cargo or rustc trying to invoke lsb_release -a or something at its startup and enable LLD if it's Ubuntu 20.04? What if you cross compile to Windows or Mac? Or do you just want to talk to Ubuntu's rustc packagers so that they make it use LLD in the Ubuntu package?

[Gankra]

I don't intend to explicitly check for the distro/release, but rather am willing to have toolchain checks which succeed on vanilla Ubuntu 20.04 but e.g. don't succeed on the latest Arch or older Ubuntus.

So we don't need to worry about supporting using rust-lld with older versions of gcc/clang or whatever other tools, as long as our checks are solid enough to detect those things and fall back to using the native ld.

The starting point of this would be to not have this detection set up. We would instead blindly try to use rust-lld on x64 linux gnu targets if it's explicitly requested with the -C linker or -C linker-flavor options. This means using those options could lead to catastrophic failures in many configurations, but we will do our best to ensure they work on typical Ubuntu 20.04 installs (e.g. no older/weird toolchains installed).

In all likelihood this will incidentally get most major modern distro releases working well, but I would regard that as a Happy Accident, and not a priority. It would however provide a tool for distro maintainers to check if rust-lld mode works for them, which is good and useful.

After that we would add the detection or expand support until we could have the compiler try to automatically enable rust-lld without an explicit request on x64 linux gnu targets (possibly this step would be done by Cargo and not rustc). Once this is in good enough shape (everything working, not necessarily using rust-lld), we would enable this behaviour in production.

From there we will have a solid foundation for folks to push forward rust-lld support on their platforms by either making our toolchain detection more sophisticated, or by updating their distros to meet our requirements.

[est31]

@Gankra thanks for the explainer. Have there been any "please test LLD" threads? Those were done e.g. for pipelined compilation and might be helpful in this instance as well.

Also maybe one should think about adding an LLD label for issues so that they can be tracked.

[Gankra]

we should have an mvp of #71519 before we ask folks to try out linux lld support

[Gankra]

(CC to @rui314 in case they're interested in the fact that we're investigating this -- see also #71520)

[sledgehammervampire]

@Gankra I've made an MVP of #71519. What's next?

Edit: It should work for pure Rust projects. It doesn't always work when C libraries are dynamically linked.

[Logarithmus]

@1000teslas provided that C libraries have been fixed in #86740, what's the next thing to do to finally use lld by default? It's more than a year since this issue was created!

[sledgehammervampire]

@Logarithmus I have no idea to be honest. I was pretty much just following what @Gankra suggested.

[Gankra]

@1000teslas oh wow, great work!

All that's left to do is advertise the new flag and provide some instructions for people to test out if it works correctly/and how it performs. I know there was some discussion about the final flag name, could you post instructions on how to use it here? Then someone more familiar with community rallying can encourage people to check it out.

[sledgehammervampire]

@Gankra It is used like this: -Z gcc-ld=lld as part of RUSTFLAGS.

Where should I post the instructions?

[Gankra]

I've prodded the compiler folks to do a writeup, we should be able to take care of the rest for you. I'll post a link the the internals/users thread when it's ready (hopefully very soon, sorry for the slow response on your initial ping).

[sledgehammervampire]

Unfortunately, it seems like my PR only works for a stage1 toolchain. I tried to dogfood my PR using a nightly toolchain from rustup on one of my pure-Rust projects and I get some weird errors. When I use ldd on the rust-lld from the nightly toolchain provided by rustup, I see: libLLVM-12-rust-1.55.0-nightly.so => not found, whereas I see no such dependency in my stage1 toolchain.

Then again, I'm using a "weird" distro (NixOS), so maybe it'll work on a more conventional Linux distro.

[bjorn3]

$ ls $(rustc --print sysroot)/lib
libLLVM-12-rust-1.55.0-nightly.so  librustc_driver-8922750e6d18b0c7.so  libstd-37fdd654777c296f.so  libtest-08487f8e282f19e0.so  rustlib

libLLVM-12-rust-1.55.0-nightly.so should be available in $(rustc --print sysroot)/lib. You may need to set LD_LIBRARY_PATH. Alternatively you can run rustup run nightly ldd /path/to/rust-lld.

[sledgehammervampire]

@bjorn3 Are you saying that even if my system's ldd can't find the location of libLLVM-12-rust-1.55.0-nightly.so, when rust-lld is invoked by rustc, it knows where to find libLLVM-12-rust-1.55.0-nightly.so?

Anyway, I get errors that look like

error: linking with `cc` failed: exit status: 1
<bunch of cc gobbledygook>
  = note: collect2: fatal error: execvp: No such file or directory

[bstrie]

@1000teslas I'd like to help get the word out about testing this more broadly, would you be able to join us in the compiler team's Zulip so that we can hash out the details? https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Need.20users.2Finternals.20thread.20for.20gcc-lld/near/245177775

[Tergeltengis]

what about aarch64-unknown-linux-musl-clang

[Nemo157]

[...] but I am purposefully constraining the scope to one distro for the sake of focusing the effort

This issue is purely about a single target (and even a single distro within that target), any other targets would fall under the more general issue linked in the OP or their own tracking issues.

[lqd]

status update: as of #124129, rustup-distributed rustc (not distros') on x86_64-unknown-linux-gnu uses the self-contained lld linker by default, (rust-lld, itself also distributed with rustup), on nightly only for now and starting from tomorrow's nightly.

[bstrie]

status update: as of #124129, rustup-distributed rustc (not distros') on x86_64-unknown-linux-gnu uses the self-contained lld linker by default, (rust-lld, itself also distributed with rustup), on nightly only for now and starting from tomorrow's nightly.

What's the next step here? Now that we have some experience with LLD on nightly, in order to proceed further we'd need to start identifying LLD-related bugs and determining which of them are blockers. I think the A-linkage tag is the appropriate tag to watch, but AFAICT there's a lot of stuff there unrelated to LLD specifically. Should we have an A-lld tag to keep better track?

And even beyond blocking bugs, surely we can think about the steps after that and start pursuing them concurrently. For example, are all the rustc command line flags properly designed, bikeshedded, implemented, and stabilized? What about the Cargo-level config options? (Following the history of how the linker is intended to be configured seems to be surprisingly difficult!)

And even once we have no more blocking bugs and we've solved the user interface question, have we determined if we're allowed to change the default? In other words, is changing the default linker a breaking change? (IME linker scripts are incompatible between LD and LLD, which is an example of something that's sure to break.) If so, then we'd need to wait until the next edition to change this over to be the default.

[the8472]

Is it still the case that lld lacks jobserver support? Is that going to be a blocker to turning it on by default?

Was this ever solved?

[lqd]

Probably not, and we're likely not going to implement it in lld ourselves. You could maybe hack something via setting the number of threads lld can use according to the number of tokens rustc has.

Though it may not be that big of a problem for us, since linking is usually at the latest stages of the process, where's less contention. We can at the very least document this and the issues it can cause, and if people absolutely need such support, they can switch back to bfd.

[the8472]

People have filed issues about too much linker parallelism causing OOM kills, e.g. rust-lang/cargo#9157

[lqd]

I'm aware, and also don't think firefox/chromium builds are representative. Jobserver support will not make memory consumption lower by itself. If people want to add some finer-grained link job control to cargo and rustc that would be cool.

Are you saying this is a blocker for landing lld on stable?

[the8472]

I suspect that it wouldn't just affect big projects but also people on machines low on free ram. At least a few perf items in the PR that enabled it in nightly regressed MaxRSS, and those are fairly small jobs, it might be more for bigger applications.

So we should at least make it part of the release announcement too if it's going to regress reliability for some users.

[lqd]

Oh fully agreed; I had already mentioned that the added parallelism may cause issues on resource-constrained environments back in the nightly announcement. Mentioning this, as well as the other more obscure issues related to GC sections or .ctors support, in the blog post, tracking issues, and stabilization report will be done for sure.

[dralley]

The status of this is a bit unclear - the "blocking issues" are resolved, are any of the remaining issues blocking?

[bstrie]

@lqd Can we get this nominated for compiler team discussion? Do we need a stabilization PR first?

[lqd]

There’s no need for the team to discuss this just yet, but we’re working on it: in the last few days, we’ve been checking all the issues that have been opened since the switch on nightly, doing crater runs, etc. There still are a few things to do before preparing a stabilization report.