Job server strongly conflicts with multithreaded linkers

[clarfonthey]

Problem

Normally, cargo is only ever indirectly running one linker at a time, at the very end of compilation, and so, multithreaded linkers don't really matter. However, for cargo test, these linkers might be run in parallel, and cargo makes the assumption that all of these linkers are single-threaded, which is not the case for mold.

This can result in a potentially quadratic number of threads being run at the same time, which can crash systems.

Proposed Solution

One simple solution for the specific case of mold as a linker is to pass the MOLD_JOBS=1 environment variable to rustc, which will then be passed to the linker, which will then tell mold to ensure that the number of instances of the linker running at the same time is only ever one. This still means that the number of concurrent threads can be double what the CPU supports, since there can still be concurrent rustc jobs at the same time as the linker, but it won't end up being quadratic. However, this makes more sense as a mitigation done by users instead of something done by Cargo, since it affects the environment for more users than it needs to affect.

A potentially better solution is to delay all linking tasks until the very end of compilation, and have the option to run them serially instead of in parallel. This could be detected automatically via -Clink-args=-fuse-ld=mold, but more it would probably be fine just as a configurable flag initially. It's not really possible to separate the linking from rustc at the moment, but at least limiting the rustc calls that are expected to call a linker until the end seems reasonable.

Notes

At some point, it might be desirable to have the linker fully controlled by cargo, so that there isn't a problem not running them in parallel at the end of compilation. However, this is substantially larger of a change than the one proposed, so, it's left out for now.

[weihanglo]

Thanks for the idea and the workaround for mold. I didn't know the trick until today!

It seems like a duplicate of #9157. We have a general issue for parallelism controls in #12912. I am going to close this in favor of those. Let us know if you think the issue should be kept open separately :)

[clarfonthey]

Ah, I had tried to search for previous issues but missed that one. Thank you!