Cross compilation fails on Rust nightly when using proc macro crates due to GCC not understanding -fuse-ld=lld

[VorpalBlade]

Code

I tried this code:

❯ cargo new hello_world
    Creating binary (application) `hello_world` package
cnote: see more `Cargo.toml` keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
❯ cd hello_world/
❯ cross +nightly build --target aarch64-unknown-linux-gnu
[cross] warning: using newer rustc `1.80.0-nightly (d84b90375 2024-05-19)` for the target. Current active rustc on the host is `rustc 1.78.0 (9b00956e5 2024-04-29)`.
 > Update with `rustup update`
   Compiling hello_world v0.1.0 (/project)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.82s
❯ cargo add syn
    Updating crates.io index
      Adding syn v2.0.65 to dependencies
             Features:
             + clone-impls
             + derive
             + parsing
             + printing
             + proc-macro
             - extra-traits
             - fold
             - full
             - test
             - visit
             - visit-mut
    Updating crates.io index
❯ cross +nightly build --target aarch64-unknown-linux-gnu
[cross] warning: using newer rustc `1.80.0-nightly (d84b90375 2024-05-19)` for the target. Current active rustc on the host is `rustc 1.78.0 (9b00956e5 2024-04-29)`.
 > Update with `rustup update`
   Compiling proc-macro2 v1.0.83
   Compiling unicode-ident v1.0.12
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/rust/lib/rustlib/x86_64-unknown-linux-gnu/bin:/rust/lib/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/rust/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcAyUfnb/symbols.o" "/target/debug/build/proc-macro2-b1e0569744dc7f5d/build_script_build-b1e0569744dc7f5d.build_script_build.48684d15272f5bdb-cgu.0.rcgu.o" "/target/debug/build/proc-macro2-b1e0569744dc7f5d/build_script_build-b1e0569744dc7f5d.build_script_build.48684d15272f5bdb-cgu.1.rcgu.o" "/target/debug/build/proc-macro2-b1e0569744dc7f5d/build_script_build-b1e0569744dc7f5d.crzselpkxut2yhaw0phvg3mu9.rcgu.o" "-Wl,--as-needed" "-L" "/target/debug/deps" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-de48b8168d6cf4fa.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9f3477fb95a0bba7.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-210d920812faea91.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-f3d3451767410a17.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-1a79dd36d08251de.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-886230e7120831b2.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-2caea079085a58a2.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-a9d7d97cdc65a449.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-5727477b0a78105a.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-7e555563aa211118.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-c8f13465f1a795b2.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-ee5b5774583426df.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-14df174c91007922.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-e8bfe52be756260a.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-6275035a459b3ada.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-b6892f3c52c68f01.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-f72b956e24d1de70.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-632ae0f28c5e55ff.rlib" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-e8b7e96e438f08f6.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-B/rust/lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld" "-fuse-ld=lld" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/self-contained" "-o" "/target/debug/build/proc-macro2-b1e0569744dc7f5d/build_script_build-b1e0569744dc7f5d" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: cc: error: unrecognized command line option '-fuse-ld=lld'
          

error: could not compile `proc-macro2` (build script) due to 1 previous error

I expected to see this happen: Cross compilation should work fine

Instead, this happened: Cross compilation fails due to gcc not accepting -fuse-ld=lld

Version it worked on

It most recently worked on: Nightly as of about a week ago, but it also works fine on stable.

Version with regression

rustc --version --verbose:

❯ rustc +nightly --version
rustc 1.80.0-nightly (d84b90375 2024-05-19)

Backtrace

Backtrace

<backtrace>

@rustbot modify labels: +regression-from-stable-to-nightly -regression-untriaged

[VorpalBlade]

Also reported as cross-rs/cross#1496 but it seems to me that it is a rust regression that rust tries to use a gcc option when the installed GCC version is too old to support said option.

[saethlin]

I agree that this is a regression. Our tier 1 targets support GLIBC 2.17+ which dates back to 2012 but -fuse-ld=lld in gcc was merged in 2018. While we do not explicitly declare support for any gcc or distro version, GLIBC 2.17+ is generally interpreted as RHEL/CentOS 7.

Unfortunately, the platforms that the toolchain can run on often constrains what platforms can be targeted. We currently do not provide a mechanism to target GLIBC versions older than the host.

So I do not think it is not a big stretch to argue that passing -fuse-ld=lld without a check for a compatible gcc breaches the target tier support that we declare.

[VorpalBlade]

So I do not think it is not a big stretch to argue that passing

@saethlin The double negation is hurting my brain, what do you actually mean here?

[saethlin]

I mean that the new behavior violates of the spirit of the target tier policy, but not the very precise guarantees we document.

[lqd]

I had expected us to have a gcc version check already, especially since our dist i686/x64 linux CI builders are based on CentOS 7, but it seems the original -Zgcc-ld=lld implementation doesn't have such a version check, and that CI had gcc 9.5 available because it actually builds it from source using the GCC 4.8.5 package available.

I'll look into adding the version check.

[lqd]

Hmm, it seems

• version detection was deliberately avoided in the implementation, not forgotten
• and that this issue of old GCCs was discussed in How to find LLD with old versions of gcc? #97402 where consensus at the time (amongst participants, not a wide-reaching discussion to be clear) seemed to be "there was no need to support GCCs this old". Which also could explain why we don't have GCC 4 when running tests on CI.

Maybe we need to reassess both of these and discuss more broadly, and/or document more precisely what we intend to rely on in the target support documentation, and/or more easily offer feature detection for users where these cases are important.

The latter could be a workflow like the following, where users want to dynamically opt-out of using lld:

• until stabilization, check if using a nightly compiler -- and use -Z/-Zunstable-options flags that won't be needed in the future
• running rustc --print linker-features returns whether lld is supported (we'd need to add support for linker-features in rustc --print first of course). We could also check if rust-lld exists in the sysroot here, instead of only as a fallback during compilation like rust-lld: fallback to rustc's sysroot if there's no path to the linker in the target sysroot #125263.
• if that succeeds (handwaving details), then users wanting to opt-out could do so with -Z/-Clinker-features=-lld. Of course, here they could also add their own dynamic opt-in/opt-out checks, like a GCC version check.
• otherwise, the default linker would not be lld in the first place, and nothing would need to be done

That would allow for users supporting multiple versions of the compiler and multiple versions of GCC if they need to.

cc @petrochenkov what do you think?

[VorpalBlade]

My use case is building in CI, both for testing and for distributable binaries. I don't run old distros myself (I run Arch, which is rolling release). I just rely on cross selecting a suitable docker image such that it can cross compile binaries that are portable to any relevant glibc based Linux distros. And it seems that works okay (even on nightly), until it tries to build a proc macro that should execute on the host in the docker container.

Now I would say that "building portable binaries for distribution" is an important use case for Rust. Any binary crate that is not targeted at rust developers might want to do this. I don't expect my users to have a rust toolchain (or any toolchain at all) installed. So asking them to use cargo install does not make much sense.

Because of the unfortunate situation on Linux with glibc backward compatibility, the only reasonable way to do this is to build on an old distro (since glibc makes sure the sure the binary is forwards compatible, but not the reverse).

I believe this is a use case that needs to work. I don't particularly care about how it is made to work. Maybe rust can just bypass cc and call lld directly for example.

[apiraino]

WG-prioritization assigning priority (Zulip discussion).

@rustbot label -I-prioritize +P-high

[petrochenkov]

On re-read it looks like in #97402 we largely assumed that LLD is enabled explicitly somehow. In that case it's quite reasonable to not support LLD on systems where GCC doesn't support it, because that's a quite small version window (between the appearance of functioning LLD and -fuse-ld=lld support in GCC).

But now LLD is enabled implicitly in rustc and that increases the unsupported version window significantly (now it's from around 2012 to around 2018).
So I agree that we need to fix this somehow.

[petrochenkov]

One possible solution is a hack similar to fallback for unsupported -no-pie or -static-pie -

rust/compiler/rustc_codegen_ssa/src/back/link.rs

Lines 780 to 861 in f0038a7

	// Check to see if the link failed with an error message that indicates it
	// doesn't recognize the -no-pie option. If so, re-perform the link step
	// without it. This is safe because if the linker doesn't support -no-pie
	// then it should not default to linking executables as pie. Different
	// versions of gcc seem to use different quotes in the error message so
	// don't check for them.
	if matches!(flavor, LinkerFlavor::Gnu(Cc::Yes, _))
	&& unknown_arg_regex.is_match(&out)
	&& out.contains("-no-pie")
	&& cmd.get_args().iter().any(|e| e.to_string_lossy() == "-no-pie")
	{
	info!("linker output: {:?}", out);
	warn!("Linker does not support -no-pie command line option. Retrying without.");
	for arg in cmd.take_args() {
	if arg.to_string_lossy() != "-no-pie" {
	cmd.arg(arg);
	}
	}
	info!("{:?}", &cmd);
	continue;
	}
	// Detect '-static-pie' used with an older version of gcc or clang not supporting it.
	// Fallback from '-static-pie' to '-static' in that case.
	if matches!(flavor, LinkerFlavor::Gnu(Cc::Yes, _))
	&& unknown_arg_regex.is_match(&out)
	&& (out.contains("-static-pie") || out.contains("--no-dynamic-linker"))
	&& cmd.get_args().iter().any(|e| e.to_string_lossy() == "-static-pie")
	{
	info!("linker output: {:?}", out);
	warn!(
	"Linker does not support -static-pie command line option. Retrying with -static instead."
	);
	// Mirror `add_(pre,post)_link_objects` to replace CRT objects.
	let self_contained_crt_objects = self_contained_components.is_crt_objects_enabled();
	let opts = &sess.target;
	let pre_objects = if self_contained_crt_objects {
	&opts.pre_link_objects_self_contained
	} else {
	&opts.pre_link_objects
	};
	let post_objects = if self_contained_crt_objects {
	&opts.post_link_objects_self_contained
	} else {
	&opts.post_link_objects
	};
	let get_objects = |objects: &CrtObjects, kind| {
	objects
	.get(&kind)
	.iter()
	.copied()
	.flatten()
	.map(|obj| {
	get_object_file_path(sess, obj, self_contained_crt_objects).into_os_string()
	})
	.collect::<Vec<_>>()
	};
	let pre_objects_static_pie = get_objects(pre_objects, LinkOutputKind::StaticPicExe);
	let post_objects_static_pie = get_objects(post_objects, LinkOutputKind::StaticPicExe);
	let mut pre_objects_static = get_objects(pre_objects, LinkOutputKind::StaticNoPicExe);
	let mut post_objects_static = get_objects(post_objects, LinkOutputKind::StaticNoPicExe);
	// Assume that we know insertion positions for the replacement arguments from replaced
	// arguments, which is true for all supported targets.
	assert!(pre_objects_static.is_empty() || !pre_objects_static_pie.is_empty());
	assert!(post_objects_static.is_empty() || !post_objects_static_pie.is_empty());
	for arg in cmd.take_args() {
	if arg.to_string_lossy() == "-static-pie" {
	// Replace the output kind.
	cmd.arg("-static");
	} else if pre_objects_static_pie.contains(&arg) {
	// Replace the pre-link objects (replace the first and remove the rest).
	cmd.args(mem::take(&mut pre_objects_static));
	} else if post_objects_static_pie.contains(&arg) {
	// Replace the post-link objects (replace the first and remove the rest).
	cmd.args(mem::take(&mut post_objects_static));
	} else {
	cmd.arg(arg);
	}
	}
	info!("{:?}", &cmd);
	continue;
	}

.
If linker execution fails, we check its output and rerun it again after removing unsupported options.

[lqd]

I'll look into doing that, it seems preferable to doing the version detection on the happy path that I was working on.

[petrochenkov]

Another possible solution is to keep a fifth copy of lld-wrapper with a name ld (or ld.exe) in addition to these four

rust/src/bootstrap/src/lib.rs

Line 75 in f0038a7

const LLD_FILE_NAMES: &[&str] = &["ld.lld", "ld64.lld", "lld-link", "wasm-ld"];

Then we won't have to pass -fuse-ld=lld at all, only -B, at least in the self-contained case.
This doesn't solve the problem in the non-self-contained mode though.

Also self-contained lld-named-ld will conflict with self-contained real-ld on targets where both are shipped (like windows-gnu).
But this can also be solved, by moving them to separate directories.

Let's try the linker rerun hack first, I think.