Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rust-lld: fallback to rustc's sysroot if there's no path to the linker in the target sysroot #125263

Merged
merged 3 commits into from
May 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions compiler/rustc_codegen_ssa/messages.ftl
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,8 @@ codegen_ssa_rlib_only_rmeta_found = could not find rlib for: `{$crate_name}`, fo

codegen_ssa_select_cpp_build_tool_workload = in the Visual Studio installer, ensure the "C++ build tools" workload is selected

codegen_ssa_self_contained_linker_missing = the self-contained linker was requested, but it wasn't found in the target's sysroot, or in rustc's sysroot

codegen_ssa_shuffle_indices_evaluation = could not evaluate shuffle_indices at compile time

codegen_ssa_specify_libraries_to_link = use the `-l` flag to specify native libraries to link
Expand Down
10 changes: 9 additions & 1 deletion compiler/rustc_codegen_ssa/src/back/link.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3116,13 +3116,21 @@ fn add_lld_args(

let self_contained_linker = self_contained_cli || self_contained_target;
if self_contained_linker && !sess.opts.cg.link_self_contained.is_linker_disabled() {
let mut linker_path_exists = false;
for path in sess.get_tools_search_paths(false) {
let linker_path = path.join("gcc-ld");
linker_path_exists |= linker_path.exists();
cmd.arg({
let mut arg = OsString::from("-B");
arg.push(path.join("gcc-ld"));
arg.push(linker_path);
arg
});
}
if !linker_path_exists {
// As a sanity check, we emit an error if none of these paths exist: we want
// self-contained linking and have no linker.
sess.dcx().emit_fatal(errors::SelfContainedLinkerMissing);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if a distro wants to drop Rust lld binaries and rely on system lld?

Then the corresponding self-contained directories either won't exist or will be empty.
If we just push the -B options without checking then gcc will go through them and then proceed searching and finding lld on the system without errors.

If we report an error here, then such distros will also need to reset self-contained flags in target specs to false, probably through env vars like the current CFG_USE_SELF_CONTAINED_LINKER.
This will have a negative effect on portability of options like -Clink-self-contained=+linker between distros (if we need such portability).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I'm fine with reporting a conservative error here for now, it can always be relaxed later.
At least we'll be able see what breaks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another reason why I preferred the fallback compared to the error :3, because falling back to system lld is what we currently do in the case you mention of course.

But one can also say having the self-contained linker enabled is an opt-in for the toolchain maker, it goes hand in hand with bootstrap building lld and putting in the sysroot. Not having both can be seen as a mismatch, and that situation would be properly modeled as -Clink-self-contained=-linker -Clinker-features=+lld.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that I only check for the existence of the gcc-ld folder, so in theory an empty directory with no lld binaries would pass this test 🤡

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One problem with the fallback is that if there is no system lld and we try to use it anyway, the error is really bad:

  = note: collect2: fatal error: cannot find 'ld'
          compilation terminated.

It doesn't even say that it was trying to find lld, mentioning ld instead which is not what it was looking for.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if a distro wants to drop Rust lld binaries and rely on system lld?

Sounds to me like they should then disable self-contained linking in their rustc builds.

Copy link
Member Author

@lqd lqd May 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's definitely not going to be winning an oscar of the best error message, but also it's in gcc, a non-zero number of people are used to it, and look for the -fuse-ld option that is in the command just above that note. (Maybe it also depends on the gcc version and newer ones have improved it 🤔)

But it's neither here nor there, the PR currently emits an error and we'll see what happens with distros. Right now, they'll be fine, and in the future they just may need to learn how to have the system lld fallback which is how you and I describe, quite sensible imho.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a non-zero number of people are used to it, and look for the -fuse-ld option that is in the command just above that note. (Maybe it also depends on the gcc version and newer ones have improved it 🤔)

I dare say most Rust users are not used to it. I certainly was entirely flabbergasted, and I know more than the average Rust programmer about how the compiler works. If an error is too obscure for me I think we can safely assume that it is useless-or-worse for the majority of our users.

That should obviously be reported against gcc, I just didn't have the time for that. But in general we try to work around weaknesses of other tools we employ when we are aware of them.

But as you say indeed for this PR this isn't relevant. I was just replying to @petrochenkov's arguments.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That should obviously be reported against gcc, I just didn't have the time for that. But in general we try to work around weaknesses of other tools we employ when we are aware of them.

I think we could actually help with this issue in rustc by intervening into the linker error and emitting an extra note.
Like in #125417, but without retrying.

}
}

// 2. Implement the "linker flavor" part of this feature by asking `cc` to use some kind of
Expand Down
4 changes: 4 additions & 0 deletions compiler/rustc_codegen_ssa/src/errors.rs
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,10 @@ pub struct UnableToExeLinker {
#[diag(codegen_ssa_msvc_missing_linker)]
pub struct MsvcMissingLinker;

#[derive(Diagnostic)]
#[diag(codegen_ssa_self_contained_linker_missing)]
pub struct SelfContainedLinkerMissing;

#[derive(Diagnostic)]
#[diag(codegen_ssa_check_installed_visual_studio)]
pub struct CheckInstalledVisualStudio;
Expand Down
8 changes: 8 additions & 0 deletions compiler/rustc_session/src/filesearch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,14 @@ pub fn make_target_lib_path(sysroot: &Path, target_triple: &str) -> PathBuf {
PathBuf::from_iter([sysroot, Path::new(&rustlib_path), Path::new("lib")])
}

/// Returns a path to the target's `bin` folder within its `rustlib` path in the sysroot. This is
/// where binaries are usually installed, e.g. the self-contained linkers, lld-wrappers, LLVM tools,
/// etc.
pub fn make_target_bin_path(sysroot: &Path, target_triple: &str) -> PathBuf {
let rustlib_path = rustc_target::target_rustlib_path(sysroot, target_triple);
PathBuf::from_iter([sysroot, Path::new(&rustlib_path), Path::new("bin")])
}

#[cfg(unix)]
fn current_dll_path() -> Result<PathBuf, String> {
use std::ffi::{CStr, OsStr};
Expand Down
25 changes: 17 additions & 8 deletions compiler/rustc_session/src/session.rs
Original file line number Diff line number Diff line change
Expand Up @@ -449,15 +449,24 @@ impl Session {
)
}

/// Returns a list of directories where target-specific tool binaries are located.
/// Returns a list of directories where target-specific tool binaries are located. Some fallback
/// directories are also returned, for example if `--sysroot` is used but tools are missing
/// (#125246): we also add the bin directories to the sysroot where rustc is located.
pub fn get_tools_search_paths(&self, self_contained: bool) -> Vec<PathBuf> {
let rustlib_path = rustc_target::target_rustlib_path(&self.sysroot, config::host_triple());
let p = PathBuf::from_iter([
Path::new(&self.sysroot),
Path::new(&rustlib_path),
Path::new("bin"),
]);
if self_contained { vec![p.clone(), p.join("self-contained")] } else { vec![p] }
let bin_path = filesearch::make_target_bin_path(&self.sysroot, config::host_triple());
let fallback_sysroot_paths = filesearch::sysroot_candidates()
.into_iter()
.map(|sysroot| filesearch::make_target_bin_path(&sysroot, config::host_triple()));
let search_paths = std::iter::once(bin_path).chain(fallback_sysroot_paths);

if self_contained {
// The self-contained tools are expected to be e.g. in `bin/self-contained` in the
// sysroot's `rustlib` path, so we add such a subfolder to the bin path, and the
// fallback paths.
search_paths.flat_map(|path| [path.clone(), path.join("self-contained")]).collect()
} else {
search_paths.collect()
}
}

pub fn init_incr_comp_session(&self, session_dir: PathBuf, lock_file: flock::Lock) {
Expand Down
Loading