rust stable 1.78 using 'x86_64-pc-windows-gnu' host tool to build 'thumbv7em-none-eabihf' arm bare metal target got rust-lld failed

[pbl-pw]

Failed reason is can't find 'libgcc_s_seh-1.dll' and 'libwinpthread-1.dll'
when fallback to 1.77.2 or using 'x86_64-pc-windows-msvc' host tool then every thing is ok

[mati865]

Those libraries are provided alongside other Rust binaries:

$ ls ~/.rustup/toolchains/stable-x86_64-pc-windows-gnu/bin/
cargo.exe*         cargo-fmt.exe*      libgcc_s_seh-1.dll*   rustc.exe*                          rustdoc.exe*  rust-gdb*     rust-lldb*
cargo-clippy.exe*  clippy-driver.exe*  libwinpthread-1.dll*  rustc_driver-35a7c4c28db0aaf0.dll*  rustfmt.exe*  rust-gdbgui*  std-a1f74822451877d1.dll*

Note rust-lld is located inside rustlib:

$ file ~/.rustup/toolchains/stable-x86_64-pc-windows-gnu/lib/rustlib/x86_64-pc-windows-gnu/bin/rust-lld.exe
/c/Users/mateusz/.rustup/toolchains/stable-x86_64-pc-windows-gnu/lib/rustlib/x86_64-pc-windows-gnu/bin/rust-lld.exe: PE32+ executable (console) x86-64, for MS Windows, 17 sections

I rust-lld should be called with PATH set to ~/.rustup/toolchains/stable-x86_64-pc-windows-gnu/bin/, or at least used to be. Do you have an example that reproduces the problem? Hopefully, we can bisect it to easily find the problem.

[pbl-pw]

I have tested using clean rust host toolchain installed by rustinit, and just build cortex-m-quickstart, then it reproduce the problem.
note: Cargo.toml in cortex-m-quickstart need change 'package.name' and 'bin.name' before build, because they have '{{' and '}}'

[mati865]

searched nightlies: from nightly-2024-02-01 to nightly-2024-03-16
regressed nightly: nightly-2024-02-22
searched commit range: bb59453...3406ada
regressed commit: 0987e41

bisected with cargo-bisect-rustc v0.6.8

Host triple: x86_64-pc-windows-gnu
Reproduce with:

cargo bisect-rustc --start=1.77.0 --end=1.78.0 --target thumbv7m-none-eabi --prompt -c rust-mingw --preserve --script C:/Program Files/Git/usr/bin/bash.exe -- ./test.sh

Won't be as easy to debug as I hoped.
@weihanglo do you have any idea which change could case it?

EDIT: perhaps rust-lang/cargo#13468?
No idea how to test it without CI builds though.

[weihanglo]

If I understand correctly, when ~/.rustup/toolchains/stable-x86_64-pc-windows-gnu/bin/ is in PATH on Windows, it makes no difference. What the removed code path did was adding that to PATH. In rustup 1.27.1, RUSTUP_WINDOWS_PATH_ADD_BIN was changed to false by default (rust-lang/rustup#3703), so probably the rustup change is related.

@mati865 is it possible for you to add RUSTUP_WINDOWS_PATH_ADD_BIN=true and check it again?
I don't have a Windows machine available at hand :(

Also, could you check if rustc appends host tool bin to PATH? It looks like it but I am not particularly sure.

[weihanglo]

Would you mind sharing the output of rustup --version that shows the version of rustup you are using, @pbl-pw?

[mati865]

Thought it might be important to add; all my tests were made with only ~/.cargo/bin present in PATH:

$ echo $PATH
/c/Users/mateusz/.cargo/bin

If I understand correctly, when ~/.rustup/toolchains/stable-x86_64-pc-windows-gnu/bin/ is in PATH on Windows, it makes no difference.

Yes, but apparently it's not in PATH when tools from ~/.rustup/toolchains/stable-x86_64-pc-windows-gnu/lib/rustlib/x86_64-pc-windows-gnu/bin are called. $ PATH=$PATH:~/.rustup/toolchains/stable-x86_64-pc-windows-gnu/bin cargo +stable b works fine so that sort of confirms it.

In rustup 1.27.1, RUSTUP_WINDOWS_PATH_ADD_BIN was changed to false by default (rust-lang/rustup#3703), so probably the rustup change is related.

@mati865 is it possible for you to add RUSTUP_WINDOWS_PATH_ADD_BIN=true and check it again? I don't have a Windows machine available at hand :(

Adding that env makes no difference:

$ rustup -V
rustup 1.27.1 (54dd3d00f 2024-04-24)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.80.0-nightly (a330e4959 2024-06-04)`

$ RUSTUP_WINDOWS_PATH_ADD_BIN=true cargo +stable b
...
error: linking with `rust-lld` failed: exit code: 0xc0000135

Also, could you check if rustc appends host tool bin to PATH? It looks like it but I am not particularly sure.

I can do some builds during the weekend.

Also, could you check if rustc appends host tool bin to PATH? It looks like it but I am not particularly sure.

Judging from that code, it only adds ~/.rustup/toolchains/stable-x86_64-pc-windows-gnu/lib/rustlib/x86_64-pc-windows-gnu/bin.

[weihanglo]

My fault. Should be RUSTUP_WINDOWS_PATH_ADD_BIN=1 not true.

[mati865]

Oh, I had missed that too 😄
RUSTUP_WINDOWS_PATH_ADD_BIN=1 cargo +stable b works fine.

[pbl-pw]

@weihanglo

rustup --version
rustup 1.27.1 (54dd3d00f 2024-04-24)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.78.0 (9b00956e5 2024-04-29)`

I set RUSTUP_WINDOWS_PATH_ADD_BIN=1 then build works fine too

[weihanglo]

@ehuss do you have any thought on this? Should we revert rust-lang/cargo#13468 or should rustc do something else instead?
I don't know how msvc works but gnu doesn't.

[ChrisDenton]

Another option would be to add it to the end of PATH. So it has the lowest precedence but any necessary DLLs it provides can still be found.

[pbl-pw]

@weihanglo I guess msvc works because it's required dlls has been installed by Windows OS or MSVC build tool

[weihanglo]

@ChrisDenton are you talking about a fix in cargo or rustc?
What happens if we invoke rustc directly to cross-compile? If it doesn't work either, I feel like this should be addressed on rustc or linker invocation side.

[mati865]

Runtime libraries are located alongside rustc, so invoking it directly would work fine.
The same cannot be said about bins inside ~/.rustup/toolchains/stable-x86_64-pc-windows-gnu/lib/rustlib/x86_64-pc-windows-gnu/bin, but is it even supported to invoke them manually?

[ehuss]

if we invoke rustc directly to cross-compile? If it doesn't work either, I feel like this should be addressed on rustc or linker invocation side.

This would be my inclination that rustc ensures that it has the PATH set up correctly. We just need to make sure that things like rust-lang/cargo#8531 and rust-lang/rustup#3036 don't regress again.

I suspect adding the bin directory to PATH will make proc-macros fail recursive calls to rustc or cargo via rustup. I'm not sure how much we should care about that. The only solution I can think there is to separate the DLLs into a separate directory.

[ColinFinck]

I'm solving this now in my PR #128876 by shipping the required runtime DLLs along with rust-lld.exe too.
It must be possible to run rust-lld.exe without relying on any PATH variable being set. We already ship these runtime DLLs alongside rustc.exe, so doing the same for rust-lld.exe feels like a reasonable thing to do. This solution won't require RUSTUP_WINDOWS_PATH_ADD_BIN or any other modification to the PATH environment variable.