Fix x test core when download-rustc is enabled

[jyn514]

Fix x test --stage 2 core when download-rustc is enabled

This works by building std from source instead of downloading it, for library tests only.

This was somewhat complicated because of the following requirements:

1. Unconditionally downloading libstd breaks x test core, because coretests requires the std loaded from the sysroot to match the std that's currently being tested.
2. Unconditionally rebuilding libstd breaks x test ui-fulldeps librustdoc, because anything loading rustc_private needs to use the same libstd that rustc was built with.

Break the knot by introducing a new stage2-test-sysroot, used only for testing std itself. This
holds a freshly compiled std, while stage2 and ci-rustc-sysroot still hold the downloaded std.

This also extends the existing cp_filtered in Sysroot to apply to the rust-std component, not just the rustc-dev component, to avoid having both versions of std in stage2-test-sysroot.

Fixes #110352.

[rustbot]

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

[jyn514]

Update on this: since #110699 merged, x test ui is fixed, but x test ui-fulldeps rustdoc are both broken with errors similar to

error[E0460]: found possibly newer version of crate `std` which `thin_vec` depends on
  --> src/librustdoc/lib.rs:24:1
   |
24 | extern crate thin_vec;
   | ^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: perhaps that crate needs to be recompiled?
   = note: the following crate versions were found:
           crate `std`: /home/jyn/src/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-9768f62f7b152226.rlib
           crate `std`: /home/jyn/src/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-9768f62f7b152226.so
           crate `thin_vec`: /home/jyn/src/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libthin_vec-812d4d34d6de684d.rmeta

I'm too tired tonight to figure out if these are fixable or not; they're similar to the errors fixed in #110121 so maybe it's possible to do something clever like run x test core against the stage1 sysroot but all rustc and tool tests against the stage2 sysroot, and rely on using ci-rustc being the same in both stages to avoid issues? But that's really hacky and I worry it will make --stage even more confusing ... maybe we should disallow --stage altogether when download-rustc is enabled?

Anyway, I'm not going to dig into it more tonight.

[jyn514]

maybe it's possible to do something clever like run x test core against the stage1 sysroot but all rustc and tool tests against the stage2 sysroot, and rely on using ci-rustc being the same in both stages to avoid issues

(actually even now i think this is a terrible idea, if i'm going to do something like it at all i'd introduce a new stage2-std-sysroot directory or something)

[jyn514]

actually hold on - the only reason I added the downloaded std to the sysroot in the first place was so ui-fulldeps tests would pass (since they need the std loaded at runtime to match the one the downloaded rustc was linked against). I think if we disable those tests when download-rustc is enabled we can avoid these issues altogether. That seems ok to me; ui-fulldeps tests are only testing the compiler itself, not std, and if download-rustc is enabled that means you haven't modified the compiler.

[jyn514]

the only reason I added the downloaded std to the sysroot in the first place was so ui-fulldeps tests would pass

this is not actually true - the version of std used for the downloaded rustc also needs to match the version used for the freshly built rustdoc:

error[E0460]: found possibly newer version of crate `std` which `thin_vec` depends on
  --> src/librustdoc/lib.rs:24:1
   |
24 | extern crate thin_vec;
   | ^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: perhaps that crate needs to be recompiled?
   = note: the following crate versions were found:
           crate `std`: /home/jyn/src/rust/build/x86_64-unknown-linux-gnu/ci-rustc-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-9768f62f7b152226.rlib
           crate `std`: /home/jyn/src/rust/build/x86_64-unknown-linux-gnu/ci-rustc-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-9768f62f7b152226.so
           crate `thin_vec`: /home/jyn/src/rust/build/x86_64-unknown-linux-gnu/ci-rustc-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libthin_vec-812d4d34d6de684d.rmeta

[Mark-Simulacrum]

Sad that all this complexity is needed, but seems OK -- I don't see obvious ways to get around it.

@bors r+

[bors]

📌 Commit a80d69a has been approved by Mark-Simulacrum

It is now in the queue for this repository.