Fix spurious error when running `build --stage 2 compiler/rustc` #96859

jyn514 · 2022-05-09T05:32:28Z

#89759 introduced a panic:

Assembling stage3 compiler (x86_64-apple-darwin)
thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/Users/user/rust2/build/x86_64-apple-darwin/stage2-rustc/x86_64-apple-darwin/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24

This wasn't actually a bug in that PR - the problem was that x build --stage 3
is broken, and has been for quite some time, even ignoring the stamp file error:

thread 'main' panicked at 'src.symlink_metadata() failed with No such file or directory (os error 2) ("failed to determine metadata of /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage2-rustc/x86_64-unknown-linux-gnu/release/rustc-main")', src/bootstrap/lib.rs:1414:24

It needs to take into account whether the artifacts from stage1 are being reused, rather than blindly assuming rustc will be recompiled.
Doing so is kind of annoying, because it requires knowing the target the compiler is being built for.
Instead, just revert to the old behavior of build --stage 2 compiler/rustc, which avoids trying to create the sysroot in the first place.

Note that this does not help with #96798 (comment), which really does need the sysroot available. I think requiring the stage 3 sysroot there is a bug - we should either use test --stage 1 in CI, or change the stage numbering so that test --stage 2 matches the behavior of other tools (i.e. it runs the clippy in the stage 2 toolchain, not the clippy built by the stage 2 toolchain).

Fixes #90244.
r? @Mark-Simulacrum

Mark-Simulacrum · 2022-05-15T14:36:57Z

src/bootstrap/compile.rs

-        });
+        // NOTE: `builder.compiler` has the side effect of running `Assemble`
+        let target_compiler = Compiler { stage: run.builder.top_stage + 1, host: run.target };
+        if run.builder.top_stage >= 2 {


I would have expected full_bootstrap to play a role here, I think.

It also feels a little iffy that this is only changing make_run -- builder.compiler(...) invokes Assemble via ensure, which won't run this code, right?

Correct, that's what I meant in the description by "if you don't want to change the stage numbering, then I'll need to fix the sysroot issue eventually anyway and it doesn't make sense to merge this PR". The clippy PR uses builder.compiler and so won't be fixed by this change.

I am not going to put a ton more time into this PR until the clippy PR is unblocked, but I'll investigate full_bootstrap at that time, I hadn't thought about it.

Someone commented on #90244 this was causing issues, so I've updated the PR. It now takes into account full-bootstrap; I've tested that and it works fine. I also tested that build --stage 2 compiler/rustc now works:

$ x build --stage 2 compiler/rustc Building rustbuild Compiling bootstrap v0.0.0 (/home/jnelson/rust-lang/rust2/src/bootstrap) Finished dev [unoptimized] target(s) in 12.80s Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.19s Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu) Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.32s Copying stage0 rustc from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu) Assembling stage1 compiler (x86_64-unknown-linux-gnu) Building stage1 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.20s Copying stage1 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu) Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu) Copying stage3 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu) Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu) Finished release [optimized] target(s) in 0.31s Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu) Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu) Copying stage3 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu) Build completed successfully in 0:00:14

src/bootstrap/compile.rs

jyn514 · 2022-05-22T04:11:32Z

@rustbot label -S-waiting-on-author +S-blocked

rust-lang#89759 introduced a panic: ``` Assembling stage3 compiler (x86_64-apple-darwin) thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/Users/user/rust2/build/x86_64-apple-darwin/stage2-rustc/x86_64-apple-darwin/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24 ``` This wasn't actually a bug in that PR - the problem was that `x build --stage 3` is broken, and has been for quite some time, even ignoring the stamp file error: ``` thread 'main' panicked at 'src.symlink_metadata() failed with No such file or directory (os error 2) ("failed to determine metadata of /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage2-rustc/x86_64-unknown-linux-gnu/release/rustc-main")', src/bootstrap/lib.rs:1414:24 ``` It needs to take into account whether the artifacts from stage1 are being reused, rather than blindly assuming rustc will be recompiled. Doing so is kind of annoying, because it requires knowing the target the compiler is being built for. Instead, just revert to the old behavior of `build --stage 2 compiler/rustc`, which avoids trying to create the sysroot in the first place.

rust-log-analyzer · 2022-07-11T05:40:54Z

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: rust.debug-assertions := True
configure: rust.overflow-checks := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---

######################################################################## 100.0%
extracting /checkout/obj/build/cache/2022-06-29/rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz to /checkout/obj/build/x86_64-unknown-linux-gnu/stage0
skip untracked path cpu-usage.csv during rustfmt invocations
Diff in /checkout/src/bootstrap/compile.rs at line 1118:
         // panic (see #90224). In practice, that's probably ok: nothing currently uses the stage 2
         // sysroot.
         if run.builder.top_stage >= 2 && !run.builder.config.full_bootstrap {
-            let build_compiler = run.builder.compiler(run.builder.top_stage, run.builder.config.build);
+            let build_compiler =
+                run.builder.compiler(run.builder.top_stage, run.builder.config.build);
             run.builder.ensure(Rustc {
                 target: run.target,
                 target: run.target,
Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustfmt" "--config-path" "/checkout" "--edition" "2021" "--unstable-features" "--skip-children" "--check" "/checkout/src/bootstrap/tarball.rs" "/checkout/src/bootstrap/cc_detect.rs" "/checkout/src/bootstrap/native.rs" "/checkout/src/bootstrap/tool.rs" "/checkout/src/bootstrap/dist.rs" "/checkout/src/bootstrap/compile.rs" "/checkout/src/bootstrap/bin/rustdoc.rs" "/checkout/src/bootstrap/toolstate.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
thread '<unnamed>' panicked at 'tx.send(entry.into_path()) failed with sending on a closed channel', format.rs:166:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'tx.send(entry.into_path()) failed with sending on a closed channel', format.rs:166:17
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }', /cargo/registry/src/github.com-1ecc6299db9ec823/ignore-0.4.18/src/walk.rs:1302:31

Mark-Simulacrum · 2022-07-11T12:11:01Z

src/bootstrap/compile.rs

+        // NOTE: Anywhere in bootstrap that calls `builder.compiler` has the side effect of running
+        // `Assemble`. In those cases, this workaround in `make_run` doesn't help, we'll still hit a
+        // panic (see #90224). In practice, that's probably ok: nothing currently uses the stage 2
+        // sysroot.


I'm a little confused why we put this in make_run, rather than in the main body of the Assemble step. That would seem to prevent this FIXME from being necessary.

JohnCSimon · 2023-01-29T05:54:17Z

@jyn514
ping from triage - can you post your status on this PR? This has sat untouched for a while. Thanks!

FYI: when a PR is ready for review, send a message containing
@rustbot ready to switch to S-waiting-on-review so the PR is in the reviewer's backlog.

jyn514 · 2023-01-29T09:21:14Z

ping from triage - can you post your status on this PR? This has sat untouched for a while. Thanks!

This change is important, and complicated enough I doubt someone else will pick it up, but I also don't really have the energy to work on it for the foreseeable future.

jyn514 · 2023-03-18T20:35:43Z

closing in favor of #108288

rust-highfive assigned Mark-Simulacrum May 9, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 9, 2022

This comment has been minimized.

Sign in to view

Mark-Simulacrum reviewed May 15, 2022

View reviewed changes

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 15, 2022

rustbot added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 22, 2022

apiraino added the T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. label May 23, 2022

jyn514 added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Jul 11, 2022

jyn514 force-pushed the fix-stage3-panic branch from 90759a7 to 1f2c234 Compare July 11, 2022 03:39

jyn514 added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Jul 11, 2022

This comment has been minimized.

Sign in to view

jyn514 force-pushed the fix-stage3-panic branch 2 times, most recently from abd46ff to 32a6c02 Compare July 11, 2022 04:24

This comment has been minimized.

Sign in to view

jyn514 force-pushed the fix-stage3-panic branch from 32a6c02 to f707e72 Compare July 11, 2022 05:28

Mark-Simulacrum reviewed Jul 11, 2022

View reviewed changes

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 11, 2022

jyn514 closed this Mar 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix spurious error when running `build --stage 2 compiler/rustc` #96859

Fix spurious error when running `build --stage 2 compiler/rustc` #96859

jyn514 commented May 9, 2022 •

edited

Loading

This comment has been minimized.

Mark-Simulacrum May 15, 2022

jyn514 May 22, 2022

jyn514 Jul 11, 2022 •

edited

Loading

jyn514 commented May 22, 2022

This comment has been minimized.

This comment has been minimized.

rust-log-analyzer commented Jul 11, 2022

Mark-Simulacrum Jul 11, 2022

JohnCSimon commented Jan 29, 2023

jyn514 commented Jan 29, 2023

jyn514 commented Mar 18, 2023

Fix spurious error when running build --stage 2 compiler/rustc #96859

Fix spurious error when running build --stage 2 compiler/rustc #96859

Conversation

jyn514 commented May 9, 2022 • edited Loading

This comment has been minimized.

Mark-Simulacrum May 15, 2022

Choose a reason for hiding this comment

jyn514 May 22, 2022

Choose a reason for hiding this comment

jyn514 Jul 11, 2022 • edited Loading

Choose a reason for hiding this comment

jyn514 commented May 22, 2022

This comment has been minimized.

This comment has been minimized.

rust-log-analyzer commented Jul 11, 2022

Mark-Simulacrum Jul 11, 2022

Choose a reason for hiding this comment

JohnCSimon commented Jan 29, 2023

jyn514 commented Jan 29, 2023

jyn514 commented Mar 18, 2023

Fix spurious error when running `build --stage 2 compiler/rustc` #96859

Fix spurious error when running `build --stage 2 compiler/rustc` #96859

jyn514 commented May 9, 2022 •

edited

Loading

jyn514 Jul 11, 2022 •

edited

Loading