x.py panics when trying to build "stage3" compiler

[camelid]

I don't think there even is a stage3, so I think this is a bug in bootstrap. I haven't encountered this before. Rerunning the command produces the same panic. Despite the panic, rustc +r2stage2 --help works, presumably because the build has finished and x.py is trying to do extra steps by building a non-existent stage.

In case it helps, my branch's merge-base is 00d5e42. My branch only modifies the Miri engine, so I'm pretty sure it's unrelated to this bug.

$ x build --stage 2 compiler/rustc
[...]
Copying stage1 rustc from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Assembling stage2 compiler (x86_64-apple-darwin)
Uplifting stage1 std (x86_64-apple-darwin -> x86_64-apple-darwin)
Copying stage2 std from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Uplifting stage1 rustc (x86_64-apple-darwin -> x86_64-apple-darwin)
Copying stage2 rustc from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Assembling stage3 compiler (x86_64-apple-darwin)
thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/Users/user/rust2/build/x86_64-apple-darwin/stage2-rustc/x86_64-apple-darwin/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:29:07

Backtrace

thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/Users/user/rust2/build/x86_64-apple-darwin/stage2-rustc/x86_64-apple-darwin/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24
stack backtrace:
   0: rust_begin_unwind
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/std/src/panicking.rs:517:5
   1: core::panicking::panic_fmt
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/core/src/panicking.rs:101:14
   2: bootstrap::Build::read_stamp_file
             at ./src/bootstrap/lib.rs:1296:24
   3: <bootstrap::compile::Assemble as bootstrap::builder::Step>::run
             at ./src/bootstrap/compile.rs:1107:27
   4: bootstrap::builder::Builder::ensure
             at ./src/bootstrap/builder.rs:1592:23
   5: bootstrap::builder::Builder::compiler
             at ./src/bootstrap/builder.rs:627:9
   6: <bootstrap::compile::Assemble as bootstrap::builder::Step>::make_run
             at ./src/bootstrap/compile.rs:1034:30
   7: bootstrap::builder::StepDescription::maybe_run
             at ./src/bootstrap/builder.rs:175:13
   8: bootstrap::builder::StepDescription::run
             at ./src/bootstrap/builder.rs:231:21
   9: bootstrap::builder::Builder::run_step_descriptions
             at ./src/bootstrap/builder.rs:619:9
  10: bootstrap::builder::Builder::execute_cli
             at ./src/bootstrap/builder.rs:599:9
  11: bootstrap::Build::build
             at ./src/bootstrap/lib.rs:624:13
  12: bootstrap::main
             at ./src/bootstrap/bin/main.rs:33:5
  13: core::ops::function::FnOnce::call_once
             at /rustc/e784c962ea252f0874a4305168077e7048cb39e9/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Build completed unsuccessfully in 0:00:05

[Mark-Simulacrum]

cc @jyn514 -- I think this is caused by the move of compiler assembly into the rustc step (or similar).

I think the fix here is likely to make sure the code is reading from stage1-rustc's stamp file if it's reusing those libraries. We may also prefer to just skip the copy entirely -- stage3/bin/rustc is not generally useful -- but that should get gated on full-bootstrap if it still exists (I forget).

[jyn514]

stage3/bin/rustc is not generally useful -- but that should get gated on full-bootstrap if it still exists

My understanding is that stage3 is still useful for ensuring we have reproducible builds. Yes, full-bootstrap still exists (although I'm not sure what the use case is).

[RalfJung]

Independent of whether stage 3 exists or not, the command here explicitly asks for stage 2 so something seems to have gone wrong when stage 3 steps are happening.

This problem still persists in current master, I just ran into it with ./x.py build compiler/rustc --stage 2.

[jyn514]

@RalfJung for context this is likely caused by #89759 - I'm ok with reverting that for now if bootstrap has other bugs, but I don't think assembling the compiler before you've asked for it is inherently a bug.

[Mark-Simulacrum]

Since you've already built the stage 3 compiler (in all but copying it into place), it makes some sense to assemble it. At least, this more closely matches the likely-desired behavior at earlier stages, IMO.

I'd be fine with us just adding a stage == 3 and not full-bootstrap bailout to this code for now, which should mitigate this error, or a more complete patch -- I think reverting #89759 is probably not the right patch, though.

[RalfJung]

Since you've already built the stage 3 compiler (in all but copying it into place), it makes some sense to assemble it.

Have I? All I want is a stage 2 that I can use to build Miri. build --stage 2 compiler/rustc seemed like the easiest way to do that. I guess this will also build the 'stage2 compiler artifacts' (aka the stage 3 compiler), but that is just a copy so I don't mind this extra work.

From what you said it sounds like ./x.py build library/test --stage 2 should suffice for my usecase?

[RalfJung]

From what you said it sounds like ./x.py build library/test --stage 2 should suffice for my usecase?

Hm, no, it does not since Miri needs the rustc libs themselves. build --stage 2 compiler/rustc does the extra work of putting those into the stage 2 sysroot, it seems.

[Mark-Simulacrum]

Without full-bootstrap, stages >= 2 don't actually have distinct libraries (we uplift the stage 1 compiler artifacts). So in some sense any stage is "built" and copy-able into the relevant directories once you have a stage2/bin/rustc.

But, yes, you do need to copy the rustc libraries into the appropriate sysroot. I suspect that your particular use case may be painful/unworkable with the stage 3 assembly step being broken (as noted in this issue), but am not sure.

[RalfJung]

Well, running build --stage 2 compiler/rustc and then just ignoring the fact that it panics seems to be good enough to build Miri. But yeah something is wrong, as the panic shows.^^

[smijolovic]

What is the proper way to build the stage2 compiler now?
With 1.57.0,
x.py build --stage 2 compiler/rustc

Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Assembling stage3 compiler (x86_64-unknown-linux-gnu)
thread 'main' panicked at 'fs::read(stamp) failed with No such file or directory (os error 2) ("/home/chaasmadmin/nimbus8/rust/rust-chaasm-1.57.0/rust/build/x86_64-unknown-linux-gnu/stage2-rustc/x86_64-unknown-linux-gnu/release/.librustc.stamp")', src/bootstrap/lib.rs:1296:24
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Build completed unsuccessfully in 5:08:19

[RalfJung]

x.py build --stage 2 compiler/rustc works for me, you just have to ignore the error.

[camelid]

Yeah, the error is just annoying; it doesn't actually break anything.

[jyn514]

I think I found the issue. The stamp file is written in run_cargo, but for stage 2 we just copy the files rather than rebuilding a third time.

rust/src/bootstrap/compile.rs

Line 1377 in f6cef57

t!(fs::write(&stamp, &new_contents));

[jyn514]

This is kind of tricky to fix, because everywhere in bootstrap that touches self.rustc or self.cargo_out suddenly needs to know whether those artifacts actually exist or whether we're reusing the ones from stage 1. I think a simpler fix is just to avoid assembling the sysroot for stage 3 and up.

[jyn514]

I think a simpler fix is just to avoid assembling the sysroot for stage 3 and up.

That won't actually help with #96798 though :( but as I posted on that PR, I think we should never be trying to use the stage 3 sysroot anyway.

[smijolovic]

While ignoring the issue might work for manual builds, it certainly is not trivial to ignore this error for automated build systems or package build systems (debbuild).

Might be worth providing a doc on this and adding a workaround, as most would not be building manually for distros or from source CI/CD automation.

config.toml:
extended = true
tools = ["cargo"]

./x.py build --stage 2 -j12

works.

librustc_macros-*.so is no longer present in the lib directory. Was that removed/depreciated from the compiler build?

[RalfJung]

Yeah, the error is just annoying; it doesn't actually break anything.

FWIW that does not seem to be always true; when using multi-step x.py invocations such as x.py build src/tools/rustdoc compiler/rustc --stage 2 this error causes later steps to be skipped, which becomes an actual issue.

[onur-ozkan]

@rustbot claim

[jaskij]

Independent of whether stage 3 exists or not, the command here explicitly asks for stage 2 so something seems to have gone wrong when stage 3 steps are happening.

@RalfJung (and others who come across this issue), there's this note in bootstraping docs about the unintuitive behavior of this flag:

Note that build --stage N compiler/rustc does not build the stage N compiler: instead it builds the stage N+1 compiler using the stage N compiler.

Perhaps it would be good to replicate it in chapter 1. How to build and run the compiler.

[jyn514]

please no, that chapter is already long enough and bootstrapping is too complicated to try to summarize it in a single sentence

[RalfJung]

Also this bug is not caused by a user misunderstanding the stage numbering, it is caused by x.py not treating that stage correctly.

[bjorn3]

I can't reproduce this anymore even with #108288 reverted. Would it be ok to revert that PR?

[onur-ozkan]

I can't reproduce this anymore even with #108288 reverted. Would it be ok to revert that PR?

If revert isn't blocking any other stuff, I would like to spend some time to understand the underlying reason.

[bjorn3]

It isn't. I just noticed this issue being referenced while reading the bootstrap code and figured I did try if I could remove the special case.

[onur-ozkan]

I can't reproduce this anymore even with #108288 reverted. Would it be ok to revert that PR?

Problem still exists actually.

change-id = 102579
profile = "compiler"

[llvm]
download-ci-llvm = true

[build]
docs = false
docs-minification = false
compiler-docs = false

[rust]
download-rustc = false
deny-warnings = false

x build --stage 2 compiler/rustc panics.