redesign stage 0 std

[onur-ozkan]

Summary

This PR changes how bootstrap builds the stage 1 compiler by switching to precompiled stage 0 standard library instead of building the in-tree one. The goal was to update bootstrap to use the beta standard library at stage 0 rather than compiling it from source (see the motivation at rust-lang/compiler-team#619).

Previously, to build a stage 1 compiler bootstrap followed this path:

download stage0 compiler -> build in-tree std -> compile stage1 compiler with in-tree std

With this PR, the new path is:

download stage0 compiler -> compile stage1 compiler with precompiled stage0 std

This also means that cfg(bootstrap)/cfg(not(bootstrap)) is no longer needed for library development.

Building "library"

Since stage0 std is no longer in-tree x build/test/check library --stage 0 is now no-op. The minimum supported stage to build std is now 1. For the same reason, default stage values in the library profile is no longer 0.

Because building the in-tree library now requires a stage1 compiler, I highly recommend library developers to enable download-rustc to speed up compilation time.

---

If you encounter a bug or unexpected results please open a topic in the #t-infra/bootstrap Zulip channel or create a bootstrap issue.

Blocked on #122709

[rustbot]

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

[onur-ozkan]

@rustbot ready

r? bootstrap
cc @rust-lang/libs

[onur-ozkan]

@rustbot author (currently stage 2 std is copied from stage 1 and this behaviour should change with the beta std change)

[onur-ozkan]

Rebased.

[Kobzol]

I think that this will also need updating.

Done, I replaced --keep-stage suggestion with download-rustc.

Sorry, to clarify, it shouldn't be removed altogether. The modifications that you made are good, but we should still keep the information about --keep-stage. When I want to repeatedly rebuild the compiler locally without rebuilding stdlib, ./x build --keep-stage 1 is still incredibly useful, and neither this redesign nor download-ci-rustc can replace it 😅

I played with this PR locally a bit and it seems to work how I expected it to. It's great not having to rebuild the library as a first thing in the build! It should also make rebases much less costly, as modifications to stdlib will no longer invalidate rustc (but this is harder to test without the PR being merged).

[RalfJung]

When I want to repeatedly rebuild the compiler locally without rebuilding stdlib, ./x build --keep-stage 1 is still incredibly useful, and neither this redesign nor download-ci-rustc can replace it 

Isn't that what keep-stage-std is for?

[Kobzol]

I use --keep-stage for that, and that is also what rustc-dev-guide suggests 😅 But in any case, the suggested workflows page should still mention this super useful use-case, regardless of which flag is used.

[Kobzol]

This is a huge change, which IMO requires a much more thorough explanation to rustc contributors. We should provide a link to some resource that will clearly describe what has changed and what implications it has. The current description of this PR contains almost no information, and links to an old MCP that describes changes that were already made, but also some things that weren't made yet, and mentions things like --target-sysroot that don't exist in bootstrap currently.

I think that expanding the description of this PR to include a more comprehensive explanation of what changes would be enough.

[jieyouxu]

FWIW, I think we should have a dedicated Inside Rust blog post to announce this and explain the changes to not just rustc contributors but also libs contributors (I'm happy to help write one, just haven't quite caught up to the changes here yet).

[Kobzol]

Yeah, or that :)

[onur-ozkan]

I can update the PR description with more details.

[Kobzol]

Is this still needed after this PR?

[onur-ozkan]

Is this still needed after this PR?

Yes, it's needed even more.

[onur-ozkan]

Sorry, to clarify, it shouldn't be removed altogether. The modifications that you made are good, but we should still keep the information about --keep-stage. When I want to repeatedly rebuild the compiler locally without rebuilding stdlib, ./x build --keep-stage 1 is still incredibly useful, and neither this redesign nor download-ci-rustc can replace it 😅

I am not really sure if that's even possible. When --keep-stage is used compiler will be uplifted from that stage to avoid compilation. But I can re-document it with some useful scenarios for sure.

[RalfJung]

Is this still needed after this PR?

Yes, it's needed even more.

After this PR library should always be in the exclusion list, not just on CI, right? So the if !CiEnv::is_ci() { should be removed.

[bjorn3]

Before this PR library would be on the exclusion list only for local builds, where it should stay. CI should not exclude anything that could affect the contents of rust-dev I think (which includes library).

[RalfJung]

I don't see why CI would use different settings than local builds. The library folder cannot impact the stage 0 compiler build since that compiler now uses the bootstrap std.

[bjorn3]

Doesn't download-rustc use the downloaded rustc from rust-dev as stage 1 compiler and thus use the associated std for stage 1 builds whereas disabling download-rustc would use the local library folder for stage 1 builds?

[Kobzol]

If you only modify the library, download-rustc should still download a precompiled compiler, to make library modifications faster, that use-case should be supported. The difference is that locally that will build a stage 2 stdlib, while on CI I suppose that we still want to build a stage 1 stdlib (although for most test builders I suspect that the difference doesn't really matter).

[RalfJung]

locally that will build a stage 2 stdlib,

Uh, what? It skips a stage? That's very odd. Why?
I am also extremely suspicious of having any difference between CI and local builds. That's setting us up for nightmare debugging sessions.

[Kobzol]

(That's pre-existing behavior, so I'm not sure if this PR is relevant to discuss that, tbh). I suppose that it is stage 2 because you download the whole stage1 compiler from CI (including its stdlib), so since you already have stage1 downloaded, you then built stage 2 locally. But Onur can most likely explain it better.

[RalfJung]

With this PR, the stage 2 library should be built by the stage 2 compiler, just like stage 1 works. So I think it is relevant for this PR, as this PR exactly changes how the staging is chained:

• before: a stage starts with a library build, and then a compiler build that uses that library
• after: a stage starts with a compiler build, using the previous stage's library; and then the library is built by the just-built compiler

So the dependency chain now is:
stage 0 compiler -> stage 0 library -> stage 1 compiler -> stage 1 library

Even if you only modify the library, the stage 1 compiler will be different, and therefore everything past this point as well. The stage 2 library is quite far down the chain and nothing there can be cached.

But it also shouldn't be needed as the stage 0 compiler + library should be enough for almost everything.

[Kobzol]

Ah, I meant the CI vs local difference, not the staging itself. Yeah, you're right that the staging with download-rustc should be diferent here, it seems a bit suspicious.

If I unconditionally enable download-ci-rustc and modify library, bootstrap prints this:

Creating a sysroot for stage2 compiler (use `rustup toolchain link 'name' build/host/stage2`)
Building stage2 library artifacts {alloc, core, literal-escaper, panic_abort, panic_unwind, proc_macro, std, sysroot, test, unwind} (x86_64-unknown-linux-gnu)

While it should instead build stage1 stdlib.

But it also shouldn't be needed as the stage 0 compiler + library should be enough for almost everything.

What shouldn't be needed? download-ci-rustc? You still need that to enable fast work on the library, right?

[RalfJung]

What shouldn't be needed?

A stage 2 library build.

[onur-ozkan]

Is this still needed after this PR?

Yes, it's needed even more.

After this PR library should always be in the exclusion list, not just on CI, right? So the if !CiEnv::is_ci() { should be removed.

I misunderstood #119899 (comment)... I thought Jakub was asking whether we should still keep the library in the exclusion list.

We can definitely keep the library path unconditionally in the exclusion list and remove the CI check! I will do that in couple of minutes.