Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add x.py check --stage 1 #80952

Closed
wants to merge 2 commits into from
Closed

Conversation

jyn514
Copy link
Member

@jyn514 jyn514 commented Jan 12, 2021

This is useful for several cases:

It is also very easy to support.

Closes #46955.

This is useful for several cases:

- Checking changes to the standard library with `cfg(not(bootstrap))`
- Checking platform-specific changes to the compiler. In particular,
  checking with stage 1 doesn't require having a cross-compiling C native
  toolchain, unlike building.
- Testing newly added internal lints without having to rebuild
  rustc_middle twice

It is also very easy to support.
@jyn514 jyn514 added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself labels Jan 12, 2021
@rust-highfive
Copy link
Collaborator

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 12, 2021
@jyn514
Copy link
Member Author

jyn514 commented Jan 12, 2021

cc @Diggsey - is this what you expected for #46955 ?

@Diggsey
Copy link
Contributor

Diggsey commented Jan 12, 2021

Heh, that's an old issue 😄 - yeah it looks like this solves it.

@jyn514
Copy link
Member Author

jyn514 commented Jan 12, 2021

Yeah, I was triaging bootstrap issues a while ago and it popped up as something I'd want myself too 😆

@jyn514
Copy link
Member Author

jyn514 commented Jan 12, 2021

I don't know why I thought this would be easy 🤦

error: internal compiler error: compiler/rustc_mir/src/monomorphize/collector.rs:826:9: cannot create local mono-item for DefId(1:1638 ~ std[338b]::env::var)

query stack during panic:
#0 [collect_and_partition_mono_items] collect_and_partition_mono_items
end of query stack

Otherwise, we may not have MIR for the standard library available:

```
error: internal compiler error: compiler/rustc_mir/src/monomorphize/collector.rs:826:9: cannot create local mono-item for DefId(2:6820 ~ core[92cf]::fmt::{impl#2}::new_v1)
```

The current error after this change is

```
error[E0514]: found crate `chalk_derive` compiled by an incompatible version of rustc
  --> /home/joshua/.local/lib/cargo/registry/src/github.com-1ecc6299db9ec823/chalk-ir-0.36.0/src/lib.rs:13:5
   |
13 | use chalk_derive::{Fold, HasInterner, SuperVisit, Visit, Zip};
   |     ^^^^^^^^^^^^
   |
   = help: please recompile that crate using this compiler (rustc 1.51.0-dev)
   = note: the following crate versions were found:
           crate `chalk_derive` compiled by rustc 1.49.0-beta.1 (21dea46 2020-11-18): /home/joshua/rustc3/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps/libchalk_derive-843b3d0b9b865ca6.so
```
@rust-log-analyzer
Copy link
Collaborator

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
configure: rust.channel         := nightly
configure: rust.debug-assertions := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
skip untracked path cpu-usage.csv during rustfmt invocations
skip untracked path src/doc/book/ during rustfmt invocations
skip untracked path src/doc/rust-by-example/ during rustfmt invocations
skip untracked path src/llvm-project/ during rustfmt invocations
Diff in /checkout/src/bootstrap/check.rs at line 84:
Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustfmt" "--config-path" "/checkout" "--edition" "2018" "--unstable-features" "--skip-children" "--check" "/checkout/src/bootstrap/check.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
         );
         std_cargo(builder, target, compiler.stage, &mut cargo);
 
-        builder.info(&format!("Checking stage{} std artifacts ({} -> {})", builder.top_stage, &compiler.host, target));
+        builder.info(&format!(
+            "Checking stage{} std artifacts ({} -> {})",
+            builder.top_stage, &compiler.host, target
+        ));
         run_cargo(
             cargo,
Diff in /checkout/src/bootstrap/check.rs at line 187:
Diff in /checkout/src/bootstrap/check.rs at line 187:
             cargo.arg("-p").arg(krate.name);
 
 
-        builder.info(&format!("Checking stage{} compiler artifacts ({} -> {})", builder.top_stage, &compiler.host, target));
+        builder.info(&format!(
+            "Checking stage{} compiler artifacts ({} -> {})",
+            builder.top_stage, &compiler.host, target
+        ));
         run_cargo(
             cargo,
failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --stage 2 src/tools/tidy
Build completed unsuccessfully in 0:00:18

@jyn514
Copy link
Member Author

jyn514 commented Jan 13, 2021

Reporting on my progress so far, because I won't get it done tonight and I will definitely forget by tomorrow:

  • The ICE comes from
    if !tcx.is_mir_available(def_id) {
    . It happens because there's no MIR built for the standard library, only check metadata (because x.py check didn't build it). In particular, this only happens for build scripts and proc-macros, because they need to actually run.
  • The way stage 0 solves this is with a 'snapshot sysroot':

    rust/src/bootstrap/builder.rs

    Lines 1173 to 1194 in 150d1fe

    // Almost all of the crates that we compile as part of the bootstrap may
    // have a build script, including the standard library. To compile a
    // build script, however, it itself needs a standard library! This
    // introduces a bit of a pickle when we're compiling the standard
    // library itself.
    //
    // To work around this we actually end up using the snapshot compiler
    // (stage0) for compiling build scripts of the standard library itself.
    // The stage0 compiler is guaranteed to have a libstd available for use.
    //
    // For other crates, however, we know that we've already got a standard
    // library up and running, so we can use the normal compiler to compile
    // build scripts in that situation.
    if mode == Mode::Std {
    cargo
    .env("RUSTC_SNAPSHOT", &self.initial_rustc)
    .env("RUSTC_SNAPSHOT_LIBDIR", self.rustc_snapshot_libdir());
    } else {
    cargo
    .env("RUSTC_SNAPSHOT", self.rustc(compiler))
    .env("RUSTC_SNAPSHOT_LIBDIR", self.rustc_libdir(compiler));
    }
    . This works because the metadata generated by x.py check, stage0-std, is separate from the beta libstd, in stage0. This doesn't work for stage 1 because the beta libstd was built by beta, and the build scripts/proc macros are being compiled by stage1. It's not possible to compile the proc macros with stage 0 because the ABI is different and they need to link to the standard library with the stage1 compiler.
  • A possible fix I thought of is to unconditionally build a standard library for the host with stage1. It will only be used for building build scripts/proc macros and nothing else; all other crates should use the library of the target instead.

I tried implementing that just now and it went badly. I'm planning to wait to hear from @Mark-Simulacrum about whether this is a good idea first and whether there's a good way to do this without rewriting half of bootstrap.

camelid added a commit to camelid/rust that referenced this pull request Jan 13, 2021
The ICE message is somewhat confusing and overly specific - the issue is
that there's no MIR available.

This should make debugging these ICEs easier since the error tells you
what's actually wrong, not what it was trying to do when it failed.

cc rust-lang#80952 (comment)
JohnTitor added a commit to JohnTitor/rust that referenced this pull request Jan 14, 2021
Use better ICE message when no MIR is available

The ICE message is somewhat confusing and overly specific - the issue is
that there's no MIR available.

This should make debugging these ICEs easier since the error tells you
what's actually wrong, not what it was trying to do when it failed.

cc rust-lang#80952 (comment)
cc `@jyn514`
JohnTitor added a commit to JohnTitor/rust that referenced this pull request Jan 14, 2021
Use better ICE message when no MIR is available

The ICE message is somewhat confusing and overly specific - the issue is
that there's no MIR available.

This should make debugging these ICEs easier since the error tells you
what's actually wrong, not what it was trying to do when it failed.

cc rust-lang#80952 (comment)
cc ``@jyn514``
JohnTitor added a commit to JohnTitor/rust that referenced this pull request Jan 14, 2021
Use better ICE message when no MIR is available

The ICE message is somewhat confusing and overly specific - the issue is
that there's no MIR available.

This should make debugging these ICEs easier since the error tells you
what's actually wrong, not what it was trying to do when it failed.

cc rust-lang#80952 (comment)
cc ```@jyn514```
m-ou-se added a commit to m-ou-se/rust that referenced this pull request Jan 14, 2021
Use better ICE message when no MIR is available

The ICE message is somewhat confusing and overly specific - the issue is
that there's no MIR available.

This should make debugging these ICEs easier since the error tells you
what's actually wrong, not what it was trying to do when it failed.

cc rust-lang#80952 (comment)
cc ````@jyn514````
m-ou-se added a commit to m-ou-se/rust that referenced this pull request Jan 14, 2021
Use better ICE message when no MIR is available

The ICE message is somewhat confusing and overly specific - the issue is
that there's no MIR available.

This should make debugging these ICEs easier since the error tells you
what's actually wrong, not what it was trying to do when it failed.

cc rust-lang#80952 (comment)
cc `````@jyn514`````
@Mark-Simulacrum
Copy link
Member

This sort of makes sense, but I think I'd prefer folks use --keep-stage-std 0 for this case. They're going to need to build a compiler anyway, and the difference between checking and building std is relatively minimal.

  • Checking platform-specific changes to the compiler. In particular,
    checking with stage 1 doesn't require a cross-compiling C native
    toolchain, unlike building. (Add x.py check --stage 1 #46955)

I admit I don't fully follow this use case. I think there's very little platform-specific code in the cargo check phase of the compiler (indeed, other than differences between 32/64 bit, I can't obviously think of some). Maybe would be good to elaborate on this a bit.

This, on the other hand, is where the value is much more likely to be. I've filed #81064 to replace this PR, and set you as the reviewer. It looks like there's some configuration etc changes here which would be good to propagate, so feel free to push to my branch (or cherry-pick my commit, whatever).

@jyn514
Copy link
Member Author

jyn514 commented Jan 16, 2021

I admit I don't fully follow this use case. I think there's very little platform-specific code in the cargo check phase of the compiler (indeed, other than differences between 32/64 bit, I can't obviously think of some). Maybe would be good to elaborate on this a bit.

I think the point is not that the compiler is platform specific but that the code being compiled is platform specific. @Diggsey probably knows more than I do, but it seems useful to check libstd for various different targets without having to have a cross-compiling C toolchain installed.

@jyn514
Copy link
Member Author

jyn514 commented Jan 16, 2021

Anyway, closing this in favor of #81064, which I think is a better approach.

@jyn514 jyn514 closed this Jan 16, 2021
@Mark-Simulacrum
Copy link
Member

Well, the intersection of cfg not bootstrap and target specific code is probably really low, so probably regular check should work ok for that.

@jyn514 jyn514 deleted the check-stage-1 branch May 1, 2021 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add x.py check --stage 1
5 participants