diff --git a/src/building/bootstrapping.md b/src/building/bootstrapping.md index 543c68c2e4..6ea2822c84 100644 --- a/src/building/bootstrapping.md +++ b/src/building/bootstrapping.md @@ -18,8 +18,23 @@ rustc, then uses it to compile the new compiler. ## Stages of bootstrapping -Compiling `rustc` is done in stages. Here's a diagram, adapted from Joshua Nelson's -[talk on bootstrapping][rustconf22-talk] at RustConf 2022, with detailed explanations below. +The build system defers most of the complicated logic managing invocations +of rustc and rustdoc to Cargo itself. However, moving through various stages +and copying artifacts is still necessary for it to do. Each time rustbuild +is invoked, it will iterate through the list of predefined steps and execute +each serially in turn if it matches the paths passed or is a default rule. +For each step rustbuild relies on the step internally being incremental and +parallel. Note, though, that the `-j` parameter to rustbuild gets forwarded +to appropriate test harnesses and such. + +Most of the "meaty" steps that matter are backed by Cargo, which does indeed +have its own parallelism and incremental management. Later steps, like +tests, aren't incremental and simply run the entire suite currently. +However, compiletest itself tries to avoid running tests when the artifacts +that are involved (mainly the compiler) haven't changed. + +Here's a diagram, adapted from Joshua Nelson's [talk on bootstrapping][rustconf22-talk] +at RustConf 2022, with detailed explanations below. The `A`, `B`, `C`, and `D` show the ordering of the stages of bootstrapping. Blue nodes are downloaded, @@ -448,3 +463,66 @@ This is an incomplete reference for the outputs generated by bootstrap: | copy `rustdoc` | `build/HOST/stage2/bin` | `--stage=2` stops here. + +### Clarification of build command's stdout + +In this part, we will investigate the build command's stdout in an action +(similar, but more detailed and complete documentation compare to topic above). +When you execute `x.py build` command, the build output will be something +like the following: + +```text +Building stage0 std artifacts +Copying stage0 std +Building stage0 test artifacts +Copying stage0 test +Building stage0 compiler artifacts +Copying stage0 rustc +Assembling stage1 compiler +Building stage1 std artifacts +Copying stage1 std +Building stage1 test artifacts +Copying stage1 test +Building stage1 compiler artifacts +Copying stage1 rustc +Assembling stage2 compiler +Uplifting stage1 std +Uplifting stage1 test +Uplifting stage1 rustc +``` + +#### Building stage0 {std,test,compiler} artifacts + +These steps use the provided (downloaded, usually) compiler to compile the +local Rust source into libraries we can use. + +#### Copying stage0 {std,test,rustc} + +This copies the build output from Cargo into +`build/$HOST/stage0-sysroot/lib/rustlib/$ARCH/lib`. +[comment]: FIXME: this step's documentation should be expanded -- the information already here may be incorrect. + +#### Assembling stage1 compiler + +This copies the libraries we built in "building stage0 ... artifacts" into +the stage1 compiler's lib directory. These are the host libraries that the +compiler itself uses to run. These aren't actually used by artifacts the new +compiler generates. This step also copies the rustc and rustdoc binaries we +generated into `build/$HOST/stage/bin`. + +The stage1/bin/rustc is a fully functional compiler, but it doesn't yet have +any libraries to link built binaries or libraries to. The next 3 steps will +provide those libraries for it; they are mostly equivalent to constructing +the stage1/bin compiler so we don't go through them individually. + +#### Uplifting stage1 {std,test,rustc} + +This step copies the libraries from the stage1 compiler sysroot into the +stage2 compiler. This is done to avoid rebuilding the compiler; libraries +we'd build in this step should be identical (in function, if not necessarily +identical on disk) so there's no need to recompile the compiler again. Note +that if you want to, you can enable the full-bootstrap option to change this +behavior. + +Each step is driven by a separate Cargo project and rustbuild orchestrates +copying files between steps and otherwise preparing for Cargo to run. diff --git a/src/building/how-to-build-and-run.md b/src/building/how-to-build-and-run.md index 24786c0c0e..d9d5425de4 100644 --- a/src/building/how-to-build-and-run.md +++ b/src/building/how-to-build-and-run.md @@ -54,12 +54,43 @@ clean` will not cause a rebuild of LLVM. ## Building the Compiler +Building compiler is unfortunately not quite as simple as building other +rust projects using`cargo build`. Instead, we use `x.py` script for building +the compiler. The main reason behind it is the bootstrap stages where you use +the output of one cargo invocation as the input compiler for another. Which +is not possible with `cargo` itself. + Note that building will require a relatively large amount of storage space. You may want to have upwards of 10 or 15 gigabytes available to build the compiler. -Once you've created a `config.toml`, you are now ready to run -`x.py`. There are a lot of options here, but let's start with what is -probably the best "go to" command for building a local compiler: +The script accepts commands, flags, and arguments to determine what to do. +To see capabilities of x in an action, check the following examples: + + ```bash + # build the whole compiler + ./x.py build --stage 2 + + # build the stage1 compiler + ./x.py build + + # build stage0 libstd + ./x.py build --stage 0 library/std + + # build a particular crate in stage0 + ./x.py build --stage 0 library/test + ``` + + If files that would normally be rebuilt from stage 0 are dirty, the rebuild can be + overridden using `--keep-stage 0`. Using `--keep-stage n` will skip all steps + that belong to stage n or earlier: + + ```bash + # build stage 1, keeping old build products for stage 0 + ./x.py build --keep-stage 0 + ``` + +Let's dive into details with probably the best "go to" +command for building a local compiler: ```bash ./x.py build library @@ -98,6 +129,24 @@ build. The **full** `rustc` build (what you get with `./x.py build You almost never need to do this. +## Incremental builds + +You can configure rustbuild to use incremental compilation with the +`--incremental` flag: + +```sh +$ ./x.py build --incremental +``` + +The `--incremental` flag will store incremental compilation artifacts +in `build//stage0-incremental`. Note that we only use incremental +compilation for the stage0 -> stage1 compilation -- this is because +the stage1 compiler is changing, and we don't try to cache and reuse +incremental artifacts across different versions of the compiler. + +You can always drop the `--incremental` to build as normal (note that +you will still be using the downloaded beta as your bootstrap). + ## Build specific components If you are working on the standard library, you probably don't need to build @@ -242,3 +291,62 @@ everything up then you only need to run one command! `rm -rf build` works too, but then you have to rebuild LLVM, which can take a long time even on fast computers. + +## Build tools + +We've actually got quite a few tools that we use in the compiler's build system +and for testing. To organize these, each tool is a project in `src/tools` with a +corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having +independent `Cargo.lock` files) and do not currently explicitly depend on the +compiler or standard library. Compiling each tool is sequenced after the +appropriate libstd/libtest/librustc compile above. + +## Extending rustbuild + +So, you'd like to add a feature to the rustbuild build system or just fix a bug. +Great! One of the major motivational factors for moving away from `make` is that +Rust is in theory much easier to read, modify, and write. If you find anything +excessively confusing, please open an issue on this, and we'll try to get it +documented or simplified, pronto. + +First up, you'll probably want to read over the documentation above, as that'll +give you a high level overview of what rustbuild is doing. You also probably +want to play around a bit yourself by just getting it up and running before you +dive too much into the actual build system itself. + +After that, each module in rustbuild should have enough documentation to keep +you up and running. Some general areas that you may be interested in modifying +are: + +* Adding a new build tool? Take a look at `bootstrap/tool.rs` for examples of + other tools. +* Adding a new compiler crate? Look no further! Adding crates can be done by + adding a new directory with `Cargo.toml` followed by configuring all + `Cargo.toml` files accordingly. +* Adding a new dependency from crates.io? This should just work inside the + compiler artifacts stage (everything other than libtest and libstd). +* Adding a new configuration option? You'll want to modify `bootstrap/flags.rs` + for command line flags and then `bootstrap/config.rs` to copy the flags to the + `Config` struct. +* Adding a sanity check? Take a look at `bootstrap/sanity.rs`. + +If you make a major change, please remember to: + ++ Update `VERSION` in `src/bootstrap/main.rs`. +* Update `changelog-seen = N` in `config.toml.example`. +* Add an entry in `src/bootstrap/CHANGELOG.md`. + +A 'major change' includes + +* A new option or +* A change in the default options. + +Changes that do not affect contributors to the compiler or users +building rustc from source don't need an update to `VERSION`. + +If you have any questions, feel free to reach out on the `#t-infra` channel in +the [Rust Zulip server][rust-zulip] or ask on internals.rust-lang.org. When +you encounter bugs, please file issues on [Rust issue tracker][rust-issue-tracker]. + +[rust-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra +[rust-issue-tracker]: https://github.com/rust-lang/rust/issues diff --git a/src/getting-started.md b/src/getting-started.md index 4e1f520ffa..b9e96844e4 100644 --- a/src/getting-started.md +++ b/src/getting-started.md @@ -114,6 +114,10 @@ serious development work. In particular, `./x.py build` and `./x.py test` provide many ways to compile or test a subset of the code, which can save a lot of time. +Also, note that `x.py` supports all kinds of path suffixes for `compiler`, `library`, +and `src/tools` directories. So, you can simply run `x.py test tidy` instead of +`x.py test src/tools/tidy`. Or, `x.py build std` instead of `x.py build library/std`. + [rust-analyzer]: ./building/suggested.html#configuring-rust-analyzer-for-rustc See the chapters on [building](./building/how-to-build-and-run.md),