Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consolidate bootstrap docs #107116

Merged
merged 1 commit into from
Feb 5, 2023
Merged

consolidate bootstrap docs #107116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 38 additions & 161 deletions src/bootstrap/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,105 +4,31 @@ This is an in-progress README which is targeted at helping to explain how Rust
is bootstrapped and in general, some of the technical details of the build
jyn514 marked this conversation as resolved.
Show resolved Hide resolved
system.

## Using rustbuild
Note that this README only covers internal information, not how to use the tool.
Please check [bootstrapping dev guide][bootstrapping-dev-guide] for further information.

The rustbuild build system has a primary entry point, a top level `x.py` script:
[bootstrapping-dev-guide]: https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html

```sh
$ python ./x.py build
```

Note that if you're on Unix, you should be able to execute the script directly:

```sh
$ ./x.py build
```

The script accepts commands, flags, and arguments to determine what to do:

* `build` - a general purpose command for compiling code. Alone, `build` will
bootstrap the entire compiler, and otherwise, arguments passed indicate what to
build. For example:

```
# build the whole compiler
./x.py build --stage 2

# build the stage1 compiler
./x.py build

# build stage0 libstd
./x.py build --stage 0 library/std

# build a particular crate in stage0
./x.py build --stage 0 library/test
```

If files that would normally be rebuilt from stage 0 are dirty, the rebuild can be
overridden using `--keep-stage 0`. Using `--keep-stage n` will skip all steps
that belong to stage n or earlier:

```
# build stage 1, keeping old build products for stage 0
./x.py build --keep-stage 0
```

* `test` - a command for executing unit tests. Like the `build` command, this
will execute the entire test suite by default, and otherwise, it can be used to
select which test suite is run:

```
# run all unit tests
./x.py test

# execute tool tests
./x.py test tidy

# execute the UI test suite
./x.py test tests/ui

# execute only some tests in the UI test suite
./x.py test tests/ui --test-args substring-of-test-name

# execute tests in the standard library in stage0
./x.py test --stage 0 library/std

# execute tests in the core and standard library in stage0,
# without running doc tests (thus avoid depending on building the compiler)
./x.py test --stage 0 --no-doc library/core library/std
## Introduction

# execute all doc tests
./x.py test src/doc
```
The build system defers most of the complicated logic managing invocations
of rustc and rustdoc to Cargo itself. However, moving through various stages
and copying artifacts is still necessary for it to do. Each time rustbuild
is invoked, it will iterate through the list of predefined steps and execute
each serially in turn if it matches the paths passed or is a default rule.
For each step rustbuild relies on the step internally being incremental and
parallel. Note, though, that the `-j` parameter to rustbuild gets forwarded
to appropriate test harnesses and such.

* `doc` - a command for building documentation. Like above, can take arguments
for what to document.

## Configuring rustbuild

rustbuild offers a TOML-based configuration system with a `config.toml`
file. An example of this configuration can be found at `config.toml.example`,
and the configuration file can also be passed as `--config path/to/config.toml`
if the build system is being invoked manually (via the python script).

You can generate a config.toml using `./configure` options if you want to automate creating the file without having to edit it.

Finally, rustbuild makes use of the [cc-rs crate] which has [its own
method][env-vars] of configuring C compilers and C flags via environment
variables.

[cc-rs crate]: https://github.com/alexcrichton/cc-rs
[env-vars]: https://github.com/alexcrichton/cc-rs#external-configuration-via-environment-variables

## Build stages
## Build phases

jyn514 marked this conversation as resolved.
Show resolved Hide resolved
The rustbuild build system goes through a few phases to actually build the
compiler. What actually happens when you invoke rustbuild is:

1. The entry point script, `x.py` is run. This script is
responsible for downloading the stage0 compiler/Cargo binaries, and it then
compiles the build system itself (this folder). Finally, it then invokes the
actual `bootstrap` binary build system.
1. The entry point script(`x` for unix like systems, `x.ps1` for windows systems,
`x.py` cross-platform) is run. This script is responsible for downloading the stage0
compiler/Cargo binaries, and it then compiles the build system itself (this folder).
Finally, it then invokes the actual `bootstrap` binary build system.
2. In Rust, `bootstrap` will slurp up all configuration, perform a number of
sanity checks (whether compilers exist, for example), and then start building the
stage0 artifacts.
Expand All @@ -115,24 +41,6 @@ compiler. What actually happens when you invoke rustbuild is:
The goal of each stage is to (a) leverage Cargo as much as possible and failing
that (b) leverage Rust as much as possible!

## Incremental builds

You can configure rustbuild to use incremental compilation with the
`--incremental` flag:

```sh
$ ./x.py build --incremental
```

The `--incremental` flag will store incremental compilation artifacts
in `build/<host>/stage0-incremental`. Note that we only use incremental
compilation for the stage0 -> stage1 compilation -- this is because
the stage1 compiler is changing, and we don't try to cache and reuse
incremental artifacts across different versions of the compiler.

You can always drop the `--incremental` to build as normal (but you
will still be using the local nightly as your bootstrap).

## Directory Layout

This build system houses all output under the `build` directory, which looks
jyn514 marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -236,63 +144,31 @@ build/
# system will link (using hard links) output from stageN-{std,rustc} into
# each of these directories.
#
# In theory, there is no extra build output in these directories.
# In theory these are working rustc sysroot directories, meaning there is
# no extra build output in these directories.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we need to mention this in the docs, but I'm hoping at some point to distinguish these "public" sysroots from the one's we're using for bootstrap internally: https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Performing.20.60.2E.2Fx.2Epy.20dist.60.20results.20in.20a.20broken.20compiler.20at.20.2E.2E.2E

stage1/
stage2/
stage3/
jyn514 marked this conversation as resolved.
Show resolved Hide resolved
```

## Cargo projects

The current build is unfortunately not quite as simple as `cargo build` in a
directory, but rather the compiler is split into three different Cargo projects:

* `library/std` - the standard library
* `library/test` - testing support, depends on libstd
* `compiler/rustc` - the actual compiler itself

Each "project" has a corresponding Cargo.lock file with all dependencies, and
this means that building the compiler involves running Cargo three times. The
structure here serves two goals:

1. Facilitating dependencies coming from crates.io. These dependencies don't
depend on `std`, so libstd is a separate project compiled ahead of time
before the actual compiler builds.
2. Splitting "host artifacts" from "target artifacts". That is, when building
code for an arbitrary target, you don't need the entire compiler, but you'll
end up needing libraries like libtest that depend on std but also want to use
crates.io dependencies. Hence, libtest is split out as its own project that
is sequenced after `std` but before `rustc`. This project is built for all
targets.

There is some loss in build parallelism here because libtest can be compiled in
parallel with a number of rustc artifacts, but in theory, the loss isn't too bad!

## Build tools

We've actually got quite a few tools that we use in the compiler's build system
and for testing. To organize these, each tool is a project in `src/tools` with a
corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having
independent `Cargo.lock` files) and do not currently explicitly depend on the
compiler or standard library. Compiling each tool is sequenced after the
appropriate libstd/libtest/librustc compile above.

## Extending rustbuild
jyn514 marked this conversation as resolved.
Show resolved Hide resolved

So, you'd like to add a feature to the rustbuild build system or just fix a bug.
Great! One of the major motivational factors for moving away from `make` is that
Rust is in theory much easier to read, modify, and write. If you find anything
excessively confusing, please open an issue on this, and we'll try to get it
documented or simplified, pronto.
When you use the bootstrap system, you'll call it through the entry point script
(`x`, `x.ps1`, or `x.py`). However, most of the code lives in `src/bootstrap`.
`bootstrap` has a difficult problem: it is written in Rust, but yet it is run
before the Rust compiler is built! To work around this, there are two components
of bootstrap: the main one written in rust, and `bootstrap.py`. `bootstrap.py`
is what gets run by entry point script. It takes care of downloading the `stage0`
compiler, which will then build the bootstrap binary written in Rust.

First up, you'll probably want to read over the documentation above, as that'll
give you a high level overview of what rustbuild is doing. You also probably
want to play around a bit yourself by just getting it up and running before you
dive too much into the actual build system itself.
Because there are two separate codebases behind `x.py`, they need to
be kept in sync. In particular, both `bootstrap.py` and the bootstrap binary
parse `config.toml` and read the same command line arguments. `bootstrap.py`
keeps these in sync by setting various environment variables, and the
programs sometimes have to add arguments that are explicitly ignored, to be
read by the other.

After that, each module in rustbuild should have enough documentation to keep
you up and running. Some general areas that you may be interested in modifying
are:
Some general areas that you may be interested in modifying are:

* Adding a new build tool? Take a look at `bootstrap/tool.rs` for examples of
other tools.
Expand Down Expand Up @@ -320,8 +196,9 @@ A 'major change' includes
Changes that do not affect contributors to the compiler or users
building rustc from source don't need an update to `VERSION`.

If you have any questions, feel free to reach out on the `#t-infra` channel in
the [Rust Zulip server][rust-zulip] or ask on internals.rust-lang.org. When
you encounter bugs, please file issues on the rust-lang/rust issue tracker.
If you have any questions, feel free to reach out on the `#t-infra/bootstrap` channel
at [Rust Bootstrap Zulip server][rust-bootstrap-zulip]. When you encounter bugs,
please file issues on the [Rust issue tracker][rust-issue-tracker].

[rust-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra
[rust-bootstrap-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/t-infra.2Fbootstrap
[rust-issue-tracker]: https://github.com/rust-lang/rust/issues
87 changes: 0 additions & 87 deletions src/bootstrap/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,93 +11,6 @@
//! crates.io and Cargo.
//! * A standard interface to build across all platforms, including MSVC
//!
//! ## Architecture
//!
//! The build system defers most of the complicated logic managing invocations
//! of rustc and rustdoc to Cargo itself. However, moving through various stages
//! and copying artifacts is still necessary for it to do. Each time rustbuild
//! is invoked, it will iterate through the list of predefined steps and execute
//! each serially in turn if it matches the paths passed or is a default rule.
//! For each step rustbuild relies on the step internally being incremental and
//! parallel. Note, though, that the `-j` parameter to rustbuild gets forwarded
//! to appropriate test harnesses and such.
//!
//! Most of the "meaty" steps that matter are backed by Cargo, which does indeed
//! have its own parallelism and incremental management. Later steps, like
//! tests, aren't incremental and simply run the entire suite currently.
//! However, compiletest itself tries to avoid running tests when the artifacts
//! that are involved (mainly the compiler) haven't changed.
//!
//! When you execute `x.py build`, the steps executed are:
//!
//! * First, the python script is run. This will automatically download the
//! stage0 rustc and cargo according to `src/stage0.json`, or use the cached
//! versions if they're available. These are then used to compile rustbuild
//! itself (using Cargo). Finally, control is then transferred to rustbuild.
//!
//! * Rustbuild takes over, performs sanity checks, probes the environment,
//! reads configuration, and starts executing steps as it reads the command
//! line arguments (paths) or going through the default rules.
//!
//! The build output will be something like the following:
//!
//! Building stage0 std artifacts
//! Copying stage0 std
//! Building stage0 test artifacts
//! Copying stage0 test
//! Building stage0 compiler artifacts
//! Copying stage0 rustc
//! Assembling stage1 compiler
//! Building stage1 std artifacts
//! Copying stage1 std
//! Building stage1 test artifacts
//! Copying stage1 test
//! Building stage1 compiler artifacts
//! Copying stage1 rustc
//! Assembling stage2 compiler
//! Uplifting stage1 std
//! Uplifting stage1 test
//! Uplifting stage1 rustc
//!
//! Let's disect that a little:
//!
//! ## Building stage0 {std,test,compiler} artifacts
//!
//! These steps use the provided (downloaded, usually) compiler to compile the
//! local Rust source into libraries we can use.
//!
//! ## Copying stage0 {std,test,rustc}
//!
//! This copies the build output from Cargo into
//! `build/$HOST/stage0-sysroot/lib/rustlib/$ARCH/lib`. FIXME: this step's
//! documentation should be expanded -- the information already here may be
//! incorrect.
//!
//! ## Assembling stage1 compiler
//!
//! This copies the libraries we built in "building stage0 ... artifacts" into
//! the stage1 compiler's lib directory. These are the host libraries that the
//! compiler itself uses to run. These aren't actually used by artifacts the new
//! compiler generates. This step also copies the rustc and rustdoc binaries we
//! generated into build/$HOST/stage/bin.
//!
//! The stage1/bin/rustc is a fully functional compiler, but it doesn't yet have
//! any libraries to link built binaries or libraries to. The next 3 steps will
//! provide those libraries for it; they are mostly equivalent to constructing
//! the stage1/bin compiler so we don't go through them individually.
//!
//! ## Uplifting stage1 {std,test,rustc}
//!
//! This step copies the libraries from the stage1 compiler sysroot into the
//! stage2 compiler. This is done to avoid rebuilding the compiler; libraries
//! we'd build in this step should be identical (in function, if not necessarily
//! identical on disk) so there's no need to recompile the compiler again. Note
//! that if you want to, you can enable the full-bootstrap option to change this
//! behavior.
//!
//! Each step is driven by a separate Cargo project and rustbuild orchestrates
//! copying files between steps and otherwise preparing for Cargo to run.
//!
//! ## Further information
//!
//! More documentation can be found in each respective module below, and you can
Expand Down