Add docs for building Fuchsia locally and in CI (#1989)

src/SUMMARY.md

@@ -24,8 +24,10 @@
     - [Compiletest](./tests/compiletest.md)
         - [UI tests](./tests/ui.md)
         - [Test headers](./tests/headers.md)
+    - [Integration testing](./tests/integration.md)
+        - [Crater](./tests/crater.md)
+        - [Fuchsia](./tests/fuchsia.md)
     - [Performance testing](./tests/perf.md)
-    - [Crater](./tests/crater.md)
     - [Suggest tests tool](./tests/suggest-tests.md)
 - [Debugging the compiler](./compiler-debugging.md)
     - [Using the tracing/logging instrumentation](./tracing.md)

src/tests/fuchsia.md

@@ -0,0 +1,167 @@
+# Fuchsia integration tests
+
+[Fuchsia](https://fuchsia.dev) is an open-source operating system with about 2
+million lines of Rust code.[^loc] It has caught a large number of [regressions]
+in the past and was subsequently included in CI.
+
+## Building Fuchsia in CI
+
+Fuchsia builds as part of the suite of bors tests that run before a pull request
+is merged.
+
+If you are worried that a pull request might break the Fuchsia builder and want
+to test it out before submitting it to the bors queue, simply add this line to
+your PR description:
+
+> try-job: x86_64-gnu-integration
+
+Then when you `@bors try` it will pick the job that builds Fuchsia.
+
+## Building Fuchsia locally
+
+Because Fuchsia uses languages other than Rust, it does not use Cargo as a build
+system. It also requires the toolchain build to be configured in a [certain
+way][build-toolchain].
+
+The recommended way to build Fuchsia is to use the Docker scripts that check out
+and run a Fuchsia build for you. If you've run Docker tests before, you can
+simply run this command from your Rust checkout to download and build Fuchsia
+using your local Rust toolchain.
+
+```
+src/ci/docker/run.sh x86_64-gnu-integration
+```
+
+See the [Testing with Docker](docker.md) chapter for more details on how to run
+and debug jobs with Docker.
+
+Note that a Fuchsia checkout is *large* – as of this writing, a checkout and
+build takes 46G of space – and as you might imagine, it takes awhile to
+complete.
+
+### Modifying the Fuchsia checkout
+
+The main reason you would want to build Fuchsia locally is because you need to
+investigate a regression. After running a Docker build, you'll find the Fuchsia
+checkout inside the `obj/fuchsia` directory of your Rust checkout.  If you
+modify the `KEEP_CHECKOUT` line in the [build-fuchsia.sh] script to
+`KEEP_CHECKOUT=1`, you can change the checkout as needed and rerun the build
+command above. This will reuse all the build results from before.
+
+You can find more options to customize the Fuchsia checkout in the
+[build-fuchsia.sh] script.
+
+### Customizing the Fuchsia build
+
+You can find more info about the options used to build Fuchsia in Rust CI in the
+[build_fuchsia_from_rust_ci.sh] script invoked by [build-fuchsia.sh].
+
+The Fuchsia build system uses [GN], a metabuild system that generates [Ninja]
+files and then hands off the work of running the build to Ninja.
+
+Fuchsia developers use `fx` to run builds and perform other development tasks.
+This tool is located in `.jiri_root/bin` of the Fuchsia checkout; you may need
+to add this to your `$PATH` for some workflows.
+
+There are a few `fx` subcommands that are relevant, including:
+
+* `fx set` accepts build arguments, writes them to `out/default/args.gn`, and runs GN.
+* `fx build` builds the Fuchsia project using Ninja. It will automatically pick
+  up changes to build arguments and rerun GN. By default it builds everything,
+  but it also accepts target paths to build specific targets (see below).
+* `fx clippy` runs Clippy on specific Rust targets (or all of them). We use this
+  in the Rust CI build to avoid running codegen on most Rust targets. Underneath
+  it invokes Ninja, just like `fx build`. The clippy results are saved in json
+  files inside the build output directory before being printed.
+
+#### Target paths
+
+GN uses paths like the following to identify build targets:
+
+```
+//src/starnix/kernel:starnix_core
+```
+
+The initial `//` means the root of the checkout, and the remaining slashes are
+directory names. The string after `:` is the _target name_ of a target defined
+in the `BUILD.gn` file of that directory.
+
+The target name can be omitted if it is the same as the directory name. In other
+words, `//src/starnix/kernel` is the same as `//src/starnix/kernel:kernel`.
+
+These target paths are used inside `BUILD.gn` files to reference dependencies,
+and can also be used in `fx build`.
+
+#### Modifying compiler flags
+
+You can put custom compiler flags inside a GN `config` that is added to a target.
+As a simple example:
+
+```
+config("everybody_loops") {
+    rustflags = [ "-Zeverybody-loops" ]
+}
+
+rustc_binary("example") {
+    crate_root = "src/bin.rs"
+    # ...existing keys here...
+    configs += [ ":everybody_loops" ]
+}
+```
+
+This will add the flag `-Zeverybody-loops` to rustc when building the `example`
+target. Note that you can also use [`public_configs`] for a config to be added
+to every target that depends on that target.
+
+If you want to add a flag to every Rust target in the build, you can add
+rustflags to the [`//build/config:compiler`] config or to the OS-specific
+configs referenced in that file. Note that `cflags` and `ldflags` are ignored on
+Rust targets.
+
+#### Running ninja and rustc commands directly
+
+Going down one layer, `fx build` invokes `ninja`, which in turn eventually
+invokes `rustc`. All build actions are run inside the out directory, which is
+usually `out/default` inside the Fuchsia checkout.
+
+You can get ninja to print the actual command it invokes by forcing that command
+to fail, e.g. by adding a syntax error to one of the source files of the target.
+Once you have the command, you can run it from inside the output directory.
+
+After changing the toolchain itself, the build setting `rustc_version_string` in
+`out/default/args.gn` needs to be changed so that `fx build` or `ninja` will
+rebuild all the Rust targets. This can be done in a text editor and the contents
+of the string do not matter, as long as it changes from one build to the next.
+[build_fuchsia_from_rust_ci.sh] does this for you by hashing the toolchain
+directory.
+
+The Fuchsia website has more detailed documentation of the [build system].
+
+#### Other tips and tricks
+
+When using `build_fuchsia_from_rust_ci.sh` you can comment out the `fx set`
+command after the initial run so it won't rerun GN each time. If you do this you
+can also comment out the version_string line to save a couple seconds.
+
+`export NINJA_PERSISTENT_MODE=1` to get faster ninja startup times after the
+initial build.
+
+## Fuchsia target support
+
+To learn more about Fuchsia target support, see the Fuchsia chapter in [the
+rustc book][platform-support].
+
+[regressions]: https://gist.github.com/tmandry/7103eba4bd6a6fb0c439b5a90ae355fa
+[build-toolchain]: https://fuchsia.dev/fuchsia-src/development/build/rust_toolchain
+[build-fuchsia.sh]: https://github.com/rust-lang/rust/blob/99f77a2eda555b50b518f74823ab636a20efb87f/src/ci/docker/host-x86_64/x86_64-gnu-integration/build-fuchsia.sh
+[build_fuchsia_from_rust_ci.sh]: https://cs.opensource.google/fuchsia/fuchsia/+/main:scripts/rust/build_fuchsia_from_rust_ci.sh?q=build_fuchsia_from_rust_ci&ss=fuchsia
+[platform-support]: https://doc.rust-lang.org/nightly/rustc/platform-support/fuchsia.html
+[GN]: https://gn.googlesource.com/gn/+/main#gn
+[Ninja]: https://ninja-build.org/
+[`public_configs`]: https://gn.googlesource.com/gn/+/main/docs/reference.md#var_public_configs
+[`//build/config:compiler`]: https://cs.opensource.google/fuchsia/fuchsia/+/main:build/config/BUILD.gn;l=121;drc=c26c473bef93b33117ae417893118907a026fec7
+[build system]: https://fuchsia.dev/fuchsia-src/development/build/build_system
+
+[^loc]: As of June 2024, Fuchsia had about 2 million lines of first-party Rust code
+and a roughly equal amount of third-party code, as counted by tokei (excluding
+comments and blanks).

src/tests/integration.md

@@ -0,0 +1,49 @@
+# Integration testing
+
+Rust tests integration with real-world code to catch regressions and make
+informed decisions about the evolution of the language.
+
+## Testing methods
+
+### Crater
+
+Crater is a tool which runs tests on many thousands of public projects. This
+tool has its own separate infrastructure for running, and is not run as part of
+CI. See the [Crater chapter](crater.md) for more details.
+
+### Cargo test
+
+`cargotest` is a small tool which runs `cargo test` on a few sample projects
+(such as `servo`, `ripgrep`, `tokei`, etc.).
+This runs as part of CI and ensures there aren't any significant regressions.
+
+> Example: `./x test src/tools/cargotest`
+
+### Integration builders
+
+Integration jobs build large open-source Rust projects that are used as
+regression tests in CI. Our integration jobs build the following projects:
+
+- [Fuchsia](fuchsia.md)
+- Linux kernel
+
+## A note about terminology
+
+The term "integration testing" can be used to mean many things. Many of the
+compiletest tests within the Rust repo could be justifiably called integration
+tests, because they test the integration of many parts of the compiler, or test
+the integration of the compiler with other external tools. Calling all of them
+integration tests would not be very helpful, especially since those kinds of
+tests already have their own specialized names.
+
+We use the term "integration" here to mean integrating the Rust compiler and
+toolchain with the ecosystem of Rust projects that depend on it. This is partly
+for lack of a better term, but it also reflects a difference in testing approach
+from other projects and the comparative advantage it implies.
+
+The Rust compiler is part of the ecosystem, and the ecosystem is in many cases
+part of Rust, both in terms of libraries it uses and in terms of the efforts of many
+contributors who come to "scratch their own itch". Finally, because Rust has the
+ability to do integration testing at such a broad scale, it shortens development
+cycles by finding defects earlier.
+

src/tests/intro.md

@@ -132,19 +132,12 @@ More information can be found in the [toolstate documentation].
 [toolstate documentation]: https://forge.rust-lang.org/infra/toolstate.html
 [toolstate website]: https://rust-lang-nursery.github.io/rust-toolstate/
 
-### Cargo test
+### Integration testing
 
-`cargotest` is a small tool which runs `cargo test` on a few sample projects
-(such as `servo`, `ripgrep`, `tokei`, etc.).
-This ensures there aren't any significant regressions.
-
-> Example: `./x test src/tools/cargotest`
-
-### Crater
-
-Crater is a tool which runs tests on many thousands of public projects.
-This tool has its own separate infrastructure for running.
-See the [Crater chapter](crater.md) for more details.
+Rust tests integration with real-world code to catch regressions and make
+informed decisions about the evolution of the language. There are several kinds
+of integration tests, including Crater. See the [Integration testing
+chapter](integration.md) for more details.
 
 ### Performance testing