Merge remote-tracking branch 'upstream/main' into cr2-organization

README.md

@@ -7,9 +7,10 @@
 This repository has the source code for Comprehensive Rust 🦀, a multi-day Rust
 course developed by the Android team. The course covers all aspects of Rust,
 from basic syntax to generics and error handling. It also includes deep dives on
-[Android], [bare-metal], and [concurrency].
+[Android], [Chromium], [bare-metal], and [concurrency].
 
 [Android]: https://google.github.io/comprehensive-rust/android.html
+[Chromium]: https://google.github.io/comprehensive-rust/comprehensive.html
 [bare-metal]: https://google.github.io/comprehensive-rust/bare-metal.html
 [concurrency]: https://google.github.io/comprehensive-rust/concurrency.html
 

src/SUMMARY.md

@@ -235,6 +235,42 @@
 - [Exercises](exercises/android/morning.md)
 
 
+
+# Chromium
+
+----
+
+- [Welcome](chromium.md)
+- [Setup](chromium/setup.md)
+- [Using cargo for experimental tools](chromium/cargo.md)
+- [Policy](chromium/policy.md)
+- [Build Rules](chromium/build-rules.md)
+  - [Unsafe code](chromium/build-rules/unsafe.md)
+  - [Depending on Rust code from Chromium C++](chromium/build-rules/depending.md)
+  - [Visual Studio code](chromium/build-rules/vscode.md)
+  - [Exercise](exercises/chromium/build-rules.md)
+- [Interoperability with C++](chromium/interoperability-with-cpp.md)
+  - [Example bindings](chromium/interoperability-with-cpp/example-bindings.md)
+  - [Limitations of cxx](chromium/interoperability-with-cpp/limitations-of-cxx.md)
+  - [cxx error handling](chromium/interoperability-with-cpp/error-handling.md)
+  - [Using cxx in Chromium](chromium/interoperability-with-cpp/using-cxx-in-chromium.md)
+  - [Exercise](exercises/chromium/interoperability-with-cpp.md)
+- [Adding third party crates](chromium/adding-third-party-crates.md)
+  - [Configuring Cargo.toml](chromium/adding-third-party-crates/configuring-cargo-toml.md)
+  - [Configuring gnrt_config.toml](chromium/adding-third-party-crates/configuring-gnrt-config-toml.md)
+  - [Downloading crates](chromium/adding-third-party-crates/downloading-crates.md)
+  - [Generating gn build rules](chromium/adding-third-party-crates/generating-gn-build-rules.md)
+  - [Resolving problems](chromium/adding-third-party-crates/resolving-problems.md)
+    - [Build scripts which generate code](chromium/adding-third-party-crates/resolving-problems/build-scripts-which-generate-code.md)
+    - [Build scripts which build C++ or take arbitrary actions](chromium/adding-third-party-crates/resolving-problems/build-scripts-which-take-arbitrary-actions.md)
+  - [Depending on a crate](chromium/adding-third-party-crates/depending-on-a-crate.md)
+  - [Reviews and audits](chromium/adding-third-party-crates/reviews-and-audits.md)
+  - [Checking into Chromium source code](chromium/adding-third-party-crates/checking-in.md)
+  - [Keeping crates up to date](chromium/adding-third-party-crates/keeping-up-to-date.md)
+  - [Exercise](exercises/chromium/third-party.md)
+- [Bringing it together - Exercise](exercises/chromium/bringing-it-together.md)
+
+
 # Bare Metal: Morning
 
 ----

src/chromium.md

@@ -0,0 +1,9 @@
+# Welcome to Rust in Chromium
+
+Rust is supported for third-party libraries in Chromium, with first-party glue
+code to connect between Rust and existing Chromium C++ code.
+
+> Today, we'll call into Rust to do something silly with strings. If you've
+> got a corner of the code where you're displaying a UTF8 string to the user,
+> feel free to follow this recipe in your part of the codebase instead of
+> the exact part we talk about.

src/chromium/adding-third-party-crates.md

@@ -0,0 +1,32 @@
+# Adding third party crates
+
+Rust libraries are called "crates" and are found at [crates.io][0]. It's *very
+easy* for Rust crates to depend upon one another. So they do!
+
+| Property | C++ library | Rust crate |
+| --- | --- | --- |
+| Build system | Lots | Consistent - `Cargo.toml` |
+| Typical library size | Large-ish | Small |
+| Transitive dependencies | Few | Lots |
+
+For a Chromium engineer, this has pros and cons:
+
+* All crates use a common build system so we can automate their inclusion into
+  Chromium...
+* ... but, crates typically have transitive dependencies, so you will
+  likely have to bring in multiple libraries.
+
+We'll discuss:
+
+* How to put a crate in the Chromium source code tree
+* How to make `gn` build rules for it
+* How to audit its source code for sufficient safety.
+
+[0]: https://crates.io
+
+<details>
+All of the things in the table on this slide are generalizations, and
+counter-examples can be found. But in general it's important for students
+to understand that most Rust code depends on other Rust libraries, because
+it's easy to do so, and that this has both benefits and costs.
+</details>

src/chromium/adding-third-party-crates/checking-in.md

@@ -0,0 +1,23 @@
+# Checking crates into Chromium source code
+
+`git status` should reveal:
+* Crate code in `//third_party/rust/chromium_crates_io`
+* Metadata (`BUILD.gn` and `README.chromium`) in `//third_party/rust/<crate>/<version>`
+
+Please also add an `OWNERS` file in the latter location.
+
+You should land all this, along with your `Cargo.toml` and `gnrt_config.toml` changes, into
+the Chromium repo.
+
+**Important**: you need to use `git add -f` because otherwise `.gitignore` files
+may result in some files being skipped.
+
+As you do so, you might find presubmit checks fail because of non-inclusive
+language. This is because Rust crate data tends to include names of git branches,
+and many projects still use non-inclusive terminology there. So you may need
+to run:
+
+```shell
+infra/update_inclusive_language_presubmit_exempt_dirs.sh > infra/inclusive_language_presubmit_exempt_dirs.txt
+git add -p infra/inclusive_language_presubmit_exempt_dirs.txt # add whatever changes are yours
+```

src/chromium/adding-third-party-crates/configuring-cargo-toml.md

@@ -0,0 +1,23 @@
+# Configuring the `Cargo.toml` file to add crates
+
+Chromium has a single set of centrally-managed direct crate dependencies.
+These are managed through a single [`Cargo.toml`][0]:
+
+```toml
+[dependencies]
+bitflags = "1"
+cfg-if = "1"
+cxx = "1"
+# lots more...
+```
+
+As with any other `Cargo.toml`, you can specify [more details about
+the dependencies][1] - most commonly, you'll want to specify the `features` that
+you wish to enable in the crate.
+
+When adding a crate to Chromium, you'll often need to provide some extra
+information in an additional file, `gnrt_config.toml`, which we'll meet next.
+
+
+[0]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/rust/chromium_crates_io/Cargo.toml
+[1]: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html

src/chromium/adding-third-party-crates/configuring-gnrt-config-toml.md

@@ -0,0 +1,29 @@
+# Configuring `gnrt_config.toml`
+
+Alongside `Cargo.toml` is [`gnrt_config.toml`][0]. This contains Chromium-specific
+extensions to crate handling.
+
+If you add a new crate, you should specify at least the `group`.
+This is one of:
+
+```toml
+#   'safe': The library satisfies the rule-of-2 and can be used in any process.
+#   'sandbox': The library does not satisfy the rule-of-2 and must be used in
+#              a sandboxed process such as the renderer or a utility process.
+#   'test': The library is only used in tests.
+```
+
+For instance,
+
+```toml
+[crate.my-new-crate]
+group = 'test' # only used in test code
+```
+
+Depending on the crate source code layout, you may also need to use this
+file to specify where its `LICENSE` file(s) can be found.
+
+Later, we'll see some other things you will need to configure in this file
+to resolve problems.
+
+[0]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/rust/chromium_crates_io/gnrt_config.toml

src/chromium/adding-third-party-crates/depending-on-a-crate.md

@@ -0,0 +1,23 @@
+# Depending on a crate
+
+Once you've added a third-party crate and generated build rules,
+depending on a crate is simple. Find your `rust_static_library` target,
+and add a `dep` on the `:lib` target within your crate.
+
+Specifically,
+
+```bob
+                     +------------+      +----------------------+
+"//third_party/rust" | crate name | "/v" | major semver version | "/:lib"
+                     +------------+      +----------------------+
+```
+
+For instance,
+
+```gn
+rust_static_library("my_rust_lib") {
+  crate_root = "lib.rs"
+  sources = [ "lib.rs" ]
+  deps = [ "//third_party/rust/example_rust_crate/v1:lib" ]
+}
+```

src/chromium/adding-third-party-crates/downloading-crates.md

@@ -0,0 +1,30 @@
+# Downloading crates
+
+A tool called `gnrt` knows how to download crates and how to generate `BUILD.gn`
+rules.
+
+To start, download the crate you want like this:
+
+```shell
+cd chromium/src
+vpython3 tools/crates/run_gnrt.py -- vendor
+```
+
+> Although the `gnrt` tool is part of the Chromium source code,
+> by running this command you will be downloading and running its dependencies
+> from `crates.io`. See [the earlier section][0] discussing this security
+> decision.
+
+This `vendor` command may download:
+* Your crate
+* Direct and transitive dependencies
+* New versions of other crates, as required by `cargo` to resolve
+  the complete set of crates required by Chromium.
+
+If a crate in `//third_party/rust/chromium_crates_io/patches` was updated as
+part of vendoring, then reapply patches to it by running
+`cd third_party/rust/chromium_crates_io; ./apply_patches.sh`.
+
+
+
+[0]: ../cargo.md

src/chromium/adding-third-party-crates/generating-gn-build-rules.md

@@ -0,0 +1,25 @@
+# Generating `gn` build rules
+
+Once you've downloaded the crate, generate the `BUILD.gn` files like this:
+
+```shell
+vpython3 tools/crates/run_gnrt.py -- gen
+```
+
+Now run `git status`. You should find:
+
+* At least one new crate source code in `third_party/rust/chromium_crates_io/vendor`
+* At least one new `BUILD.gn` in `third_party/rust/<crate name>/v<major semver version>`
+* An appropriate `README.chromium`
+
+The "major semver version" is a [Rust "semver" version number][0].
+
+Take a close look, especially at the things generated in `third_party/rust`.
+
+<details>
+Talk a little about semver - and specifically the way that in Chromium
+it's to allow multiple incompatible versions of a crate, which is discouraged
+but sometimes necessary in the cargo ecosystem.
+</detail>
+
+[0]: https://doc.rust-lang.org/cargo/reference/semver.html

src/chromium/adding-third-party-crates/keeping-up-to-date.md

@@ -0,0 +1,8 @@
+# Keeping crates up to date
+
+As the OWNER of any third party Chromium dependency, you are
+[expected to keep it up to date with any security fixes][0]. It is hoped
+that we will soon automate this for Rust crates, but for now, it's still
+your responsibility just as it is for any other third party dependency.
+
+[0]: https://chromium.googlesource.com/chromium/src/+/main/docs/adding_to_third_party.md#add-owners

src/chromium/adding-third-party-crates/resolving-problems.md

@@ -0,0 +1,21 @@
+# Resolving problems
+
+If your build fails, it may be because of a `build.rs`: programs which do arbitrary
+things at build time. This is fundamentally at odds with the design of `gn`
+and `ninja` which aim for static, deterministic, build rules to maximize
+parallelism and repeatability of builds.
+
+Some `build.rs` actions are automatically supported; others require action:
+
+| build script effect | Supported by our gn templates | Work required by you |
+|-----|-----|-----|
+| Checking rustc version to configure features on and off | Yes | None |
+| Checking platform or CPU to configure features on and off | Yes | None |
+| Generating code | Yes | Yes - specify in `gnrt_config.toml` |
+| Building C/C++ | No | Patch around it |
+| Arbitrary other actions | No | Patch around it |
+
+Fortunately, most crates don't contain a build script, and fortunately, most
+build scripts only do the top two actions.
+
+[0]: https://doc.rust-lang.org/cargo/reference/build-scripts.html

src/chromium/adding-third-party-crates/resolving-problems/build-scripts-which-generate-code.md

@@ -0,0 +1,22 @@
+# Build scripts which generate code
+
+If `ninja` complains about missing files, check the `build.rs` to see if it
+writes source code files.
+
+If so, modify [`gnrt_config.toml`][1] to add `build-script-outputs` to the
+crate. If this is a transitive dependency, that is, one on which Chromium
+code should not directly depend, also add `allow-first-party-usage=false`.
+There are several examples already in that file:
+
+```toml
+[crate.unicode-linebreak]
+allow-first-party-usage = false
+build-script-outputs = [ "tables.rs" ]
+```
+
+Now rerun [`gnrt.py -- gen`][2] to regenerate `BUILD.gn` files to inform ninja
+that this particular output file is input to subsequent build steps.
+
+
+[1]: ../configuring-gnrt-config-toml.md
+[2]: ../generating-gn-build-rules.md

src/chromium/adding-third-party-crates/resolving-problems/build-scripts-which-take-arbitrary-actions.md

@@ -0,0 +1,24 @@
+# Build scripts which build C++ or take arbitrary actions
+
+Some crates use the [`cc`][2] crate to build and link C/C++ libraries.
+Other crates parse C/C++ using [`bindgen`][3] within their build scripts.
+These actions can't be supported in a Chromium context - our gn, ninja
+and LLVM build system is very specific in expressing relationships between
+build actions.
+
+So, your options are:
+
+* Avoid these crates
+* Apply a patch to the crate.
+
+Patches should be kept in `third_party/rust/chromium_crates_io/patches/<crate>` -
+see for example the [patches against the cxx crate][4]. There is currently
+no automation - [simply create and apply patches manually][5] to remove the
+problematic actions from the build script.
+
+If your patches modify the `Cargo.toml` file, rerun `gnrt gen`.
+
+[2]: https://crates.io/crates/cc
+[3]: https://crates.io/crates/bindgen
+[4]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/rust/chromium_crates_io/patches/cxx/
+[5]: https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/rust.md#patching-third_party-crates

src/chromium/adding-third-party-crates/reviews-and-audits.md

@@ -0,0 +1,33 @@
+# Auditing third party crates
+
+Adding new libraries is subject to Chromium's standard [policies][0], but of
+course also subject to code review. As you may be bringing in not just a single
+crate but also transitive dependencies, there may be a lot of code to review.
+On the other hand, safe Rust code can have limited negative side effects.
+How should you review it?
+
+Over time Chromium aims to move to a process based around [cargo vet][1].
+
+Meanwhile, for each new crate addition, we are checking for the following:
+
+* Understand why each crate is used. What's the relationship between crates?
+  If the build system for each crate contains a `build.rs` or procedural
+  macros, work out what they're for. Are they compatible with the way
+  Chromium is normally built?
+* Check each crate seems to be reasonably well maintained
+* Use `cd third-party/rust/chromium_crates_io; cargo audit` to check for
+  known vulnerabilities (first you'll need to `cargo install cargo-audit`,
+  which ironically involves downloading lots of dependencies from the internet[2])
+* Ensure any unsafe code is good enough for the [Rule of Two][3]
+* Check for any use of `fs` or `net` APIs
+* Read all the code at a sufficient level to look for anything out of place
+  that might have been maliciously inserted. (You can't realistically aim
+  for 100% perfection here: there's often just too much code.)
+
+These are just guidelines - work with reviewers from `security@chromium.org`
+to work out the right way to become confident of the crate.
+
+[0]: https://chromium.googlesource.com/chromium/src/+/refs/heads/main/docs/rust.md#Third_party-review
+[1]: https://mozilla.github.io/cargo-vet/
+[2]: ../cargo.md
+[3]: https://chromium.googlesource.com/chromium/src/+/main/docs/security/rule-of-2.md#unsafe-code-in-safe-languages