Explore sandboxed build scripts

[nikomatsakis]

Metadata
Owner(s)	@weihanglo
Team(s)	cargo, compiler
Goal document	2024h2/sandboxed-build-script

Summary

Explore different strategies for sandboxing build script executions in Cargo.

Tasks and status

• Explore prior art (@weihanglo) Explore sandboxed build scripts #108 (comment)
• Design & Experiment (@weihanglo)
  • Custom sandbox runtime unstable config feat(sandbox): add -Zsandbox unstable flag weihanglo/cargo#66
  • Custom std for wasm runtime to support process spawning
• Discussion and moral support (cargo )
  • Not rolling over to 2025H1 project goals
• Security reviews ()
• Standard reviews (cargo )
• Collaboration with GSoC proc-macro project (compiler )
• Summary of experiments or RFC (@weihanglo)

[nikomatsakis]

This issue is intended for status updates only.

For general questions or comments, please contact the owner(s) directly.

[weihanglo]

Key developments:

Have been looking into different sandbox runtime choices. Here is a simple version of the comparison of three potential choices:

• eBPF
  • Required privilege
    • When installing: need other capability for like network monitoring CAP_PERFMON CAP_NET_ADMIN
    • When running: CAP_SYS_ADMIN or CAP_BPF for loading eBPF programs, unless unprivileged eBPF is enabled
    • Most Linux distributions don't allow unprivileged users to run eBPF programs.
  • Cross-platform
    • Linux: ✅
    • Windows: 🚧 https://github.com/microsoft/ebpf-for-windows
    • Darwin: ❓ https://www.reddit.com/r/eBPF/comments/y1whyh/plans_for_ebpf_on_macos/
  • Abilities
    • Process spawn: ✅
    • Network access: ✅
    • Filesystem access: ✅
  • How to use: leverage LLVM or any compiler to build eBPF bytecode
  • Security
    • Check capabilities when loading a eBPF program
• OCI spec compliant runtime
  • Required privilege
  • Cross-platform: ✅
  • Abilities
    • Process spawn: ✅ varies on different platforms
    • Network access: ✅ varies on different platforms
    • Filesystem access: ✅ via mounts in config.json
  • How to use: usually invoke a standalone binary (youki), but some are embeddable (northstar).
  • Security
  • Container technologies are based on system virtualization and isolation mechanisms, suchas cgropu or Hyper-V. These may bring large footprints (Can you imagine cargo run ubuntu:latest to run docker image!?), and is not really portable.
  • Well-defined (it has a specification!)
• WASI
  • Required privilege
    • Usually not required.
  • Cross-platform: ✅
  • Abilities
    • Process spawn: ❓ Unknown posix_spawn/CreateProcess API WebAssembly/WASI#414.
    • Network access: 🚧 Phase 3 (implementation) https://github.com/WebAssembly/wasi-sockets
    • Filesystem access: 🚧 Phase 3 (implementation) https://github.com/WebAssembly/wasi-filesystem.
  • WASI is designed exactly for sandbox use cases.
  • Might not be configurable enough; it depends on runtime implementations and/or Wasi-libc. Usually permissions are granted via CLI options, for example wasmtime --dir.

There are prior research on cross-over between each of these options. I've been busy these two weeks. Will update a more detail post for prior arts afterward.

The biggest challenge I am seeing now is spawning external processes. Most build script usages invoke some external binaries, like pkg-config for building *-sys crates, or protoc for generating protobuf bindings. If process spawning is that common, we need to find a way to provide a fine-grained permission granting scheme. I don't want it to see an “all-or-nothing” mechanism when process spawning is needed.

The other huge headache is setting library search paths. We cannot know every possible path of system libraries ahead of time, but we need to grant access to the runtime.

Blockers:

None.

Help wanted:

None.

[weihanglo]

Having a family urgency. I will be back after RustConf.

[weihanglo]

Key developments:

Building a workable version of wasm-based build script (not yet done). There are some technical difficulties. Not blocking but need to be addressed.

• Build script is assumed to build for host platform everywhere in Cargo's codebase, which is hard to abstract away or hack around.
• wasm32-wasip1 is not shipped by default; we need to rustup target add wasm32-wasip1 to build wasm.
• Also, Cargo learns info of target platforms on-demand if --target flag is provided. Not providing --target wasm32-wasip1 means no target information available. AFAIK, this is already an issue of artifact-dependencies, e.g., Panic when combining -Zbuild-std with artifact dependencies cargo#10444 and Panic when depending on a package with artifact dependencies cargo#11260.

In order to make the development independent of Cargo, RUSTC_WRAPPER was the first approach I tried, though it still lacks some extension point in Cargo:

• There is no way to intercept build script execution, so we need to hack into Cargo itself. RUSTC_WRAPPER can only intercept rustc invocations.
• target.runner is available but for cargo test and cargo run only.

Some compatibility issues came up when integrating with Cargo:

• How to handle build script override.
• artifact-dependencies can compile build script to any other platform, making it challenging to handle.

Blockers:

None.

Help wanted:

None.

[weihanglo]

Have a working-in-process pull request in weihanglo/cargo#66.

Let me copy some texts from there :)

What did we achieve in this experiment?

As you can see, we can easily swap to any sandbox runner with a custom target.

We use wasm32-wasip1 above as an example,
as it is the one with smaller footprint, very cross-platform, and pretty popular in the Rust community.
However, it turns out that `wasm32-wasip1 doesn't support POSIX process spawning.
Use cases of process spawning in build scripts are essential, such as

• Calling pkg-config to find system libraries.
• Invoking compilers to build non Rust code, e.g., C compiler, Protobuf compiler.
• Invoking cargo metadata to get crate metadata.
• Retrieving version control information via git.

According to the design axioms,

Restrcting process spawning is the top one axiom.
We made that with wasm32-wasip1,
but have no way to opt-out.
This contradicts to the "ensuring -sys crates be built" axiom.

As a result, it is unlikely to use wasm32-wasip1 as a default sandbox environment with this experiment.

Other possibilities

Have talked to some other folks, there are some potential route we could take if we chose wasi as a default sandbox environment.

The offical build-rs crate to the rescue

The Cargo team recently adopted the build-rs crate.
It is going to be the official crate providing API for writing build scripts.
We could take the advantage of it, telling everyone instead of using std::process::Command, use something like build_rs::Command. So that Cargo could have a full control over how a build script spawning processes.

While it sounds ideal, this doesn't help the current situation because

• It doesn't help old crates.
• It's hard to convince people to change their build scripts for trying an unstable feature.
• Using build-rs in build scripts may increase build time; people may refuse to use.
• I myself tend to have a big switch to turn on and off, which is easier to try a new feature.

A Cargo-flavored wasi standard library

There was a discussion in the GSoC "sandboxed proc-macro with wasm" about shipping a custom verion of the standard library for sandboxed wasm. For Cargo's build script, we could potentially ship a wasm32-wasi-cargo target. The std in this custom target could intercept any exec call or process spawning. Then it calls back to the host process (which is Cargo in our case) to determine how to handle process execution. The host process could either reject, or run the external program and post back the result.

This idea sounds pretty hacky and need more investigations of the communication mechanism between the host process and the wasm runtime. Perhaps via sockets, The WebAssembly Component Model, other host function call mechanism. There is also WASIX project which supprots fork/exec though it is currently not a WASI standard not even a proposal.

Continue with other more mature sandbox runtime choices

Since one of major design space is the user interafce of sandbox configuration,
we could leave off sandbox runtimes and explore more on the configuration side.

We could, for example, use docker or eBPF as a temporary default runtime, and explore how the configuration should look like.
We may want to take a look at the configuration of Cackle-rs as a starting point. By doing so, we wouldn't block on waiting for wasm runtime to being more mature.

[weihanglo]

Key Developments

Unfortunately, no significant progress has been made. However, we received feedback on the previous experiment: weihanglo/cargo#66.

Alex, who has been working on WebAssembly and is also an honorable Rust project contributor, expressed interest in adding POSIX process support to WASI. However, according to their comments, this seems unlikely in the near future. While the experiment with a Cargo-flavored WASI standard library remains incomplete, it is still on my to-do list. At the very least, I want to give it a try and see how gnarly it would be.

In this comment, Ralf pointed out that supporting process spawning may be somewhat pointless, as one of WASI's key selling points is avoiding the execution of external processes altogether. To me, this feels more like a UX design issue. For example, VS Code prompts users to "trust" a project when it is opened, although most users simply click "trust." With better UX design—such as interactive shell sessions for crate approvals or pre-approval lists from a company (similar to how cargo-vet operates)—an opt-in mechanism would be a better default than allowing everything by default.

Samuel shared a project called build-wrap, which leverages Bubblewrap. While not cross-platform, the configuration interface of Bubblewrap is inspiring and worth exploring, particularly for how it sandboxes system libraries and binaries.

Jeff suggested using the WebAssembly Component Model to allow interaction with a set of well-known programs (e.g., pkg-config and cc), with the host exposing an interface via Wasm components to build scripts. This would enable build scripts to be written in plain Rust while aligning with the use cases of the Wasm Component Model. This idea is particularly promising after the integration of build-rs into Cargo as a workspace member. While build-rs is a lower-level crate, it opens doors to integrating future build script helpers or interfaces.

Not Rolling Over to 2025H1

Despite the valuable feedback and interest, after discussions with the Cargo team, this initiative will not continue as a project goal for 2025H1. Instead, there are alternative approaches worth exploring that may require less effort or depend less on parallel developments in other technologies:

• Invest in system-deps: This crate provides a declarative way to configure system dependencies. Building C (system) libraries is a key use case for build scripts and one of the reasons they exist. If system-deps became a native Cargo solution, many packages could transition to being build-script-free. This issue explores how *-sys crates are used and what is needed to increase adoption.

• Cackle-like approaches: Cackle is a code checker that analyzes API usage across dependencies by wrapping rustc and the linker to perform API analysis on object and debug info files. For build scripts, APIs like std::net or std::process could be disabled by default. While this is not a true sandboxing mechanism, Cackle optionally provides Bubblewrap as a sandbox environment, similar to the earlier-mentioned build-wrap.

• A fully sandboxed build environment: Instead of partial sandboxing, we could explore integrating a fully sandboxed build environment, akin to how the Haskell tool Stack operates. Stack natively supports Docker with commands like stack docker and also integrates with the Nix build system. However, this approach has significant compatibility challenges, such as limited Windows support and the added complexity of learning Docker or similar tools. While Haskell's Cabal system offers sandboxing or Nix-style local builds, it is more analogous to Python's virtualenv than the sandboxing we're discussing here.

We want to explore various possibilities beyond the Wasm sandbox environment. That said, as mentioned earlier, personally I am still interested in experimenting further with Wasm. Sorry for not making more progress for the past month.

[weihanglo]

Just found https://github.com/AsahiLinux/muvm. Also an interesting runtime choice.