Skip to content

Latest commit

 

History

History
276 lines (223 loc) · 13.3 KB

0000-sandbox-environment.md

File metadata and controls

276 lines (223 loc) · 13.3 KB
Raw

Summary

This proposes a mechanism to precisely control what environment variables are available to Rust programs at compilation time.

Motivation

Rust supports the env! and option_env! macros which allow Rust programs to query arbitrary process environment variables at compilation time. This is a very flexible mechanism to pass compile-time information to the program; in effect it uses the environment as a form of compile-time directive.

However, in many cases it is too flexible. It poses several problems:

  1. Environment variables are generally not tracked by build systems, so changing a variable is not taken into account. Cargo has an ad-hoc mechanism for doing this in build scripts, but there's nothing to make this guaranteed correct (i.e. that all variables accessed are tracked).
  2. There's no easy way to audit which environment variables a crate accesses. This not only exacerbates the problem above, but it also means that potentially sensitive information in an environment variable can be incorporated into the compiled code.
  3. There's no way to override variables if they're needed by the build process itself. For example, the PATH variable likely needs to be set so that the compiler can execute various sub-processes, but there's no way to override this so that env!("PATH") returns something else. This would be necessary where the compilation environment differs from the deployment environment (such as when cross-compiling).

This RFC proposes a way to precisely control the environment visible to the compile-time macros, while defaulting to the current behaviour of making the entire environment available.

Guide-level explanation

Rust implements the env!() and option_env!() macros to access a logical compile time environment at compilation time. By default, this logical environment is initialized from, and is identical to, the inherited process environment of rustc itself. (The only exception being environment variables whos names or values are not valid utf8.)

There are several command-line options to control this environment and change it from the default:

  • --env-clear - remove all process environment variables from the logical environment, leaving it completely empty.
  • --env-remove VARIABLE - Remove a specific variable from the logical environment. This is an exact match, and it does nothing if that variable is not set.
  • --env-pass VARIABLE - Pass a variable from the process environment to the logical one, even if it had previously been removed. This lets specific variables to be allow-listed without having to explicitly set their value. The variable is ignored if it is not set or not utf8 encoded.
  • --env-set VARIABLE=VALUE - Set a variable in the logical environment. This will either create a new variable, or override an existing value.

The options are processed in the order listed above (ie, clear, then remove, then pass, then set). --env-clear, --env-remove and --env-pass are idempotent, so multiple instances of each are the same as one.. Multiple --env-set options affecting the same variable are processed in command-line order, so later ones override earlier ones.

Reference-level explanation

The implementation of this RFC introduces the notion of:

  • the process environment which is inherited by the rustc process from it's invoker, and
  • a logical environment which is accessed by the env!/option_env! macros

The logical environment is initialized from the complete process environment, excluding only environment variables which are not UTF-8 encoded (name or value).

Once initialized, the logical environment may be manipulated via the --env- command-line options described below.

These environments are fundamentally key-value mappings, which is how they're represented within the session state - a map from String to String.

Processing of the options

These options are processed in order:

  1. --env-clear - remove all variables from the logical environment.
  2. --env-remove VAR - remove a specific variable from the logical environment. May be specified multiple times.
  3. --env-pass VAR- set a variable in the logical environment from the process environment. Ignored if the variable is not set, or is not UTF-8 encoded.
  4. --env-set VAR=VALUE - multiple --env-set options affecting the same variable are processed in command-line order, so later ones override earlier ones.

rustc will only accept UTF-8 encoded command-line options, which affects all these options. This implies that all environment variables must have UTF-8 encoded names and values.

Compile-time behaviour

In general there are two distinct classes of compile-time environment fetch:

  1. operational, used for running the compilation process itself; the specific values of the environment variables have no effect on the compiled code. For example, fetching PATH to work out how to invoke the linker, or TMPDIR to work out where to write temporary files.
  2. directive, where the environment variable values will affect the generated code of the crate, generally by being incorporated into it - for example, using an environment variable as part of a const initializer expression.

In pure Rust code, the env!() and option_env!() macros query the logical environment with no reference to the process environment. These can be used in either an operational way:

include!(concat!(env!("GENDIR"), "/generated.rs"));

(discussion about paths in the environment)

Or as a directive:

const USER: &str = env!("DEFLUSER");

Proc-macros are similar, but since they can run arbitrary code with full access to libstd, classifying environment fetches has to be done on a case-by-case basis. However, while it is possible in principle for a proc-macro to use the environment as a directive, in practice any environment is much more like to be operational. (See discussion about interaction with he proposed tracked environment API.)

Cargo

This has no direct effect on Cargo - it can completely ignore these options and the overall behaviour would be unchanged. However, it's easy to imagine a corresponding RFC for Cargo where it does more explicitly control the logical environment. For example, it could constrain the accessible variables to:

  1. ones that Cargo itself sets
  2. ones that the build script sets via rustc-env
  3. ones that the build script notes as rerun-if-env-changed
  4. explicitly listed in the Cargo.toml

Drawbacks

The primary cost is additional complexity in invoking rustc itself, and additional complexity in documenting env!/option_env!. Procedual macros would need to be changed to access the logical environment, either by adding new environment access APIs, or overriding the implementation of std::env::var (etc) for procmacros.

Rationale and alternatives

One alternative would be to simply do nothing, and leave things as-is.

In a Unix/Linux-like system, the environment can be controlled either with the shell, or the env command. However this requires rustc to be invoked via a shell or the env command, which may not be convenient for a given build system. Alternatively, the buildsystem itself could be modified to suitably configure the environment. However, it would still be strictly less capable, as it would not be able to override variables or remove needed by:

  • rustc itself to run - such as HOME, LD_PRELOAD or LD_LIBRARY_PATH
  • rustc to invoke the linker, such as PATH
  • the linker for its own operation (PATH, and so on)

This can be particularly awkward when it isn't clear which variables are needed by the toolchain - for example, invoking rustc via rustup uses a wider range of variables than directly invoking the rustc binary without an intermediary.

This proposal gives maximal control when needed, without changing the default behaviours at all.

When rustc is embedded or long-running, such as in rls or rust-analyzer, then its necessary to explicitly set the logical environment for each crate, rather than just inheriting the process environment.

Prior art

C/C++ compilers typically have the ability set preprocessor macros via the command-line. They can be set to arbitrary values via the -D option. This is logically equivalent to both of Rust's mechanisms:

  • preprocessor macros can be used as predicates in compile-time conditional compilation tests, and
  • they can be expanded into the text of the program itself

These macros are explicit on the command-line, so they're easy to take into account as an input to the compilation process. And the tools driving the C compiler don't need any additional way to control the process environment.

Rust has a couple of mechanisms for compile-time configuration:

  • It has the --cfg options which set flags which can be tested with compile-time predicates. These are strictly binary choices, which allow for conditional complilation.
  • It has the process environment which can be queried at compile-time with env!() which evaluate to an arbitrary compile-time &'static str constant, which cannot be directly used for conditional compilation. The environment is not directly set via command-line options, but via another mechanism.

This mechanism doesn't change the semantics of either mechanism, but it does make the environment a little more like a C preprocessor macro - they can be precisely set on the command line, and if desired, only via the command line.

In general build systems need to have a precise knowledge of all inputs used to build a particular artifact. This is especially important when trying to implement fully reproducable builds, either for auditability reasons or just to get good hit rates from a build cache. Build systems don't take the environment into account because most of it isn't relevant to builds. Indeed, Rust is the only compiled language I know of which allows direct access to environment variables, so its not a thing that build systems need to take into account. Even Cargo - purpose built for building Rust programs - can't explicitly track what environment variables a piece of Rust code will access, and currently only has limited tools for tracking this.

Unresolved questions

There are two unresolved questions in my mind:

One is how to handle non-UTF-8 environment variable names and values? The standard library has std::env::var_os to fetch the environment in OS-encoded form, but there are no corresponing macros for compile-time. So I think just restricting the logical to pure UTF-8 for names and values is fine.

Future possibilities

Path handling

Environment variables are frequently used for paths - a common pattern is:

include!(env!("SOME_PATH"));

If the variable SOME_PATH expands to a relative path, it is interpreted relative to the source file which contains the include!(). This requires the variable to be set with an explicit knowledge of which source files will reference the environment variable, and how they're layed out within the directory structure. If it is being using from multiple files in different direcctories (or specifically, at different directory depths), then no one relative path can be made to work. In practice the practical alternative is to always use absolute paths.

To address this, an extension would be --env-set-path VAR=PATH (and --env-pass-path VAR) where the value is interpreted as a path relative to the rustc current working directory - in other words, it could be set as a relative path, but it would be interpreted as if it were an absolute path. Absolute paths would always be treated as absolute.

Tracked Environment API

The tracked environment API proposal adds an API to allow proc-macro accesses to the environment to be tracked appropriately. In the context of this RFC, these would be directive accesses, where the environment has a direct effect on the generate crate. As such it would access the logical environment proposed in this RFC.

By the same token, if the proc-macro performs operational environment fetches, then it could still directly use the std::env::var API to fetch directly from the process environment. It is, of course, the proc-macro author's responsibility to understand the intent of their environment use, and correctly choose and use the appropriate API.

Regex Matching

This proposal is intended to be a minimum set of functionality to control the environment. It requires very explicit control - aside from --env-clear, all variables must be explicitly listed to remove or set their values.

A possible extension would be to allow regular expressions to select which variable names should be removed from the logical environment or passed through from the process environment. For example, --env-remove-re or --env-pass-re.