From a4fe00849e3d840dd66874f4167f12ef42442a45 Mon Sep 17 00:00:00 2001 From: Stu Hood Date: Mon, 3 Oct 2022 16:08:54 -0700 Subject: [PATCH 1/3] Initial draft of `environments.md`. [ci skip-rust] [ci skip-build-wheels] --- docs/markdown/Using Pants/environments.md | 221 ++++++++++++++++++ .../Using Pants/remote-caching-execution.md | 2 +- .../remote-execution.md | 37 ++- docs/markdown/Using Pants/troubleshooting.md | 2 +- .../common-plugin-tasks/plugins-repl-goal.md | 2 +- .../common-plugin-tasks/plugins-run-goal.md | 2 +- 6 files changed, 241 insertions(+), 25 deletions(-) create mode 100644 docs/markdown/Using Pants/environments.md diff --git a/docs/markdown/Using Pants/environments.md b/docs/markdown/Using Pants/environments.md new file mode 100644 index 00000000000..35dab46a1bd --- /dev/null +++ b/docs/markdown/Using Pants/environments.md @@ -0,0 +1,221 @@ +--- +title: "Environments: Cross-Platform or Remote Builds" +slug: "environments" +hidden: false +createdAt: "2022-10-03T21:39:51.235Z" +updatedAt: "2022-10-03T21:39:51.235Z" +--- +Environments +============ + +By default, Pants will execute all sandboxed build work directly on localhost. But defining and using additional "environments" for particular targets allows Pants to transparently execute some or all of your build either: +1. locally in Docker containers +2. remotely via [remote execution](doc:remote-execution) +3. locally, but with a non-default set of environment variables and settings (such as for cross-building) + +## Defining environments + +Environments are defined using environment targets: + +* [`local_environment`](doc:reference-local_environment) - Runs without containerization on localhost (which is also the default if no environment targets are defined). +* [`docker_environment`](doc:reference-docker_environment) - Runs in a cached container using the specified Docker image. +* [`remote_environment`](doc:reference-remote_environment) - Runs in a remote worker via [remote execution](doc:remote-execution) (possibly with containerization, depending on the server implementation). + +Environment targets are given short, descriptive names using the [`[environments.names]` option](doc:reference-environments#names) (usually defined in `pants.toml`), which consuming targets use to refer to them in `BUILD` files. That might look like a `pants.toml` section and `BUILD` file (at the root of the repository in this case) containing: + +```toml +[environments.names] +linux = "//:local_linux" +linux_docker = "//:local_busybox" +``` + +```python +local_environment( + name="local_linux", + compatible_platforms=["linux_x86_64"], + fallback_environment="local_busybox", + .. +) + +docker_environment( + name="local_busybox", + platform="linux_x86_64", + image="busybox:latest@sha256-abcd123...", + .. +) +``` + +### Environment-aware options + +Environment targets have fields (target arguments) which correspond to options which are marked "environment-aware". When an option is environment-aware, the value of the option that will be used in an environment can be overridden by setting the corresponding field value on the environment target for that environment. If an environment target does not set a value, it defaults to the value which is set globally via options values. + +For example, the [`[python-bootstrap].search_path` option](doc:reference-python-bootstrap#search_path) is environment-aware, which is indicated in its help. It can be overridden for a particular environment by a corresponding environment target field, such as [the one on `local_environment`](doc:reference-local_environment#codepython_bootstrap_search_pathcode). + +> 👍 See an option which should be environment-aware, but isn't? +> +> Environments are a new concept: if you see an option value which should be marked environment-aware but isn't, please definitely [file an issue](https://github.com/pantsbuild/pants/issues/new/choose)! + +## Consuming environments + +To declare which environment they should build with, many target types (but particularly "root" targets like tests or binaries) have an `environment=` field: for example, [`python_tests(environment=..)`](doc:reference-python_tests#codeenvironmentcode). + +The `environment=` field may either: +1. refer to an environment by name +2. use a special `__local__` environment name, which resolves to any matching `local_environment` (see "Environment matching" below) + +Test targets additionally have a `runtime_environment=` field (_TODO: see workflow example below, and implement_) which defaults to the value of the target's `environment=` field, but which can be set explicitly to indicate that a test should execute in a different environment than it was built in. This can be used to enable cross-building (where a test is built on one platform, but executed on another), or to explicitly provide tools or running services at test runtime which would not otherwise be available. + +> 🚧 Environment compatibility +> +> Currently, there is no static validation that a target's environment is compatible with its dependencies' environments -- only the implicit validation of the goals that you run successfully against those targets (`check`, `lint`, `test`, `package`, etc). +> +> As we gain more experience with how environments are used in the wild, it's possible that more static validation can be added: your feedback would be very welcome! + +### Setting the environment on many targets at once + +To use an environment everywhere in your repository (or only within a particular subdirectory, or with a particular target +type), you can use the [`__defaults__` builtin](doc:targets#field-default-values). For example, to use an environment named `my_default_environment` globally by default, you would add the following to a `BUILD` file at the root of the repository: +```python +__defaults__(all=dict(environment="my_default_environment")) +``` +... and individual targets could override the default as needed. + +### Environment matching + +A single environment name may end up referring to different environment targets on different physical machines, or with different global settings applied: this is known as environment "matching". + +* `local_environment` targets will match if their `compatible_platforms=` field matches localhost's platform. +* `docker_environment` targets will match [if Docker is enabled](doc:reference-global#docker_execution), and if their `platform=` field is compatible with localhost's platform. +* `remote_environment` targets will match [if Remote execution is enabled](doc:reference-global#remote_execution). + +It a particular environment target _doesn't_ match, it can configure a `fallback_environment=` which will be attempted next. This allows for forming preference chains which are referred to by whichever environment name is at the head of the chain. + +For example: a chain like "prefer remote execution if enabled, but fall back to local execution if the platform matches, otherwise use docker" might be configured via the targets: +```python +remote_environment( + name="remote", + fallback_environment="local", + .. +) + +local_environment( + name="local", + compatible_platforms=["linux_x86_64"], + fallback_environment="docker", +) + +docker_environment( + name="docker", + .. +) +``` + +In future versions, environment targets will gain additional predicates to control whether they match (for example: `local_environment` will likely gain a predicate that looks for the [presence or value of an environment variable](https://github.com/pantsbuild/pants/issues/17107). But in the meantime, it's possible to override which environments are matched for particular use cases by overriding their configured names: see the "Toggle use of an environment" workflow below for an example. + +## Example workflows + +### Enabling remote execution globally + +`remote_environment` targets match unless the [`--remote-execution`](doc:reference-global#remote_execution) option is disabled. So to cause a particular environment name to use remote execution whenever it is enabled, you could define environment targets which tried remote execution first, and then fell back to local execution: + +```python +remote_environment( + name="remote_busybox", + platform="linux_x86_64", + extra_platform_properties={"container-image=busybox:latest"}, + fallback_environment="local", +) + +local_environment( + name="local", + compatible_platforms=[...], +) +``` + +You'd then give your `remote_environment` target an unassuming name like "default": +```toml +[environments.names] +default = "//:remote_busybox" +local = "//:local" +``` +... and use that environment by default with all targets. Users or consumers like CI could then toggle whether remote execution was used by setting `--remote-execution`. + +> 🚧 Speculation of remote execution +> +> The `2.15.x` series of Pants does not yet support ["speculating" remote execution](https://github.com/pantsbuild/pants/issues/8353) by racing it against another environment (usually local or docker). While we expect that this will be necessary to make remote execution a viable option for local execution on user's laptops (where network connections are less reliable), it is less critical for CI use-cases. + +### Use a `docker_environment` to build the inputs to a `docker_image` + +To build a `docker_image` target containing a `pex_binary` which uses native (i.e. compiled) dependencies on a `macOS` machine, you can configure the `pex_binary` to be built in a `docker_environment`. + +You'll need a `docker_environment` which uses an image containing the relevant build-time requirements of your PEX. At a minimum, you'll need Python itself: +```python +docker_environment( + name="python_bullseye", + platform="linux_x86_64", + image="python:3.9.14-slim-bullseye@sha256-abcd123...", + .. +) +``` + +Next, mark your `pex_binary` target with this environment (with the name `python_bullseye`: see "Defining environments" above), and define a `docker_image` target depending on it. + +```python +pex_binary( + name="main", + environment="python_bullseye", +) + +docker_image( + name="docker_image", + instructions=[ + "FROM python:3.9.14-slim-bullseye@sha256-abcd123...", + "ENTRYPOINT ["/main"]", + "COPY examples/main.pex /main", + ], +) +``` + +> 👍 Compatibility of `docker_environment` and `docker_image` +> +> Note that the Docker image used in your `docker_environment` does not need to match the base image of the `docker_image` targets that consume them: they only need to be compatible. This is because execution of build steps in a `docker_environment` occurs in an anonymous container, and only the required inputs are provided to the `docker_image` build. +> +> This means that your `docker_environment` can include things like compilers or other tools relevant to your build, without needing to manually use multi-stage Docker builds. + +### Execute a test in Docker, while natively cross-building it + +_TODO: Like https://github.com/pantsbuild/pants/issues/15764, but for tests._ + +_TODO: Give an example of using `environment=` vs `runtime_environment=` to use docker only for test execution, but not for building of thirdparty dependencies by using PEX to cross-build. This will require exposing options out of PEX to override (?) the target platform, and figuring out how that doesn't break `check` is an open question._ + +### Toggle use of an environment for some consumers + +As mentioned above in "Environment matching", environment targets "match" based on their field values and global options. But if two environment targets would be ambiguous in some cases, or if you'd otherwise like to control what a particular environment name means (in CI, for example), you can override an environment name via options. + +For example: if you'd like to use a particular `macOS` environment target locally, but override it for a particular use case in CI, you'd start by defining two `local_environment` targets which would usually match ambiguously: + +```python +local_environment( + name="macos_laptop", + compatible_platforms=["macos_x86_64"], +) + +local_environment( + name="macos_ci", + compatible_platforms=["macos_x86_64"], +) +``` + +... and then assign one of them a (generic) environment name in `pants.toml`: +```toml +[environments.names] +macos = "//:macos_laptop" +... +``` + +You could then _override_ that name definition in `pants.ci.toml` (note the use of the `.add` suffix, in order to preserve any other named environments): +```toml +[environments.names.add] +macos = "//:macos_ci" +``` + diff --git a/docs/markdown/Using Pants/remote-caching-execution.md b/docs/markdown/Using Pants/remote-caching-execution.md index bd99f30d38e..a9764aa0f51 100644 --- a/docs/markdown/Using Pants/remote-caching-execution.md +++ b/docs/markdown/Using Pants/remote-caching-execution.md @@ -8,7 +8,7 @@ updatedAt: "2021-03-19T21:39:51.235Z" Overview ======== -Ordinarily, Pants executes processes locally on the system on which it is run and also caches the results of those processes locally as well. Besides this "local execution" mode of operation, Pants also supports two distributed modes of operation: +By default, Pants executes processes in a local [environment](doc:environments) on the system on which it is run, and caches the results of those processes locally as well. Besides this "local execution" mode of operation, Pants also supports two distributed modes of operation: 1. "Remote caching" where Pants store results from local process execution in a remote cache and also consumes results from that remote cache; and diff --git a/docs/markdown/Using Pants/remote-caching-execution/remote-execution.md b/docs/markdown/Using Pants/remote-caching-execution/remote-execution.md index fb3fdde5e18..fcfff065076 100644 --- a/docs/markdown/Using Pants/remote-caching-execution/remote-execution.md +++ b/docs/markdown/Using Pants/remote-caching-execution/remote-execution.md @@ -37,23 +37,25 @@ remote_execution_address = "grpc://build.corp.example.com:8980" remote_instance_name = "main" ``` -### Platform Properties +### Environment-specific settings -The REAPI execution service selects a worker for a process by consulting the "platform properties" that are passed in a remote execution request. These platform properties are key/value pairs that are configured in the server. Generally, you will configure these in the server (or be provided them by your server's administrator), and then configure Pants to use what was configured. +The REAPI execution service selects a worker for a process by consulting the "platform properties" that are passed in a remote execution request. These platform properties are key/value pairs that are configured for particular workers in the server. Generally, you will configure these in the server (or be provided them by your server's administrator), and then configure Pants to match particular workers using their relevant platform properties. -Assume that the REAPI server is configured with `OSFamily=linux` as the only platform properties. Then building on the first example earlier, add the `remote_execution_extra_platform_properties` to `pants.toml`: +To define platform properties (as well as to configure any other settings which are specific to running on a remote worker), you should define a remote environment. Building on the first example earlier, you would add [`remote_environment` targets](doc:reference-remote_environment) (see [environment](doc:environments) for more information) corresponding to each set of distinct workers you want to use in the server. Assuming that the REAPI server is configured with a particular worker type labeled `docker-container=busybox:latest`, that might look like a `BUILD` file containing: -```toml -[GLOBAL] -remote_execution = true -remote_store_address = "grpc://build.corp.example.com:8980" -remote_execution_address = "grpc://build.corp.example.com:8980" -remote_instance_name = "main" -remote_execution_extra_platform_properties = [ - "OSFamily=linux", -] +```python +remote_environment( + name="remote_busybox", + platform="linux_x86_64", + extra_platform_properties = [ + "docker-container=busybox:latest", + ], + .. +) ``` +Your `remote_environment` will also need to override any [environment-aware options](doc:environments) which configure the relevant tools used in your repository. For example: if building Python code, a Python interpreter must be available and matched via the environment-aware options of `[python-bootstrap]`. If using protobuf support, then you may also need `unzip` available in the remote execution environment in order to unpack the protoc archive. Etc. + ### Concurrency Finally, you should configure Pants to limit the number of concurrent execution requests that are sent to the REAPI server. The `process_execution_remote_parallelism` option controls this concurrency. For example, if `process_execution_remote_parallelism` is set to `20`, then Pants will only send a maximum of 20 execution requests at a single moment of time. @@ -95,13 +97,6 @@ remote_ca_certs_path = "/etc/ssl/certs/ca-certificates.crt" Reference ========= -Run `./pants help-advanced global` or refer to [Global options](doc:reference-global). Most remote execution and caching options begin with the prefix `--remote`. - -Limitations -=========== - -The remote execution support in Pants is still experimental and comes with several limitations: - -1. The main limitation is that Pants assumes that the remote execution platform is the same as the local platform. Thus, if the remote execution service is running on Linux, then Pants must also be running on Linux in order to successfully submit remote execution requests. This limitation will eventually be fixed, but as of version 2.0.x, Pants still has the limitation. +For global options, run `./pants help-advanced global` or refer to [Global options](doc:reference-global). Most remote execution and caching options begin with the prefix `--remote`. -2. The remote execution environment will need to contain appropriate tooling expected by the Pants subsystems used in your repository. At a minimum, this means a Python interpreter must be available if building Python code. If using protobuf support, then you may also need `unzip` available in the remote execution environment in order to unpack the protoc archive. This documentation is incomplete with regards to what tooling needs to be available. +For environment-specific options, see `./pants help-advanced remote_environment` or the [`remote_environment` target](doc:reference-remote_environment). diff --git a/docs/markdown/Using Pants/troubleshooting.md b/docs/markdown/Using Pants/troubleshooting.md index 91553ca91b9..b552edeabb3 100644 --- a/docs/markdown/Using Pants/troubleshooting.md +++ b/docs/markdown/Using Pants/troubleshooting.md @@ -43,7 +43,7 @@ Use the option `--keep-sandboxes=always` for Pants to log the paths to these san You can also pass `--keep-sandboxes=on_failure`, to preserve only the sandboxes of failing processes. -There is even a `__run.sh` script in the directory that will run the process using the same argv and environment that Pants would use. +There is even a `__run.sh` script in the directory that will run the process using the same argv and environment variables that Pants would use. Cache or pantsd invalidation issues ----------------------------------- diff --git a/docs/markdown/Writing Plugins/common-plugin-tasks/plugins-repl-goal.md b/docs/markdown/Writing Plugins/common-plugin-tasks/plugins-repl-goal.md index 86a7497759b..a6e2bcb22ef 100644 --- a/docs/markdown/Writing Plugins/common-plugin-tasks/plugins-repl-goal.md +++ b/docs/markdown/Writing Plugins/common-plugin-tasks/plugins-repl-goal.md @@ -57,7 +57,7 @@ The `ReplRequest ` will get converted into an `InteractiveProcess` that will run The process will run in a temporary directory in the build root, which means that the script/program can access files that would normally need to be declared by adding a `file` / `files` or `resource` / `resources` target to the `dependencies` field. -The process's environment will not be hermetic, meaning that it will inherit the environment used by the `./pants process`. Any values you set in `extra_env` will add or update the specified environment variables. +The process will not be hermetic, meaning that it will inherit the environment variables used by the `./pants` process. Any values you set in `extra_env` will add or update the specified environment variables. ```python from dataclasses import dataclass diff --git a/docs/markdown/Writing Plugins/common-plugin-tasks/plugins-run-goal.md b/docs/markdown/Writing Plugins/common-plugin-tasks/plugins-run-goal.md index b79ec1d5477..384ba73cbf2 100644 --- a/docs/markdown/Writing Plugins/common-plugin-tasks/plugins-run-goal.md +++ b/docs/markdown/Writing Plugins/common-plugin-tasks/plugins-run-goal.md @@ -85,7 +85,7 @@ The `RunRequest` will get converted into an `InteractiveProcess` that will run i The process will run in a temporary directory in the build root, which means that the script/program can access files that would normally need to be declared by adding a `files` or `resources` target to the `dependencies` field. -The process's environment will not be hermetic, meaning that it will inherit the environment used by the `./pants process`. Any values you set in `extra_env` will add or update the specified environment variables. +The process will not be hermetic, meaning that it will inherit the environment variables used by the `./pants` process. Any values you set in `extra_env` will add or update the specified environment variables. ```python from dataclasses import dataclass From 1a590795ddbe7323618adabdff8a37e3464a036e Mon Sep 17 00:00:00 2001 From: Stu Hood Date: Tue, 11 Oct 2022 10:59:58 -0700 Subject: [PATCH 2/3] Review feedback. # Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels] --- docs/markdown/Using Pants/environments.md | 54 ++++++++++++++--------- 1 file changed, 32 insertions(+), 22 deletions(-) diff --git a/docs/markdown/Using Pants/environments.md b/docs/markdown/Using Pants/environments.md index 35dab46a1bd..6c39ba8e3fa 100644 --- a/docs/markdown/Using Pants/environments.md +++ b/docs/markdown/Using Pants/environments.md @@ -11,25 +11,24 @@ Environments By default, Pants will execute all sandboxed build work directly on localhost. But defining and using additional "environments" for particular targets allows Pants to transparently execute some or all of your build either: 1. locally in Docker containers 2. remotely via [remote execution](doc:remote-execution) -3. locally, but with a non-default set of environment variables and settings (such as for cross-building) +3. locally, but with a non-default set of environment variables and settings (such as when different platforms need different values, or when cross-building) ## Defining environments Environments are defined using environment targets: * [`local_environment`](doc:reference-local_environment) - Runs without containerization on localhost (which is also the default if no environment targets are defined). -* [`docker_environment`](doc:reference-docker_environment) - Runs in a cached container using the specified Docker image. +* [`docker_environment`](doc:reference-docker_environment) - Runs in a cached container using the specified Docker image using a local installation of Docker. If the image does not already exist locally, it will be pulled. * [`remote_environment`](doc:reference-remote_environment) - Runs in a remote worker via [remote execution](doc:remote-execution) (possibly with containerization, depending on the server implementation). -Environment targets are given short, descriptive names using the [`[environments.names]` option](doc:reference-environments#names) (usually defined in `pants.toml`), which consuming targets use to refer to them in `BUILD` files. That might look like a `pants.toml` section and `BUILD` file (at the root of the repository in this case) containing: +Give your environment targets short, descriptive names using the [`[environments-previews.names]` option](doc:reference-environments-preview#names) (usually defined in `pants.toml`), which consuming targets use to refer to them in `BUILD` files. That might look like a `pants.toml` section and `BUILD` file (at the root of the repository in this case) containing: -```toml -[environments.names] +```toml pants.toml +[environments-preview.names] linux = "//:local_linux" linux_docker = "//:local_busybox" ``` - -```python +```python BUILD local_environment( name="local_linux", compatible_platforms=["linux_x86_64"], @@ -47,7 +46,7 @@ docker_environment( ### Environment-aware options -Environment targets have fields (target arguments) which correspond to options which are marked "environment-aware". When an option is environment-aware, the value of the option that will be used in an environment can be overridden by setting the corresponding field value on the environment target for that environment. If an environment target does not set a value, it defaults to the value which is set globally via options values. +Environment targets have fields ([target](doc:targets) arguments) which correspond to [options](doc:options) which are marked "environment-aware". When an option is environment-aware, the value of the option that will be used in an environment can be overridden by setting the corresponding field value on the associated environment target. If an environment target does not set a value, it defaults to the value which is set globally via options values. For example, the [`[python-bootstrap].search_path` option](doc:reference-python-bootstrap#search_path) is environment-aware, which is indicated in its help. It can be overridden for a particular environment by a corresponding environment target field, such as [the one on `local_environment`](doc:reference-local_environment#codepython_bootstrap_search_pathcode). @@ -75,11 +74,22 @@ Test targets additionally have a `runtime_environment=` field (_TODO: see workfl To use an environment everywhere in your repository (or only within a particular subdirectory, or with a particular target type), you can use the [`__defaults__` builtin](doc:targets#field-default-values). For example, to use an environment named `my_default_environment` globally by default, you would add the following to a `BUILD` file at the root of the repository: -```python +```python BUILD __defaults__(all=dict(environment="my_default_environment")) ``` ... and individual targets could override the default as needed. +### Building one target in multiple environments + +If a target will always need to be built in multiple environments (rather than conditionally based on which user is building it: see the "Toggle use of an environment for some consumers" section), then you can use the [`parametrize` builtin](doc:targets#parametrizing-targets) for the `environment=` field. If you had two environments named `linux` and `macos`, that would look like: + +```python BUILD +pex_binary( + name="bin", + environment=parametrize("linux", "macos"), +) +``` + ### Environment matching A single environment name may end up referring to different environment targets on different physical machines, or with different global settings applied: this is known as environment "matching". @@ -91,7 +101,7 @@ A single environment name may end up referring to different environment targets It a particular environment target _doesn't_ match, it can configure a `fallback_environment=` which will be attempted next. This allows for forming preference chains which are referred to by whichever environment name is at the head of the chain. For example: a chain like "prefer remote execution if enabled, but fall back to local execution if the platform matches, otherwise use docker" might be configured via the targets: -```python +```python BUILD remote_environment( name="remote", fallback_environment="local", @@ -116,9 +126,9 @@ In future versions, environment targets will gain additional predicates to contr ### Enabling remote execution globally -`remote_environment` targets match unless the [`--remote-execution`](doc:reference-global#remote_execution) option is disabled. So to cause a particular environment name to use remote execution whenever it is enabled, you could define environment targets which tried remote execution first, and then fell back to local execution: +`remote_environment` targets match unless the [`--remote-execution`](doc:reference-global#remote_execution) option is disabled. So to cause a particular environment name to use remote execution whenever it is enabled, you could define environment targets which try remote execution first, and then fall back to local execution: -```python +```python BUILD remote_environment( name="remote_busybox", platform="linux_x86_64", @@ -133,12 +143,12 @@ local_environment( ``` You'd then give your `remote_environment` target an unassuming name like "default": -```toml -[environments.names] +```toml pants.toml +[environments-preview.names] default = "//:remote_busybox" local = "//:local" ``` -... and use that environment by default with all targets. Users or consumers like CI could then toggle whether remote execution was used by setting `--remote-execution`. +... and use that environment by default with all targets. Users or consumers like CI could then toggle whether remote execution is used by setting `--remote-execution`. > 🚧 Speculation of remote execution > @@ -149,7 +159,7 @@ local = "//:local" To build a `docker_image` target containing a `pex_binary` which uses native (i.e. compiled) dependencies on a `macOS` machine, you can configure the `pex_binary` to be built in a `docker_environment`. You'll need a `docker_environment` which uses an image containing the relevant build-time requirements of your PEX. At a minimum, you'll need Python itself: -```python +```python BUILD docker_environment( name="python_bullseye", platform="linux_x86_64", @@ -160,7 +170,7 @@ docker_environment( Next, mark your `pex_binary` target with this environment (with the name `python_bullseye`: see "Defining environments" above), and define a `docker_image` target depending on it. -```python +```python BUILD pex_binary( name="main", environment="python_bullseye", @@ -194,7 +204,7 @@ As mentioned above in "Environment matching", environment targets "match" based For example: if you'd like to use a particular `macOS` environment target locally, but override it for a particular use case in CI, you'd start by defining two `local_environment` targets which would usually match ambiguously: -```python +```python BUILD local_environment( name="macos_laptop", compatible_platforms=["macos_x86_64"], @@ -207,15 +217,15 @@ local_environment( ``` ... and then assign one of them a (generic) environment name in `pants.toml`: -```toml -[environments.names] +```toml pants.toml +[environments-preview.names] macos = "//:macos_laptop" ... ``` You could then _override_ that name definition in `pants.ci.toml` (note the use of the `.add` suffix, in order to preserve any other named environments): -```toml -[environments.names.add] +```toml pants.ci.toml +[environments-preview.names.add] macos = "//:macos_ci" ``` From ff9a75ba62c14f7935e6be2982c7e3327fcdee38 Mon Sep 17 00:00:00 2001 From: Stu Hood Date: Thu, 27 Oct 2022 15:59:53 -0700 Subject: [PATCH 3/3] Move TODOs out to the tracking issue, and reference it in the header. --- docs/markdown/Using Pants/environments.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/docs/markdown/Using Pants/environments.md b/docs/markdown/Using Pants/environments.md index 6c39ba8e3fa..35e0400a6f8 100644 --- a/docs/markdown/Using Pants/environments.md +++ b/docs/markdown/Using Pants/environments.md @@ -8,6 +8,10 @@ updatedAt: "2022-10-03T21:39:51.235Z" Environments ============ +> 🚧 Environments are currently in `preview`, and have not yet stabilized. +> +> We'd love your feedback on how Environments could be most useful to you! Please refer to [the tracking issue](https://github.com/pantsbuild/pants/issues/17355) for known stabilization blockers. + By default, Pants will execute all sandboxed build work directly on localhost. But defining and using additional "environments" for particular targets allows Pants to transparently execute some or all of your build either: 1. locally in Docker containers 2. remotely via [remote execution](doc:remote-execution) @@ -62,8 +66,6 @@ The `environment=` field may either: 1. refer to an environment by name 2. use a special `__local__` environment name, which resolves to any matching `local_environment` (see "Environment matching" below) -Test targets additionally have a `runtime_environment=` field (_TODO: see workflow example below, and implement_) which defaults to the value of the target's `environment=` field, but which can be set explicitly to indicate that a test should execute in a different environment than it was built in. This can be used to enable cross-building (where a test is built on one platform, but executed on another), or to explicitly provide tools or running services at test runtime which would not otherwise be available. - > 🚧 Environment compatibility > > Currently, there is no static validation that a target's environment is compatible with its dependencies' environments -- only the implicit validation of the goals that you run successfully against those targets (`check`, `lint`, `test`, `package`, etc). @@ -98,7 +100,7 @@ A single environment name may end up referring to different environment targets * `docker_environment` targets will match [if Docker is enabled](doc:reference-global#docker_execution), and if their `platform=` field is compatible with localhost's platform. * `remote_environment` targets will match [if Remote execution is enabled](doc:reference-global#remote_execution). -It a particular environment target _doesn't_ match, it can configure a `fallback_environment=` which will be attempted next. This allows for forming preference chains which are referred to by whichever environment name is at the head of the chain. +If a particular environment target _doesn't_ match, it can configure a `fallback_environment=` which will be attempted next. This allows for forming preference chains which are referred to by whichever environment name is at the head of the chain. For example: a chain like "prefer remote execution if enabled, but fall back to local execution if the platform matches, otherwise use docker" might be configured via the targets: ```python BUILD @@ -192,12 +194,6 @@ docker_image( > > This means that your `docker_environment` can include things like compilers or other tools relevant to your build, without needing to manually use multi-stage Docker builds. -### Execute a test in Docker, while natively cross-building it - -_TODO: Like https://github.com/pantsbuild/pants/issues/15764, but for tests._ - -_TODO: Give an example of using `environment=` vs `runtime_environment=` to use docker only for test execution, but not for building of thirdparty dependencies by using PEX to cross-build. This will require exposing options out of PEX to override (?) the target platform, and figuring out how that doesn't break `check` is an open question._ - ### Toggle use of an environment for some consumers As mentioned above in "Environment matching", environment targets "match" based on their field values and global options. But if two environment targets would be ambiguous in some cases, or if you'd otherwise like to control what a particular environment name means (in CI, for example), you can override an environment name via options.