Add mechanism to propagate arbitrary environment variables

[Eric-Arellano]

As demonstrated in #10282, it's not very feasible for us to anticipate which env variables users will or will not need, as we currently do, for example, with the option --python-native-code-cpp-flags.

Instead, we need to allow users to pass arbitrary env variables to processes.

Some key questions:

1. Should this be global and every single Process gets these values, or should we try to keep it more precise? For example, should all Python related Processes have it via [python-setup], or be even more granular that only Pytest processes have it via a [pytest] option?
2. In TOML, there is no way to say os.getenv() to get the current value of an env var; you would need to explicitly set the value. Is this fine, or do we need a mechanism to read from the greater environment?

[stuhood]

Relates to #9760 and #7735, probably.

[ynouri]

Very interested in that feature. We are trying to run an integration test against AWS, which requires passing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION...

In my mind it's quite similar to Tox's PASSENV feature: https://tox.readthedocs.io/en/latest/config.html#conf-passenv

[Eric-Arellano]

Hey @ynouri, that use case is supported for tests: https://www.pantsbuild.org/docs/python-test-goal#setting-environment-variables

This issue's proposal is more general, like being able to set env vars for a codegen process running Protoc.

Let us know if that works out or any way we can help y'all out! If you're not yet on it, we're a friendly bunch on Slack: https://www.pantsbuild.org/docs/community.

[ynouri]

Wonderful, thanks a lot Eric! Next time I'll spend more time reading the docs 😄

If that can be of interest to future readers, we are running Pants just after the Github Action aws-actions/configure-aws-credentials. As explained in the Pants documentation, adding extra_env_vars in pants.toml made the trick:

[test]
extra_env_vars = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN", "AWS_DEFAULT_REGION", "AWS_REGION"]

[stuhood]

The PantsEnvironment SessionValue is a very coarse-grained, not-filtered way to get these in a @rule. Using it correctly right now (ie, not depending on the entire environment) is challenging, but I think that I see how to hide that implementation detail and make it a bit easier to use. Will do as part of #7654.

[stuhood]

This issue is essentially resolved. The pattern is for a particular tool or use case to declare an --env-vars option on its subsystem (for example: docker's) which allows for including additional env vars in particular cases (either by passing them through, or by setting them explicitly).

Under the hood, this uses environment variable filtering, so the relevant @rules will only re-run if the matched env vars change (rather than if "any" env vars change).