Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mechanism to propagate arbitrary environment variables #10352

Closed
Eric-Arellano opened this issue Jul 14, 2020 · 6 comments
Closed

Add mechanism to propagate arbitrary environment variables #10352

Eric-Arellano opened this issue Jul 14, 2020 · 6 comments

Comments

@Eric-Arellano
Copy link
Contributor

As demonstrated in #10282, it's not very feasible for us to anticipate which env variables users will or will not need, as we currently do, for example, with the option --python-native-code-cpp-flags.

Instead, we need to allow users to pass arbitrary env variables to processes.

Some key questions:

  1. Should this be global and every single Process gets these values, or should we try to keep it more precise? For example, should all Python related Processes have it via [python-setup], or be even more granular that only Pytest processes have it via a [pytest] option?
  2. In TOML, there is no way to say os.getenv() to get the current value of an env var; you would need to explicitly set the value. Is this fine, or do we need a mechanism to read from the greater environment?
@stuhood
Copy link
Member

stuhood commented Jul 14, 2020

Relates to #9760 and #7735, probably.

@benjyw benjyw added this to the 2.0.x milestone Jul 16, 2020
@stuhood stuhood modified the milestones: 2.0.x, 2.0.0rc0 Sep 9, 2020
@benjyw benjyw removed this from the 2.0.0rc0 milestone Sep 9, 2020
@ynouri
Copy link

ynouri commented Feb 25, 2021

Very interested in that feature. We are trying to run an integration test against AWS, which requires passing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION...

In my mind it's quite similar to Tox's PASSENV feature: https://tox.readthedocs.io/en/latest/config.html#conf-passenv

@Eric-Arellano
Copy link
Contributor Author

Hey @ynouri, that use case is supported for tests: https://www.pantsbuild.org/docs/python-test-goal#setting-environment-variables

This issue's proposal is more general, like being able to set env vars for a codegen process running Protoc.

Let us know if that works out or any way we can help y'all out! If you're not yet on it, we're a friendly bunch on Slack: https://www.pantsbuild.org/docs/community.

@ynouri
Copy link

ynouri commented Feb 25, 2021

Wonderful, thanks a lot Eric! Next time I'll spend more time reading the docs 😄

If that can be of interest to future readers, we are running Pants just after the Github Action aws-actions/configure-aws-credentials. As explained in the Pants documentation, adding extra_env_vars in pants.toml made the trick:

[test]
extra_env_vars = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN", "AWS_DEFAULT_REGION", "AWS_REGION"]

@stuhood
Copy link
Member

stuhood commented Mar 5, 2021

The PantsEnvironment SessionValue is a very coarse-grained, not-filtered way to get these in a @rule. Using it correctly right now (ie, not depending on the entire environment) is challenging, but I think that I see how to hide that implementation detail and make it a bit easier to use. Will do as part of #7654.

@stuhood
Copy link
Member

stuhood commented Aug 18, 2022

This issue is essentially resolved. The pattern is for a particular tool or use case to declare an --env-vars option on its subsystem (for example: docker's) which allows for including additional env vars in particular cases (either by passing them through, or by setting them explicitly).

Under the hood, this uses environment variable filtering, so the relevant @rules will only re-run if the matched env vars change (rather than if "any" env vars change).

@stuhood stuhood closed this as completed Aug 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants