Allow CI related env variables in pass_env

[abelcheung]

What's the problem this feature will solve?

This can help users tailor make tox envs inside CI environment via different settings etc. I believe using tox in GitHub Actions, GitLab pipelines etc isn't so exotic. Giving access to a few extra os env variables should be able to help them a little bit.

For example, hypothesis has routine to tweak a few settings by default when running inside CI to help alleviate frequent test failures due to tighter CI resources (compare with desktop machines or local servers).

But the tweak isn't not enabled when running tests via tox. Every tox users would need to add the variable themselves in tox config if they discover the issue, or modify test settings manually if they don't.

Describe the solution you'd like

There are at least 2 env variable that fits this purpose.

• CI: For GitHub, GitLab, Travis CI, Appveyor etc
• TF_BUILD: For Azure
• Some vendors have specific os variables preset, like GITHUB_ACTIONS, GITLAB_CI etc.

[gaborbernat]

I'm not sure what's the request here. You can do this today by passing these inside your tox.ini.

[abelcheung]

Yes, I have indicate it can be added manually inside people's own tox config, but then everybody has to add that in order to fix something that should have worked out-of-box. And by extension of logic, all other vars like HOME, PATH etc shouldn't have been passed by tox by default, because people can add the var themselves.

[gaborbernat]

all other vars like HOME, PATH

We only pass through environment variables that are required for the Python interpreter (or standard library) itself to function. Those variables have not been picked out of thin air.

Application specific environment variables, such as using GItHub or GitLab should be encoded into the libraries/apps tox configuration. These are tied to the app/library, not the user.

[Liam-DeVoe]

I'm sympathetic to this position and I understand why you make the distinction between CI and HOME. At the same time, this makes it impossible for a library like Hypothesis to provide good defaults when running in CI, because tox strips CI and related variables.

Is it possible you would reconsider this? Or is there any way at all for a library like Hypothesis to see the original environment variables, so we can hardcode a special case for tox? Perhaps a new envvar TOX_ORIGINAL_ENVVARS which is a simple stringified dictionary of all envvars at the time tox launched?

(See also HypothesisWorks/hypothesis#4481.)

[gaborbernat]

Or is there any way at all for a library like Hypothesis to see the original environment variables, so we can hardcode a special case for tox

This would defeat the purpose of that separation, if the users can just work around it. Users can pass on these environment variables in their config if they want to use this.

At the same time, this makes it impossible for a library like Hypothesis to provide good defaults when running in C

Why do you need this? Making a separation between local and CI invites users to run into issues of runs locally but not on CI.

[Liam-DeVoe]

Why do you need this? Making a separation between local and CI invites users to run into issues of runs locally but not on CI.

Unfortunately this is a fundamental issue of property-based testing, just in the other direction. PBT is a randomized testing method, so every time you run your test, Hypothesis generates hundreds of random inputs. This means you may commit a test that passes when run once on your machine, but fails when run hundreds of times in CI.

Many Hypothesis users have previously commented that this is confusing and undesirable. So we made Hypothesis automatically run deterministically in CI, so that CI uses Hypothesis as a regression test by default, and if it passes once, it will always pass in the future. This blog post describes the distinction between these two styles of testing, and why you might want one or the other.

TLDR Hypothesis is randomized, so there's already the risk that a test will fail locally but not on CI, or vice versa. Letting Hypothesis hook into the CI variable actually makes this more deterministic and coincident with local use, not less.

This would defeat the purpose of that separation, if the users can just work around it.

IMO this would not be a particularly large problem if it were put into a clearly "private and for library use only" variable like __TOX_ORIGINAL_ENVVARS, which standard users should not touch.

[gaborbernat]

IMO this would not be a particularly large problem if it were put into a clearly "private and for library use only" v

There's no way to check that users are using it correctly. Can you just tell users that if you are using hypothesis set environment variables A, B, C in your test suites pass_env?

[Liam-DeVoe]

yes, but I would very strongly prefer if the defaults worked correctly out of the box, even if it means Hypothesis (the library) has to go to great lengths to do so! Users who hit an error in CI but not local aren't going to know that Hypothesis actually protects against this by default, and won't know where in the docs to look to understand they need to set passenv.

Another reason why we want CI to differ from local: Hypothesis raises HealthCheck.too_slow when input generation is very slow. CI machines have unreliable timings (due to cpu steal etc), so we disable this health check in CI to avoid annoying users. Of course this requires the CI variable and so doesn't work under tox atm.

There's no way to check that users are using it correctly

IMO this goes against "we're all consenting adults here", if marked clearly as private.

[gaborbernat]

yes, but I would very strongly prefer if the defaults worked correctly out of the box, even if it means Hypothesis (the library) has to go to great lengths to do so! Users who hit an error in CI but not local aren't going to know that Hypothesis actually protects against this by default, and won't know where in the docs to look to understand they need to set passenv.

Ok, so based on this I'd be ok to pass through CI as __TOX_ENVIRONMENT_VARIABLE_ORIGINAL_CI. The naming is deliberately long to make it not not pleasent to use by users. A PR doing this would be accepted.

[Liam-DeVoe]

I will send a PR in the near future!

[gaborbernat]

• CI: For GitHub, GitLab, Travis CI, Appveyor etc
• TF_BUILD: For Azure
• Some vendors have specific os variables preset, like GITHUB_ACTIONS, GITLAB_CI etc.

Wasn't clear if you wanted just CI, or any of these other vendors? Can we require to standardize around CI?