Add a function to detect if pytest is running

[adamchainz]

What's the problem this feature will solve?

Django settings modules often load environment variables from a .env file:

from pathlib import Path

import dotenv

BASE_DIR = Path(__file__).resolve().parent.parent

dotenv.load_dotenv(dotenv_path=BASE_DIR / ".env")

Loading from .env is not desirable during test runs. Environment variables added for local debugging may affect the test run in unexpected ways, such as making remote API calls.

The settings module should therefore only load the .env file if pytest isn't running:

from pathlib import Path

import dotenv

BASE_DIR = Path(__file__).resolve().parent.parent

if ... # pytest is not running
    dotenv.load_dotenv(dotenv_path=BASE_DIR / ".env")

Changing behaviour during tests is usually undesirable, but here it's done to isolate the test run, which is a good thing.

pytest documents a pattern to change behaviour using fixtures here: https://docs.pytest.org/en/latest/example/simple.html#detect-if-running-from-within-a-pytest-run . Unfortunately this techinque is incompatible with this particular problem. pytest-django's fixtures load the settings module early, before fixtures from conftest.py fixtures run, so the pattern cannot be applied. And environment variable changes cannot be undone after the fact.

To solve this I made pytest-is-running, which can be used like so:

from pathlib import Path

import dotenv
import pytest_is_running

BASE_DIR = Path(__file__).resolve().parent.parent

if not pytest_is_running.is_running():
    dotenv.load_dotenv(dotenv_path=BASE_DIR / ".env")

pytest-is-running works because plugin fixtures can load early, and it uses hooks with tryfirst=True.

Describe the solution you'd like

Rather than have a documented workaround and plugin, I'd like to see a function in core pytest that provides the ability to check if pytest is running. It could simplify things for those using the both the documented pattern and the plugin.

It would also be good if the solution could somehow avoid the cost of doing import pytest, so that non-test pathways do not import the heap of stuff in pytest only to ignore it because pytest isn't running. The plugin is deliberately structured to avoid this.

Alternative Solutions

I wrote the plugin.

Additional context

none

[The-Compiler]

We currently just document how to do this, probably in a way similarly to what your plugin does.

Note you could also just check if "pytest" in sys.modules: as long as your code under test does not import pytest.

We do have a PYTEST_CURRENT_TEST environment variable already. Personally I'd be fine with having a PYTEST_VERSION or whatever which gets set very early, I suppose PYTEST_CURRENT_TEST will be set too late for it to be useful at import time.

[The-Compiler]

Also see pytest-dev/pytest-django#333 which has some additional considerations and use-cases around this.

[nicoddemus]

Personally I'd be fine with having a PYTEST_VERSION or whatever which gets set very early,

I'm OK with this solution. It would also remove that env var from os.environ at the end of pytest.main(), I assume?

[bluetech]

A while ago I had an idle thought that something like this should be a python feature. I was thinking a builtin constant __test__, similar to __debug__, that would normally be false, but that a test runner would set to true while discovering and running tests.

In the Rust language for example it is very common to interleave unit tests with the code they're testing in the same file using an attribute. In Python this would be:

def fibonacci(n): ...

if __test__:
    import pytest

    def test_fibonacci(n):
        assert fibonacci(10) == 55

        with pytest.raises(ValueError):
            fibonacci(-1)

def factorial(n): ...

if __test__:
    def test_factorial(n):
        assert factorial(50) == 30414093201713378043612608166064768844377641568960512000000000000

Not for everyone, but I think there are cases I would have used this.

[RonnyPfannschmidt]

after the drop of python 3.6 we can use https://docs.python.org/3/library/contextvars.html

i would propose to use this for gathering whether pytest is running

i woudl strictly oppose exposing sometihng like config as a contextvar however unless we had a stack for nested pytest session

[adamchainz]

We currently just document how to do this, probably in a way similarly to what your plugin does.

Yes, I noted in the description why this doesn't work for the case of the django settings file 😊

---

I like the idea of the PYTEST_VERSION (or whatever) environment variable. This is general and wouldn't require import pytest in production code. Yes it should be popped out of os.environ after pytest completes its run.

A __test__ builtin would be interesting, but it wouldn't be so easy to add. If it was only set by pytest, it wouldn't exist without importing pytest, unless it were actually added to Python, and placed in a backports package.

contextvars seem like they'd also require import pytest to use, since the var must be defined in a module—unless pytest shipped a side module to contain the var?

[RonnyPfannschmidt]

One can use sys.modules.get in such helpers

Personally im -1 on adding easy and obvious fast paths to make it easy to keep messy global configuration around

This stems from stuff going horrendously wrong every time a project uses that and then suddenly some tests need to run with different configurations or multiple configurations

[adamchainz]

One can use sys.modules.get in such helpers

🤔 Idk why I didn't think of that, thanks for the hint.

I think for now I will try "pytest" in sys.modules instead of my plugin.

Agree that it can go wrong as complexity creeps in.

[Bengt]

I stopped my overengineering when I arrived at:

def we_are_running_via_pytest() -> bool:
    """Detect if we are running via pytest."""
    # When we are running via pytest it is loaded to the sys modules dictionary.
    _we_are_running_via_pytest: bool = 'pytest' in sys.modules
    return _we_are_running_via_pytest

Source: https://docs.python.org/3/library/sys.html#sys.modules

Use it like so:

def get_configuration() -> Configuration:
    configuration: Configuration

    if we_are_running_via_pytest():
        print('Creating test configuration.')
        configuration = _get_test_configuration()
    else:
        print('Creating runtime configuration.')
        configuration = _get_runtime_configuration()

    return configuration

[nicoddemus]

Thanks everyone!

I'm closing this for now, with the recommended solution being:

is_running_pytest = "pytest" in sys.modules

If somebody wants to contribute that to the documentation, it would be great!

[dheerajck]

But I think "pytest" in sys.modules returning True if theres a pytest import makes it kinda error prone

[dheerajck]

This might be over engineered but I think this works better and helps to avoid one of the edge case with "pytest" in sys.modules

import os
import inspect
import sys

# List of pytest related files
pytest_related_files = (
    os.path.join('_pytest', 'python_path.py'),
    os.path.join('_pytest', 'logging.py'),
    os.path.join('_pytest', 'main.py'),
    os.path.join('_pytest', 'runner.py'),
    os.path.join('_pytest', 'config', '__init__.py'),
    os.path.join('_pytest', 'debugging.py')
)

# Check if any of the pytest related files are in the current stack
is_pytest_in_stack = any(frame.filename.endswith(pytest_related_files) for frame in inspect.stack())

# Check if a pytest is currently running
is_pytest_running = os.environ.get("PYTEST_CURRENT_TEST")

# Check if pytest module is imported
# We probably dont need to use this as, if one of the other two condition is True, this should be True
is_pytest_imported = "pytest" in sys.modules 

if is_pytest_imported and (is_pytest_in_stack or is_pytest_running):
    print(True)
else:
    print(False)

Do let me know if theres a case where this will not work

[RonnyPfannschmidt]

too many magic constants

[adamchainz]

inspect.stack() is also relatively slow as it makes Python build all the frame objects. Checking for PYTEST_CURRENT_TEST is a good idea if you want something a bit more robust than than the sys.modules check. But IMO the sys.modules check is good enough for most projects, it should be exceedingly rare to import pytest outside of the test process.

I archived my pytest-is-running project btw.

[dheerajck]

What do you think about this
https://github.com/dheerajck/pytest/tree/dev

This has been discussed above

contexvar will not help as mentioned in this comment

[dheerajck]

Also I think this will work with _pytest/main.py and config/__init__.py, I am just trying to cover the edge cases if theres any by adding others

I dont think it inspect.stack being slow is an issue in this case unless it is noticeably slow , because we should only use it in places like django settings(which is only run once per server start, in every other cases(like when a ut is running) we can use os.environ.get("PYTEST_CURRENT_TEST") which works great

[dheerajck]

I am using IS_PYTEST = sys.argv[0].endswith('pytest') or os.environ.get('PYTEST_XDIST_WORKER_COUNT') which works with pytest and pytest -n x

[dheerajck]

I am aware of cases where this doesnt work like python -m pytest but this seems better than "pytest" in sys.modules

[nicoddemus]

PYTEST_VERSION environment variable is now defined when executing pytest (#12190). It will be available in the next minor release.

Thanks @dheerajck for the PR. 👍

[dheerajck]

Thanks for reviewing my code and letting me know about some good coding practices :)