AUTO cases cannot be used from test files with non-`test` prefix, such as `bench_*.py`. Even if the latter are correct test files according to `python_files`.

[smarie]

@mroeschke found this following #320

pytest introduced (in version 3 I guess) a python_files configuration option allowing users to change the naming patterns pytest is using to collect tests. (https://docs.pytest.org/en/stable/reference/reference.html#confval-python_files)

in #320 we became stricter on allowing AUTO cases to execute only on "valid test files" but we do not take into account the python_files glob patterns for this. Besides, our current definition of AUTO cases naming only holds for tests named test_*, and therefore will fail even for tests named *_test.py (which are considered valid by default in pytest).

See also https://docs.pytest.org/en/7.1.x/example/pythoncollection.html#changing-naming-conventions

• We should probably try to leverage one of pytest internal functions to check if a file is a test file. That would ensure that the config options are used correctly and exactly the same way than inside pytest.
• concerning the AUTO cases naming pattern, the above makes it hard to be "smart". The only thing I can think of (but it is ugly), would be to repeat the information explicitly to define a mapping. So

[pytest]
python_files =
    test_*.py
    check_*.py
    example_*.py

[pytest-cases]
auto_cases_files =
    test_*.py -> cases_*.py
    check_*.py -> cases_*_check.py
    example_*.py -> example_*_cases.py

The convention would be to have <source_pattern> -> <dest_pattern>, and it would be restricted to a single star acting as a capturing group.

Thoughts ?

[mroeschke]

Thanks for writing this up.

Looks like the pytest Config object should be able to access python_files (or any other config): https://docs.pytest.org/en/7.1.x/reference/reference.html#config

Yeah given the design of AUTO your mapping idea may be the only viable one; for every pattern in python_files.py AUTO looks in that file modified (prepended, appended) by cases

[smarie]

Thanks @mroeschke for your feedback ! Not sure when this can get implemented but it will be the next mod to implement for sure.