Testing an `__init__.py` file tests the entire package

[spagh-eddie]

I found a difference between 6.1.2 and 6.2.0. I would have expected the behavior of only testing package/__init__.py, but in 6.2.0 it tests the entirety of package/.

I think it might be due to #8016 but I am not familiar with the internals of this library.

% tree                     
.
└── package
    ├── __init__.py
    └── dont_test_me.py

% cat package/__init__.py 
"""
>>> round(22/7, 2) == 3.14
True
"""

% cat package/dont_test_me.py 
"""
>>> 2 + 2 == 5
True
"""

% pip install pytest==6.1.2  
% pytest -q --tb=no --doctest-modules package/__init__.py
.                                                                                  [100%]
1 passed in 0.02s

% pip install pytest==6.2.0  
% pytest -q --tb=no --doctest-modules package/__init__.py
.F                                                                                 [100%]
================================ short test summary info =================================
FAILED package/dont_test_me.py::package.dont_test_me
1 failed, 1 passed in 0.02s

[bluetech]

I will try to make some time to look at this, but in the meantime it would be helpful for narrowing down the issue if you can check whether this is specific to doctests, or happens in general (with regular test functions).

[spagh-eddie]

I will try to make some time to look at this, but in the meantime it would be helpful for narrowing down the issue if you can check whether this is specific to doctests, or happens in general (with regular test functions).

OK, note I changed the name to test_me_not.py to be found by test discovery. The previous name of dont_test_me.py would not get picked up. Tests in __init__.py also do not get picked up.

They seem to be consistent, but I'm not sure if expected.

% tree                     
.
└── package
    ├── __init__.py
    └── test_me_not.py

% cat package/__init__.py 
def test_good():
	assert round(22/7, 2) == 3.14

% cat package/test_me_not.py 
def test_bad():
	assert 2 + 2 == 5

% pip install pytest==6.1.2 
% pytest -q --tb=no package/__init__.py                 
F                                                                               [100%]
=============================== short test summary info ===============================
FAILED package/test_me_not.py::test_bad - assert (2 + 2) == 5
1 failed in 0.01s

% pip install pytest==6.2.0  
% pytest -q --tb=no package/__init__.py                 
F                                                                               [100%]
=============================== short test summary info ===============================
FAILED package/test_me_not.py::test_bad - assert (2 + 2) == 5
1 failed in 0.01s

[The-Compiler]

Is this a duplicate of #8976?

[spagh-eddie]

Is this a duplicate of #8976?

Thank you for finding this. It doesn't seem to be exactly the same issue. The MRE in #8976 runs one test_* file, but when adding --doctest-modules like in this PR it runs both test_* files.

% tree
.
└── foo_tests
    ├── __init__.py
    ├── test_a.py
    └── test_b.py

% pytest --no-header foo_tests/__init__.py
======================== test session starts =========================
collected 2 items

foo_tests/test_a.py ..                                         [100%]

========================= 2 passed in 0.01s ==========================

% pytest --no-header --doctest-modules foo_tests/__init__.py
======================== test session starts =========================
collected 4 items

foo_tests/test_a.py ..                                         [ 50%]
foo_tests/test_b.py ..                                         [100%]

========================= 4 passed in 0.03s ==========================

[mdengler]

Still exists in 7.2.0:

$ tree ./package
./package
├── dont_test_me.py
└── __init__.py

1 directory, 2 files

$ cat package/__init__.py 
"""
>>> round(22/7, 2) == 3.14
True
"""

$ cat package/dont_test_me.py 
"""
>>> 2 + 2 == 5
True
"""

$ poetry run pytest --version
pytest 7.2.0

$ poetry run pytest -q --tb=no --doctest-modules package/__init__.py
.F                                                                                                                                                                                                                                     [100%]
========================================================================================================== short test summary info ===========================================================================================================
FAILED package/dont_test_me.py::package.dont_test_me
1 failed, 1 passed in 0.01s