pytest==8.1.1 Import regression in some namespace package layouts

[aaraney]

As first noted in #12074, below is an example project structure with two packages under the ns namespace that now fail to import after the changes introduced in 8.1.x:

ns
└── python
    ├── bar
    │   ├── ns
    │   │   ├── bar
    │   │   └── test
    │   │       ├── __init__.py
    │   │       ├── bar.py
    │   │       └── test_bar.py
    │   └── pyproject.toml
    └── foo
        ├── ns
        │   ├── foo
        │   └── test
        │       ├── __init__.py
        │       ├── foo.py
        │       └── test_foo.py
        └── pyproject.toml

Below are the contents of test_foo.py and foo.py. test_bar.py and bar.py look nearly identical.

# python/foo/ns/test/test_foo.py
from .foo import value

def test_foo():
    assert value == "foo"

# python/foo/ns/test/foo.py
value = "foo"

In pytest==8.0.2, python -m pytest --import-mode=importlib correctly discovers and runs the tests from the top level ns directory. In pytest==8.1.1, python -m pytest --import-mode=importlib -o "consider_namespace_packages=true", results in the following error during collection:

========================== test session starts ===========================
platform darwin -- Python 3.9.16, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/user/pytest-12074
collected 0 items / 2 errors

================================= ERRORS =================================
____________ ERROR collecting python/bar/ns/test/test_bar.py _____________
ImportError while importing test module '/home/user/pytest-12074/python/bar/ns/test/test_bar.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
python/bar/ns/test/test_bar.py:1: in <module>
    from .bar import value
E   ModuleNotFoundError: No module named 'test.bar'
____________ ERROR collecting python/foo/ns/test/test_foo.py _____________
ImportError while importing test module '/home/user/pytest-12074/python/foo/ns/test/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
python/foo/ns/test/test_foo.py:1: in <module>
    from .foo import value
E   ModuleNotFoundError: No module named 'test.foo'
======================== short test summary info =========================
ERROR python/bar/ns/test/test_bar.py
ERROR python/foo/ns/test/test_foo.py
!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!
=========================== 2 errors in 0.04s ============================

To reproduce the issue clone https://github.com/aaraney/pytest-12074, pip install ns.foo and ns.bar, pip install the different versions of pytest and run with the aforementioned commands.

[nicoddemus]

Thanks @aaraney for the report, I will take a look ASAP. 👍

[nicoddemus]

@aaraney,

How are you setting up your PYTHONPATH? I managed to get your example to work by putting ns/python/bar and ns/python/foo in the PYTHONPATH, which seems to be the recommendation for using namespace packages.

λ python -m pytest .tmp\ns\python --import-mode=importlib -o consider_namespace_packages=true --no-header
======================== test session starts ========================
collected 2 items

.tmp\ns\python\bar\ns\bar\test\test_bar.py .                   [ 50%]
.tmp\ns\python\foo\ns\foo\test\test_foo.py .                   [100%]

========================= 2 passed in 0.05s =========================

λ echo %PYTHONPATH%
e:\projects\pytest\.tmp\ns\python\bar;e:\projects\pytest\.tmp\ns\python\foo

λ cd
e:\projects\pytest

One thing you can use to check if your namespace package is working is to import it inside Python directly:

λ python
Python 3.12.2 (tags/v3.12.2:6abddd9, Feb  6 2024, 21:26:36) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ns.bar.test.test_bar
>>> import ns.foo.test.test_foo
>>> ns
<module 'ns' (namespace) from ['e:\\projects\\pytest\\.tmp\\ns\\python\\bar\\ns', 'e:\\projects\\pytest\\.tmp\\ns\\python\\foo\\ns']>

Can you try the above and show what you get?

[joshbode]

I am seeing something similar - I think it is because there is effectively a collision between ns.test in each subpackage.
I deleted the __init__.py and following that it ran successfully using:

$ pytest --import-mode=importlib -o "consider_namespace_packages=true"

Without consider_namespace_packages if still failed, though.

BTW, @aaraney, there's a typo in the original command above - a missing g (consider_namespace_packaes), which may help :)

[joshbode]

Also, and this may be a different bug, but if I run python -m pytest ... instead of the pytest ... command directly then I don't need to delete the __init__.py in the test directories, though either way I do require the consider_namespace_packages option.

With pytest==8.0.2 it works fine either way (with/without running via python -m pytest) but it fails without the __init__.py for a different reason (relative imports within the test directory don't work).

[nicoddemus]

I suggest to also test using the Python REPL to import the namespace packages, taking pytest out of the picture, to ensure your configuration is correct.

[aaraney]

@nicoddemus, thanks again for looking into this! The example I provided was incomplete. When I adding __init__.py files to python/ns/bar/ns/bar/__init__.py and python/ns/foo/ns/foo/__init__.py, so python / setuptools would recognize them as actual submodules (as you alluded to) pytest ran successfully!

In the project I thought I discovered this bug in, it turns out there was an erroneous __init__.py file at the same level as a submodule's pyproject.toml / setup.py / setup.cfg (e.g. python/ns/bar/__init__.py). For some reason pytest==8.0.2 did not balk at this and collected correctly, however with pytest==8.1.1, I received ImportError: attempted relative import beyond top-level package and ModuleNotFoundError: No module named 'some-module' errors. I think that is the correct behavior per PEP 420.

@joshbode, great catch! Just fixed it!

[nicoddemus]

@aaraney thanks for the clarification!

[aaraney]

@nicoddemus, I think this can be closed unless you feel the need for it to stay open. It seems that it was a false alarm on my part. Thanks again for looking into this!

[nicoddemus]

If somebody sees this and is not working as expected, please try importing your modules using the Python REPL first, to ensure the configuration is correct. 👍

I'm closing this for now then, with "working as intended".

Thanks everyone!

[joshbode]

The REPL tip is a good one - it helped me figure out that my imports were being masked and that it wasn't being treated correctly as a namespace package, even before it hit pytest

[jaraco]

I have encountered a related issue.

In this job, the tests are failing with:

ModuleNotFoundError: No module named 'test.support'

The module test.support is part of the standard library, so should be importable. Pinning to pytest < 8.1 works around the issue.

The problem is that the namespace package jaraco is being missed in the doctest discovery, so jaraco.test is being imported as test:

py: commands[0]> pytest --pdb
============================================================== test session starts ===============================================================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
cachedir: .tox/py/.pytest_cache
rootdir: /Users/jaraco/code/jaraco/jaraco.test
configfile: pytest.ini
plugins: jaraco.test-5.3.1.dev79+g23d6c10, enabler-3.0.0, checkdocs-2.10.1, ruff-0.3.1, mypy-0.10.3, cov-4.1.0
collected 23 items                                                                                                                               

docs/conf.py ....                                                                                                                          [ 17%]
jaraco/test/__init__.py ....                                                                                                               [ 34%]
jaraco/test/cpython.py ...F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
001 
002 Compatibility shims for getting stuff from test.support across
003 Python versions (for compatibility with Python 3.9 and earlier).
004 
005 >>> os_helper = try_import('os_helper') or from_test_support('temp_dir')
UNEXPECTED EXCEPTION: ModuleNotFoundError("No module named 'test.support'")
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/doctest.py", line 1361, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest test.cpython[0]>", line 1, in <module>
  File "/Users/jaraco/code/jaraco/jaraco.test/jaraco/test/cpython.py", line 25, in from_test_support
    import test.support
ModuleNotFoundError: No module named 'test.support'
/Users/jaraco/code/jaraco/jaraco.test/jaraco/test/cpython.py:5: UnexpectedException
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/jaraco/code/jaraco/jaraco.test/jaraco/test/cpython.py(25)from_test_support()
-> import test.support
(Pdb) import test
(Pdb) test.__file__
'/Users/jaraco/code/jaraco/jaraco.test/jaraco/test/__init__.py'

Essentially, #3396 re-emerges with pytest 8.1.

[jaraco]

You can repro the issue by cloning jaraco/jaraco.test@23d6c10 and running tox. Let me know if you need a more minimal reproducer.

I'm not doing any path manipulation, but merely running the tests with the package under test installed.

[nicoddemus]

Thanks @jaraco,

I investigated this a bit and got it to work: https://github.com/nicoddemus/jaraco.test/tree/ns-pytest, but I till need to investigate more in order to understand why your original layout does not work (it is late and I'm out of time to continue on this). I will take another look ASAP.

[nicoddemus]

Thanks again @jaraco for the reproducer. #12169 improves how we handle namespace packages and works for jaraco.test, except you will need to set consider_namespace_packages in your pytest.ini. I will be happy to open a PR when #12169 gets released. 👍

[jaraco]

Today I also encountered this issue in jaraco.site. Again, this looks to be a case of the stdlib name conflicting with the inner package name. In this case, the pytest collection fails when relative imports within jaraco.site are unable to find each other.

I was able to confirm that the code in #12169 and by adding the consider_namespace_packages setting, the tests now pass. Same for jaraco.test (as you also confirmed).

I'm new to the consider_namespace_packages setting. I find it a little annoying that this setting now becomes necessary for some projects but not others. Is it safe to use on other projects (what's the harm for enabling it for packages that aren't namespace packages)? Are there plans for making this setting the default? After all, Python itself and Setuptools both support projects with or without namespace packages without enabling a setting.

[nicoddemus]

Is it safe to use on other projects (what's the harm for enabling it for packages that aren't namespace packages)?

As far as we know yes, but we would need to get reports from this in the wild.

Are there plans for making this setting the default?

Not at this moment; it introduces some performance overhead to import all modules that pytest collects. We need to measure how much impact that is, but to be safe we decided to make it opt-in.

After all, Python itself and Setuptools both support projects with or without namespace packages without enabling a setting.

Indeed, however pytest and python/setuptools need to perform different procedures.

Python needs to find a file based on a module name, say jaraco.test -> /home/jaraco/projects/jaraco.test/jaraco/test, which is easy to do because it can easily lookup jaraco (based on jaraco.test module name) using sys.path.

pytest needs to find a module name from a file, for example find out that /home/jaraco/projects/jaraco.test/jaraco/test belongs to the namespace jaraco.test , because the collection deals with files. This is non-trivial because we need to search upwards the file system from the module we are collecting to see if it is part of a namespace package, and there's little tooling to help with that.

Given namespace is not used widely (at least that is our impression), we decided to not enable it by default due to extra cost it would incur for all imports if enabled.

We plan to eventually do some profiling and see if the cost is significant to enable it by default in test suites which do not use namespace packages; if so then definitely we can switch it on by default and mark it as obsolete (probably no need to deprecated, just mark as obsolete).