AttributeError with import-mode importlib and doctest modules

[jaraco]

Likely related to #12112, discovered in jaraco/keyring#676, a new regression appears in pytest 8.1 relating to collection. Consider this minimal example:

 draft @ cat > test_something.py
import foo
from foo.bar import baz
foo.bar.baz
 draft @ mkdir -p foo/bar/baz
 draft @ touch foo/__init__.py
 draft @ touch foo/bar/__init__.py
 draft @ touch foo/bar/baz/__init__.py
 draft @ pip-run pytest -- -m pytest --import-mode importlib --doctest-modules
============================================================== test session starts ===============================================================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /Users/jaraco/draft
collected 0 items / 1 error                                                                                                                      

===================================================================== ERRORS =====================================================================
_______________________________________________________ ERROR collecting test_something.py _______________________________________________________
test_something.py:3: in <module>
    foo.bar.baz
E   AttributeError: module 'foo' has no attribute 'bar'
============================================================ short test summary info =============================================================
ERROR test_something.py - AttributeError: module 'foo' has no attribute 'bar'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================ 1 error in 0.03s ================================================================

It seems that once the foo/bar/__init__.py has been collected for doctesting, the collection of the tests in test_something will start to fail when foo has no attribute bar. Any of the following tweaks will bypass the failure:

• disable doctests
• pin to pytest<8.1
• use a different import-mode

Something about the pytest discovery and import machinery is breaking the Python convention that importing a submodule causes the submodule to be added as an attribute of the parent module.

[jaraco]

It occurred to me that the fix in #12169 might also address this issue, but it does not:

 draft @ pip-run git+https://github.com/pytest-dev/pytest@refs/pull/12169/head -- -m pytest --import-mode importlib --doctest-modules
  WARNING: Did not find branch or tag 'refs/pull/12169/head', assuming revision or ref.
============================================================== test session starts ===============================================================
platform darwin -- Python 3.12.2, pytest-8.2.0.dev95+g1e22e0f70, pluggy-1.4.0
rootdir: /Users/jaraco/draft
collected 0 items / 1 error                                                                                                                      

===================================================================== ERRORS =====================================================================
_______________________________________________________ ERROR collecting test_something.py _______________________________________________________
test_something.py:4: in <module>
    foo.bar.baz
E   AttributeError: module 'foo' has no attribute 'bar'
============================================================ short test summary info =============================================================
ERROR test_something.py - AttributeError: module 'foo' has no attribute 'bar'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================ 1 error in 0.03s ================================================================

[nicoddemus]

Thanks @jaraco, I can reproduce the issue (also verified that #12169 does not fix this).

Will investigate which incantation is necessary for this to work.

Currently we import modules in --importmode=importlib using this snippet:

pytest/src/_pytest/pathlib.py

Lines 635 to 641 in 74f0564

	if spec is not None:
	mod = importlib.util.module_from_spec(spec)
	sys.modules[module_name] = mod
	spec.loader.exec_module(mod) # type: ignore[union-attr]
	if insert_modules:
	insert_missing_modules(sys.modules, module_name)
	return mod

[flying-sheep]

I found an issue that might be related. If not, please direct me in the right direction.

My problem so far is that I haven’t figured out a minimal reproducer yet: Somehow the _frozen_importlib_external.PathFinder finds the a namespace module for our real life example but not in the test suite.

Minimal (not yet) reproducer:

$ mkdir bar
$ mkdir foo
$ touch foo/bar.py
$ tree 
+ foo/
|  + bar.py
+ bar/

Now running a pytest test that imports foo.bar leads to this traceback (simplified, oldest to newest)

mod = import_path('foo/bar.py', mode='importlib', root='.')
→
mod = _import_module_using_spec('foo.bar', 'foo/bar.py', '.')
→
spec = meta_importer.find_spec('foo.bar', ['.'])

where meta_importer is a _frozen_importlib_external.PathFinder instance, and the module created from the spec returned by the last expression is

>>> importlib.util.module_from_spec(spec)
<module 'foo.bar' (namespace) from ['./bar']>

instead of

>>> importlib.util.module_from_spec(spec)
<module 'foo.bar' from ['./foo/bar.py']>

The documentation of MetaPathFinder.find_spec says:

If this is a top-level import, path will be None.

and since _import_module_using_spec says it “imports a module by its canonical name”, that means path should be none and you should call

meta_importer.find_spec('foo.bar', None)

right?

[jaraco]

Thanks for the tip. I did find that adding this patch works around the issue (at least in the repro):

diff --git a/src/_pytest/pathlib.py b/src/_pytest/pathlib.py
index 254d9d946..bb26a433f 100644
--- a/src/_pytest/pathlib.py
+++ b/src/_pytest/pathlib.py
@@ -638,6 +638,7 @@ def _import_module_using_spec(
         assert spec is not None
         mod = importlib.util.module_from_spec(spec)
         sys.modules[module_name] = mod
+        _set_name_in_parent(mod)
         spec.loader.exec_module(mod)  # type: ignore[union-attr]
         if insert_modules:
             insert_missing_modules(sys.modules, module_name)
@@ -646,6 +647,13 @@ def _import_module_using_spec(
     return None
 
 
+def _set_name_in_parent(mod):
+    parent, sep, name = mod.__name__.rpartition(".")
+    if not sep:
+        return
+    setattr(sys.modules[parent], name, mod)
+
+
 def spec_matches_module_path(
     module_spec: Optional[ModuleSpec], module_path: Path
 ) -> bool:

I wonder if there's similar logic in CPython's importlib logic that should be modeled.

[jaraco]

Here's where that happens in CPython. Doesn't seem to be particularly re-usable, but at least shows how that behavior is implemented there.

[flying-sheep]

My issue is fixed with #12169, but I found that the change in #12207 avoids the wrong loader altogether.

@jaraco does that one fix your problem?

[jaraco]

My issue is fixed with #12169, but I found that the change in #12207 avoids the wrong loader altogether.

@jaraco does that one fix your problem?

Just checked and it does not.