ModuleNotFoundError with multiprocessing and import-mode=importlib

[jaraco]

In jaraco/keyring#673, I captured an issue stemming from the introduction of --import-mode importlib to jaraco/skeleton projects. In jaraco/keyring@6ff02e0, I'm attempting to enable importlib mode for all projects in order to support projects that use namespaces and those that don't in a uniform way. However, enabling importlib mode causes failures in tests that involve multiprocessing. Consider this minimal example:

 @ cat > tox.ini
[testenv]
deps =
	pytest
commands =
	pytest {posargs:--import-mode=importlib}
 @ mkdir tests
 @ cat > tests/test_multiprocess.py
import multiprocessing


def multiprocess_do():
    pass


def test_multiprocess():
    proc1 = multiprocessing.Process(target=multiprocess_do)
    proc1.start()
    proc1.join()
    assert proc1.exitcode == 0

 @ tree
.
├── tests
│   └── test_multiprocess.py
└── tox.ini

2 directories, 2 files
 @ tox
py: commands[0]> pytest --import-mode=importlib
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
cachedir: .tox/py/.pytest_cache
rootdir: /Users/jaraco/draft
collected 1 item

tests/test_multiprocess.py F                                             [100%]

=================================== FAILURES ===================================
______________________________ test_multiprocess _______________________________

    def test_multiprocess():
        proc1 = multiprocessing.Process(target=multiprocess_do)
        proc1.start()
        proc1.join()
>       assert proc1.exitcode == 0
E       AssertionError: assert 1 == 0
E        +  where 1 = <Process name='Process-1' pid=38572 parent=38570 stopped exitcode=1>.exitcode

tests/test_multiprocess.py:12: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'tests'
=========================== short test summary info ============================
FAILED tests/test_multiprocess.py::test_multiprocess - AssertionError: assert...
============================== 1 failed in 0.09s ===============================
py: exit 1 (0.19 seconds) /Users/jaraco/draft> pytest --import-mode=importlib pid=38570
  py: FAIL code 1 (0.22=setup[0.02]+cmd[0.19] seconds)
  evaluation failed :( (0.25 seconds)

The same error occurs even if tests/__init__.py is present. The issue occurs also with pytest 8.0.2 and 7.4.4. My guess is that it never worked with this mode.

It seems that tests.test_multiprocess is in sys.modules and objects passed between the processes might reference that module, so the child process needs to have the same path handling that the parent had but doesn't.

Running tox -- --import-mode prepend or tox -- --import-mode append bypasses the failure.

Can multiprocessing be made to work with import-mode importlib?

[jaraco]

Also, the failure doesn't occur if the tests are invoked with .tox/py/bin/python -m pytest, presumably because that mode adds . to sys.path.

[Jeitan]

AH! I have this same issue! I couldn't figure out what I was doing wrong, but now I know it's the mode that is the problem. I checked by running my tests that use multiprocessing (that had been having those errors) via python -m pytest as was suggested and suddenly they work fine. I wonder if anybody has looked at this since April?

[Jeitan]

It seems this has been the case for a while, since #10965 at least. Comments there indicate that multiprocessing might just be incompatible with the importlib import mode ... ? I'm not well-versed enough to know if that's true or not, but if that IS the case, since that mode is recommended by the documentation, perhaps it's worth a specific callout in the docs that it won't work with multiprocessing?

[RonnyPfannschmidt]

Duplicate of #10965