Test hangs when using mp.Pool on Windows/macOS using CLI runner when --import-mode=importlib

[DylanLukes]

This is a duplicate of #5429 (which does not appear to actually be resolved, or has regressed) but I come bearing more details, in particular a matrix that should help diagnose the source of the problem.

Here's a minimal example:

from multiprocessing import Pool

def times_two(x):
    return x * 2

def test_foo():
    with Pool(1) as pool:
        assert pool.map(times_two, [1, 2, 3, 4]) == [2, 4, 6, 8]

This is using pytest 7.3.1.

Tests were performed on Github Actions ubuntu-latest, windows-latest, and macos-latest, which correspond to Ubuntu 22.04, Windows Server 2022, and macOS 12 Monterey (not Ventura). I also validated the macOS and Linux results on macOS 13 Ventura and Ubuntu 20.04 on my own machines.

All of the below were tested with Python 3.11 on Github Actions runners. On my own machines, I tested with both 3.10 and 3.11, same results.

Platform	Runner	Result
Windows	pytest	Hangs
macOS	pytest	Hangs
Linux	pytest	Completes
macOS	PyCharm	Completes

The last row in particular suggests that the issue is likely in the test runner. Running all tests via PyCharm.app/Contents/plugins/python/helpers/pycharm/_jb_pytest_runner.py does not hang.

– Dylan

[DylanLukes]

After some further troubleshooting, I realized I've missed a key element of reproduction:

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
]

[evgeniiz321]

I faced the same problem on my mac. If run pytest with -s option there is a helpful stacktrace:

Process SpawnPoolWorker-4:
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/queues.py", line 368, in get
    return _ForkingPickler.loads(res)
ModuleNotFoundError: No module named 'test_main'

Looks like the problem is in the spawn method used by the multiprocessing by default on mac (and on windows). It doesn't propagate a list of modules known to a parent process to its subprocesses. importlib option doesn't affect sys.path as well due to its design.

There are multiple workarounds here:

It helps to replace the default 'spawn' with the 'fork' (but only on mac), if use the example above:

from multiprocessing import get_context


def times_two(x):
    return x * 2


def test_foo():
    with get_context("fork").Pool(1) as pool:
        assert pool.map(times_two, [1, 2, 3, 4]) == [2, 4, 6, 8]

Also, if we run pytest like - python -m pytest, it prepends current directory to sys.path and in this case the scenario above also works.

To sum up - I'm not sure this one can be fixed from the pytest perspective since it is how multiprocessing works on different platforms.

Hope it helps someone :)

[RonnyPfannschmidt]

Importlib based imports are not compatible with multiprocess at all

[Jeitan]

Folks are still hitting this (#12178) - if these two things are incompatible, can we put that into the documentation?