multiprocessing.Pool(64) throws on Windows

[saschanaz]

BPO	45077
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @corona10, @saschanaz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-09-01.13:20:44.186>
labels = ['type-bug', '3.9', '3.10', '3.11', 'library', 'OS-windows']
title = 'multiprocessing.Pool(64) throws on Windows'
updated_at = <Date 2021-10-03.11:23:58.490>
user = 'https://github.com/saschanaz'

bugs.python.org fields:

activity = <Date 2021-10-03.11:23:58.490>
actor = 'saschanaz'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2021-09-01.13:20:44.186>
creator = 'saschanaz'
dependencies = []
files = []
hgrepos = []
issue_num = 45077
keywords = []
message_count = 3.0
messages = ['400832', '400834', '400987']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'corona10', 'saschanaz']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45077'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-89240: Limit multiprocessing.Pool to 61 workers on Windows #102920
• gh-89240: Add _winapi.BatchedWaitForMultipleObjects and tests #107873

[saschanaz]

Similar issue as the previous bpo-26903.

Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.cpu_count()
64
>>> multiprocessing.Pool(multiprocessing.cpu_count())
Exception in thread Thread-1:
Traceback (most recent call last):
<multiprocessing.pool.Pool state=RUN pool_size=64>
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 973, in _bootstrap_inner
>>>     self.run()
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 884, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 66

[saschanaz]

The argument-less instantiation also fails, which is worse.

>>> multiprocessing.Pool()
<multiprocessing.pool.Pool state=RUN pool_size=64>
>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 973, in _bootstrap_inner
    self.run()
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 884, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 66

[eryksun]

See bpo-26903 for a similar problem in concurrent.futures.ProcessPoolExecutor. It was resolved by adding a limit constant, _MAX_WINDOWS_WORKERS == 61.

WaitForMultipleObjects() can wait on up to 64 object handles, but in this case 3 slots are already taken. The pool wait includes two events for its output and change-notifier queues (named pipes), plus the _winapi module always reserves a slot for the SIGINT event, even though this event is only used by waits on the main thread.

To avoid the need to limit the pool size, connection._exhaustive_wait() could be modified to combine simultaneous waits on up to 63 threads, for which each thread exhaustively populates a list of up to 64 signaled objects. I wouldn't want to modify _winapi.WaitForMultipleObjects, but the exhaustive wait should still be implemented in C, probably in the _multiprocessing extension module. A benefit of implementing _exhaustive_wait() in C is lightweight thread creation, directly with CreateThread() and a relatively small stack commit size.

[juj]

Note that the bug title is not accurate: on Windows calling multiprocessing.Pool(62) (or higher) does not (just) throw, but it prints an exception callstack like an uncaught throw would, and additionally Python hangs afterwards.

E.g. the script

a.py

import multiprocessing
a = multiprocessing.Pool(62)

never returns back to the OS on Windows, when invoked with python a.py. Calling taskkill /f /im python.exe is needed to recover the command prompt.

---

What happens on macOS and Linux if one attempts to create a multiprocessing pool with 100, 1000 or 10000 processes? I.e. what is the failure behavior on non-Windows platforms when too many are requested?

I wonder if instead of throwing, it would make sense to either silently cap the number of workers in a pool to the max limit?

Otherwise, how will users know what the limit currently is so creating a Pool will not throw? E.g. should there be a function call to ask max_worker_count_that_will_not_throw, or should users linearly search or bisect for the limit themselves? (if so, how would they discover that they have this burden on them in order to correctly use the API?)