Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows #13132

Merged
merged 3 commits into from
May 8, 2019
Merged

bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows #13132

merged 3 commits into from
May 8, 2019

Conversation

brianquinlan
Copy link
Contributor

@brianquinlan brianquinlan commented May 6, 2019 •
edited by bedevere-bot
Loading

zooba
zooba approved these changes May 6, 2019
View reviewed changes
@zooba @brianquinlan
Update Lib/test/test_concurrent_futures.py
97c1084 
Co-Authored-By: brianquinlan <brian@sweetapp.com>
JelleZijlstra
JelleZijlstra reviewed May 7, 2019
View reviewed changes
Lib/concurrent/futures/process.py
else:
if max_workers <= 0:
raise ValueError("max_workers must be greater than 0")
elif (sys.platform == 'win32' and
max_workers > _MAX_WINDOWS_WORKERS):
raise ValueError(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider automatically changing max_workers to 61 instead? That would make for a nicer API for people using max_workers=os.cpu_count(), and it would make code like what was in Black before https://github.com/python/black/pull/838/files work unchanged.

On the other hand, it may be unexpected that we silently use fewer workers than the user asked for.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought silently using fewer processes could be surprising. I also think that it's pretty unusual in the standard library to silently fix bad argument values.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like raising when too many workers are requested, and doing it quietly when it was "automatic".

@zooba zooba merged commit 3988986 into python:master May 8, 2019
@miss-islington
Copy link
Contributor

Thanks @brianquinlan for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

@zooba
Copy link
Member

zooba commented May 8, 2019

Thanks, Brian!

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 8, 2019
@brianquinlan @miss-islington
bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows (python…
c805d53 
…GH-13132)

Co-Authored-By: brianquinlan <brian@sweetapp.com>
(cherry picked from commit 3988986)

Co-authored-by: Brian Quinlan <brian@sweetapp.com>
@bedevere-bot
Copy link

GH-13206 is a backport of this pull request to the 3.7 branch.

@miss-islington
Copy link
Contributor

Thanks @brianquinlan for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 29, 2019
@brianquinlan @miss-islington
bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows (python…
4ad2b16 
…GH-13132)

Co-Authored-By: brianquinlan <brian@sweetapp.com>
(cherry picked from commit 3988986)

Co-authored-by: Brian Quinlan <brian@sweetapp.com>
@bedevere-bot
Copy link

GH-13643 is a backport of this pull request to the 3.7 branch.

ned-deily pushed a commit that referenced this pull request May 29, 2019
@miss-islington @brianquinlan @ned-deily
bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows (GH-13132…
8ea0fd8 
…) (GH-13643)

Co-Authored-By: brianquinlan <brian@sweetapp.com>
(cherry picked from commit 3988986)

Co-authored-by: Brian Quinlan <brian@sweetapp.com>
mingwandroid pushed a commit to mingwandroid/cpython that referenced this pull request Sep 3, 2020
@miss-islington @brianquinlan @mingwandroid
bpo-26903: Limit ProcessPoolExecutor to 61 workers on Windows (python…
24f3738 
…GH-13132) (pythonGH-13643)

Co-Authored-By: brianquinlan <brian@sweetapp.com>
(cherry picked from commit 3988986)

Co-authored-by: Brian Quinlan <brian@sweetapp.com>
@saschanaz saschanaz mentioned this pull request Sep 1, 2021
Closed
sbc100 added a commit to emscripten-core/emscripten that referenced this pull request Sep 20, 2022
@sbc100
Fix EM_PYTHON_MULTIPROCESSING under windows
4e73a3d 
This mirrors the upstream bugfix in python 3.8.0 which limits the
size of the worker pool to 61: python/cpython#13132

Fixes: #13785
@sbc100 sbc100 mentioned this pull request Sep 20, 2022
Merged
sbc100 added a commit to emscripten-core/emscripten that referenced this pull request Sep 20, 2022
@sbc100
Fix EM_PYTHON_MULTIPROCESSING under windows
1676673 
This mirrors the upstream bugfix in python 3.8.0 which limits the
size of the worker pool to 61: python/cpython#13132

Fixes: #13785
sbc100 added a commit to emscripten-core/emscripten that referenced this pull request Sep 20, 2022
@sbc100
Fix EM_PYTHON_MULTIPROCESSING under windows
f925fc2 
This mirrors the upstream bugfix in python 3.8.0 which limits the
size of the worker pool to 61: python/cpython#13132

Fixes: #13785
sbc100 added a commit to emscripten-core/emscripten that referenced this pull request Sep 20, 2022
@sbc100
Fix EM_PYTHON_MULTIPROCESSING under windows (#17892)
8494ac4 
This mirrors the upstream bugfix in python 3.8.0 which limits the
size of the worker pool to 61: python/cpython#13132

Fixes: #13785
@diogocp diogocp mannequin mentioned this pull request Oct 21, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JelleZijlstra JelleZijlstra JelleZijlstra left review comments

@zooba zooba zooba approved these changes

Assignees

@zooba zooba

Labels
OS-windows
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@brianquinlan @miss-islington @zooba @bedevere-bot @JelleZijlstra @ned-deily @the-knights-who-say-ni