Skip to content

Commit

Permalink
Work around multiprocessing limits on Windows (#34489)
Browse files Browse the repository at this point in the history
Python 3 may hang when starting more than about 60 child processes using
the multiprocessing module. Clamping to 56 gives some margin for error
and recognizes that going too high hits diminishing returns (due to
startup costs).

More information can be found in https://crbug.com/1190269 and
https://crbug.com/1336854.

The specific issue found is that "git cl presubmit --force --all" in a
Chromium repo on a machine with more than 60 cores will hang when it
gets to the step that runs lint.py.
  • Loading branch information
randomascii authored Jun 21, 2022
1 parent a18e6b2 commit bdc35e8
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions tools/lint/lint.py
Original file line number Diff line number Diff line change
Expand Up @@ -1025,6 +1025,11 @@ def lint(repo_root, paths, output_format, ignore_glob=None, github_checks_output

if jobs == 0:
jobs = multiprocessing.cpu_count()
if sys.platform == 'win32':
# Using too many child processes in Python 3 hits either hangs or a
# ValueError exception, and, has diminishing returns. Clamp to 56 to
# give margin for error.
jobs = min(jobs, 56)

with open(os.path.join(repo_root, "lint.ignore")) as f:
ignorelist, skipped_files = parse_ignorelist(f)
Expand Down

0 comments on commit bdc35e8

Please sign in to comment.