Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crashes in containers on Cgroupsv2 based hosts #10103

Open
DominicLavery opened this issue Dec 2, 2024 · 1 comment · May be fixed by #10089
Open

Crashes in containers on Cgroupsv2 based hosts #10103

DominicLavery opened this issue Dec 2, 2024 · 1 comment · May be fixed by #10089
Labels
Crash 💥 A bug that makes pylint crash multiprocessing
Milestone

Comments

@DominicLavery
Copy link

Bug description

In containers running on cgroupv2 systems _query_cpu currently returns None. This results in sched_getaffinity being used, which will normally return all installed CPUs of the host. This can result in crashes with the error:

.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Configuration

Command used

pylint

Pylint output

Traceback (most recent call last):
  File "/home/circleci/.local/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/home/circleci/.local/lib/python3.10/site-packages/pylint/__init__.py", line 34, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "/home/circleci/.local/lib/python3.10/site-packages/pylint/lint/run.py", line 215, in __init__
    linter.check(args)
  File "/home/circleci/.local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 679, in check
    check_parallel(
  File "/home/circleci/.local/lib/python3.10/site-packages/pylint/lint/parallel.py", line 153, in check_parallel
    for (
  File "/home/circleci/.pyenv/versions/3.10.15/lib/python3.10/concurrent/futures/process.py", line 575, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/home/circleci/.pyenv/versions/3.10.15/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/home/circleci/.pyenv/versions/3.10.15/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/home/circleci/.pyenv/versions/3.10.15/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/circleci/.pyenv/versions/3.10.15/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Expected behavior

The run of pylint to succeed rather than crash. _query_cpu should return the restricted number of cpus rather than None so that the number of jobs is calculated correctly for the container environment

Pylint version

pylint 3.3

OS / Environment

No response

Additional dependencies

@DominicLavery DominicLavery added the Needs triage 📥 Just created, needs acknowledgment, triage, and proper labelling label Dec 2, 2024
@Pierre-Sassoulas Pierre-Sassoulas added this to the 3.3.3 milestone Dec 2, 2024
@Pierre-Sassoulas Pierre-Sassoulas added Crash 💥 A bug that makes pylint crash multiprocessing and removed Needs triage 📥 Just created, needs acknowledgment, triage, and proper labelling labels Dec 2, 2024
@Pierre-Sassoulas
Copy link
Member

Thank you for opening the issue, in the future you can just references the merge request in the changelog if you fixed a problem without opening an issue first :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Crash 💥 A bug that makes pylint crash multiprocessing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants