Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ProcessLookupError when terminating a local ensemble #7704

Closed
berland opened this issue Apr 19, 2024 · 0 comments · Fixed by #7710
Closed

ProcessLookupError when terminating a local ensemble #7704

berland opened this issue Apr 19, 2024 · 0 comments · Fixed by #7710
Assignees
Labels

Comments

@berland
Copy link
Contributor

berland commented Apr 19, 2024

This line:

proc.terminate()

can trigger an error if a job has been initiated by local_driver.py but the subprocess has not yet started. Observed while bug-bashing:

Killing realization 71 failed with error 
Exception in scheduler task job-59_task: 
Traceback: Traceback (most recent call last):
  File "/private/havb/projects/ert/src/ert/scheduler/local_driver.py", line 72, in _run
    returncode = await self._wait(proc)
  File "/private/havb/projects/ert/src/ert/scheduler/local_driver.py", line 91, in _wait
    return await proc.wait()
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/subprocess.py", line 135, in wait
    return await self._transport._wait()
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_subprocess.py", line 235, in _wait
    return await waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/havb/projects/ert/src/ert/scheduler/job.py", line 131, in run
    await self._submit_and_run_once(sem)
  File "/private/havb/projects/ert/src/ert/scheduler/job.py", line 119, in _submit_and_run_once
    await self.driver.kill(self.iens)
  File "/private/havb/projects/ert/src/ert/scheduler/local_driver.py", line 47, in kill
    raise err
  File "/private/havb/projects/ert/src/ert/scheduler/local_driver.py", line 40, in kill
    await self._tasks[iens]
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/private/havb/projects/ert/src/_ert/async_utils.py", line 53, in _done_callback
    raise exc
  File "/private/havb/projects/ert/src/ert/scheduler/local_driver.py", line 76, in _run
    returncode = await self._kill(proc)
  File "/private/havb/projects/ert/src/ert/scheduler/local_driver.py", line 97, in _kill
    proc.terminate()
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/subprocess.py", line 141, in terminate
    self._transport.terminate()
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_subprocess.py", line 149, in terminate
    self._check_proc()
  File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_subprocess.py", line 142, in _check_proc
    raise ProcessLookupError()
ProcessLookupError
@berland berland added the bug label Apr 19, 2024
@berland berland added this to SCOUT Apr 19, 2024
@berland berland moved this to Todo in SCOUT Apr 19, 2024
@berland berland self-assigned this Apr 19, 2024
@berland berland moved this from Todo to In Progress in SCOUT Apr 19, 2024
@berland berland moved this from In Progress to Ready for Review in SCOUT Apr 19, 2024
@xjules xjules moved this from Ready for Review to Reviewed in SCOUT Apr 23, 2024
@github-project-automation github-project-automation bot moved this from Reviewed to Done in SCOUT Apr 23, 2024
@berland berland moved this from Done to Done-Done in SCOUT May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant