Don't panic on run_forever exceptions #562
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the event loop running in run_forever has an exception, don't panic the thread with "unwrap". Just inspect and log
the error, then exit the thread (after stopping and closing the loop). Nothing will inspect those errors anyway.
Also, sys.exit in Python raises a special SystemExit exception which inherits from BaseException but not Exception.
In those cases we don't want to set the rust panic flag, but we can just treat it as a normal python exception.
This was discovered while working on
python/tests/test_actor_error.py::test_actor_mesh_supervision_handling
,but it doesn't fully fix the problems there. Currently the run_forever thread is panic'ing when destructing thread
local storage, and I'm not sure where that is coming from.
Full exception message:
The stack makes it seem like somehow "tracing_subscriber" had some registry on this run_forever thread,
and it was failing to run its Drop