-
-
Notifications
You must be signed in to change notification settings - Fork 345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
change strict_exception_groups default to True #2886
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #2886 +/- ##
=======================================
Coverage 99.64% 99.64%
=======================================
Files 116 116
Lines 17477 17499 +22
Branches 3133 3147 +14
=======================================
+ Hits 17415 17437 +22
Misses 43 43
Partials 19 19
|
14d23d1
to
84fb527
Compare
Other than |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some initial comments
docs/source/reference-core.rst
Outdated
of ``TaskGroup`` in asyncio and anyio. This is also to avoid any bugs caused by only | ||
catching one type of exceptions/exceptiongroups. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
of ``TaskGroup`` in asyncio and anyio. This is also to avoid any bugs caused by only | |
catching one type of exceptions/exceptiongroups. | |
of ``TaskGroup`` in asyncio and anyio. This is also to avoid any bugs caused by only | |
catching single exceptions in a concurrent scenario. |
I'm not thrilled about either wording (proposed or original) though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that we'll eventually want to rewrite this whole section from the new strict-by-default perspective, under the heading "Deprecated: non-strict ExceptionGroup
s" - to explain that it only exists for backwards-compatibility, will be removed in future, and that we recommend against it for all new code.
I don't want to delay this PR further though, so let's open an issue to revisit the docs on this topic rather than waiting to merge.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work!
I've left my thoughts on the raises()
context manager over in pytest-dev/pytest#11538 (comment).
If you could pull out the _assert_raises
removal and any other test improvements that don't need this to be merged, I'd love to get them in sooner and reduce the size of this diff 🙂
newsfragments/2786.breaking.rst
Outdated
@@ -0,0 +1,2 @@ | |||
``strict_exception_groups`` now defaults to True in ``trio.run`` and ``trio.start_guest_run``, as well as ``trio.open_nursery`` as a result of that. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As as result of that what?
I'd also use rst cross-references here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
kinda awkward wording, but should be better now
src/trio/_core/_run.py
Outdated
strict_exception_groups (bool): Unless set to false, nurseries will always wrap | ||
even a single raised exception in an exception group. This can be overridden | ||
on the level of individual nurseries. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer this to the phrasing in the open_nursery()
docs above - replace that with this? It would also be good to note that strict_exception_groups=False
is deprecated and will eventually be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's important to add that "=False
is deprecated, and will be removed in a future version of Trio."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need to add a DeprecationWarning
when specifying False
to use a wording like that. But if we wanna do that as well in the same release I can get going on it in a separate PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added note about =False
being deprecated&removed in the future, and updated open_nursery
docstring.
…ups, and noted by reviewers
Reviewed two cases where I thought ExceptionGroups were missing, and they turned out to be fairly simple cases. Next steps:
|
pre-commit.ci autofix |
Ah right, a plethora of PT012. Well, first gotta settle on what message to add to them. PT017 in the helper is pretty amusing. |
and fix various errors, also disabling PT012.
Note that this has *both* RaisesGroup and ExpectedExceptionGroup
src/trio/_core/_tests/test_run.py
Outdated
assert RaisesGroup(KeyError, ValueError).matches(excinfo.value.__cause__) | ||
# TODO: triple-wrapped exceptions ?!?! | ||
assert RaisesGroup(RaisesGroup(RaisesGroup(KeyError, ValueError))).matches( | ||
excinfo.value.__cause__ | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason for the triple-wrapping turns out to be quite straightforward:
Lines 1903 to 1913 in aadd1ea
async def init( | |
self, | |
async_fn: Callable[[Unpack[PosArgT]], Awaitable[object]], | |
args: tuple[Unpack[PosArgT]], | |
) -> None: | |
# run_sync_soon task runs here: | |
async with open_nursery() as run_sync_soon_nursery: | |
# All other system tasks run here: | |
async with open_nursery() as self.system_nursery: | |
# Only the main task runs here: | |
async with open_nursery() as main_task_nursery: |
doesn't make for the cleanest __cause__
s on TrioInternalError
s, but don't think it's a problem. Updating comments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've been procrastinating on cleaning up the language in docstrings/documentation. But other than that I think this is the only remaining problem, where run_process
sometimes raises an exceptiongroup and sometimes doesn't.
There's maybe better solutions, but I'd just wrap the nursery call in a try/except and raise a new error with the caught exceptiongroup as a cause, as I described in the comment in _subprocess.py.
src/trio/_subprocess.py
Outdated
# TODO: if this nursery raises an exceptiongroup, manually raise a CalledProcessError, | ||
# or maybe a InternalTrioError, with the exceptiongroup as a cause, so run_process | ||
# never raises an exceptiongroup. (?) | ||
async with trio.open_nursery() as nursery: | ||
# options needs a complex TypedDict. The overload error only occurs on Unix. | ||
proc = await open_process(command, **options) # type: ignore[arg-type, call-overload, unused-ignore] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(surfacing from an ~hour-long dive into run_process()
) I think it's sufficiently unlikely to get an ExceptionGroup
here from any reasonable usage, that I'd favor raise InternalTrioError from the_group
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Getting multiple exceptions from inside the nursery may be unlikely, but it seems relatively straightforward to pass a bad command that will get an exception from open_process
- and we probably don't want to raise InternalTrioError
in those cases. Pushing a commit with a suggested fix, but I suspect there might be a reason the open_process
call was placed inside the nursery in the first place (?).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(the PermissionError
in the test is raised inside open_process
, and there's validation of inputs that run_process
punts to open_process
as well)
Co-authored-by: Zac Hatfield-Dodds <zac.hatfield.dodds@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thorough rewrite of docs and an eventual DeprecationWarning for setting strict_exception_groups=False
will have to wait for another PR. But other than uncertainty on messing with the nursery in open_process
I think everything has been resolved.
I think this PR should just allow an EG to propagate without the
TrioInternalError complication, and we should make a new issue for the
desired semantics of deliver cancel and EGs from this function.
…On Thu, Jan 11, 2024 at 10:56 AM John Litborn ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In src/trio/_subprocess.py
<#2886 (comment)>:
> - async with trio.open_nursery() as nursery:
- # options needs a complex TypedDict. The overload error only occurs on Unix.
- proc = await open_process(command, **options) # type: ignore[arg-type, call-overload, unused-ignore]
- try:
- if input is not None:
- assert proc.stdin is not None
- nursery.start_soon(feed_input, proc.stdin)
- proc.stdin = None
- proc.stdio = None
- if capture_stdout:
- assert proc.stdout is not None
- nursery.start_soon(read_output, proc.stdout, stdout_chunks)
- proc.stdout = None
- proc.stdio = None
- if capture_stderr:
- assert proc.stderr is not None
- nursery.start_soon(read_output, proc.stderr, stderr_chunks)
- proc.stderr = None
- task_status.started(proc)
- await proc.wait()
- except BaseException:
- with trio.CancelScope(shield=True):
- killer_cscope = trio.CancelScope(shield=True)
-
- async def killer() -> None:
- with killer_cscope:
- await deliver_cancel(proc)
-
- nursery.start_soon(killer)
+ # Opening the process does not need to be inside the nursery, so we put it outside
+ # so any exceptions get directly seen by users.
+ # options needs a complex TypedDict. The overload error only occurs on Unix.
+ proc = await open_process(command, **options) # type: ignore[arg-type, call-overload, unused-ignore]
+ try:
+ async with trio.open_nursery() as nursery:
+ try:
+ if input is not None:
+ assert proc.stdin is not None
+ nursery.start_soon(feed_input, proc.stdin)
+ proc.stdin = None
+ proc.stdio = None
+ if capture_stdout:
+ assert proc.stdout is not None
+ nursery.start_soon(read_output, proc.stdout, stdout_chunks)
+ proc.stdout = None
+ proc.stdio = None
+ if capture_stderr:
+ assert proc.stderr is not None
+ nursery.start_soon(read_output, proc.stderr, stderr_chunks)
+ proc.stderr = None
+ task_status.started(proc)
await proc.wait()
- killer_cscope.cancel()
- raise
+ except BaseException:
+ with trio.CancelScope(shield=True):
+ killer_cscope = trio.CancelScope(shield=True)
+
+ async def killer() -> None:
+ with killer_cscope:
+ await deliver_cancel(proc)
+
+ nursery.start_soon(killer)
+ await proc.wait()
+ killer_cscope.cancel()
+ raise
+ except BaseExceptionGroup as exc:
+ if all(isinstance(e, Cancelled) for e in exc.exceptions):
+ raise exc
+ raise TrioInternalError("Error interacting with opened process") from exc
but passing a bad deliver_cancel is indeed a very trivial way to get an
EG atm, hrm.
—
Reply to this email directly, view it on GitHub
<#2886 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB5BHRSFUG6MZUZCYQQ72OLYOADULAVCNFSM6AAAAAA7ZNHN56VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQMJVHEZDSMZZG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
oh and I meant the nursery in _subprocess line ~771
…On Thu, Jan 11, 2024 at 10:55 AM John Litborn ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In src/trio/_subprocess.py
<#2886 (comment)>:
> - async with trio.open_nursery() as nursery:
- # options needs a complex TypedDict. The overload error only occurs on Unix.
- proc = await open_process(command, **options) # type: ignore[arg-type, call-overload, unused-ignore]
- try:
- if input is not None:
- assert proc.stdin is not None
- nursery.start_soon(feed_input, proc.stdin)
- proc.stdin = None
- proc.stdio = None
- if capture_stdout:
- assert proc.stdout is not None
- nursery.start_soon(read_output, proc.stdout, stdout_chunks)
- proc.stdout = None
- proc.stdio = None
- if capture_stderr:
- assert proc.stderr is not None
- nursery.start_soon(read_output, proc.stderr, stderr_chunks)
- proc.stderr = None
- task_status.started(proc)
- await proc.wait()
- except BaseException:
- with trio.CancelScope(shield=True):
- killer_cscope = trio.CancelScope(shield=True)
-
- async def killer() -> None:
- with killer_cscope:
- await deliver_cancel(proc)
-
- nursery.start_soon(killer)
+ # Opening the process does not need to be inside the nursery, so we put it outside
+ # so any exceptions get directly seen by users.
+ # options needs a complex TypedDict. The overload error only occurs on Unix.
+ proc = await open_process(command, **options) # type: ignore[arg-type, call-overload, unused-ignore]
+ try:
+ async with trio.open_nursery() as nursery:
+ try:
+ if input is not None:
+ assert proc.stdin is not None
+ nursery.start_soon(feed_input, proc.stdin)
+ proc.stdin = None
+ proc.stdio = None
+ if capture_stdout:
+ assert proc.stdout is not None
+ nursery.start_soon(read_output, proc.stdout, stdout_chunks)
+ proc.stdout = None
+ proc.stdio = None
+ if capture_stderr:
+ assert proc.stderr is not None
+ nursery.start_soon(read_output, proc.stderr, stderr_chunks)
+ proc.stderr = None
+ task_status.started(proc)
await proc.wait()
- killer_cscope.cancel()
- raise
+ except BaseException:
+ with trio.CancelScope(shield=True):
+ killer_cscope = trio.CancelScope(shield=True)
+
+ async def killer() -> None:
+ with killer_cscope:
+ await deliver_cancel(proc)
+
+ nursery.start_soon(killer)
+ await proc.wait()
+ killer_cscope.cancel()
+ raise
+ except BaseExceptionGroup as exc:
+ if all(isinstance(e, Cancelled) for e in exc.exceptions):
+ raise exc
+ raise TrioInternalError("Error interacting with opened process") from exc
which nursery is "the nursery in question", the one in test_subprocess
L591, or the one in _subprocess line ~771?
If you're changing the former, you should get an EG, because it doesn't
set the global contect of strict_exception_groups=False, so the one on 771
will use the default of True (in this PR).
—
Reply to this email directly, view it on GitHub
<#2886 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB5BHRU7YSAURZWUL4P4HADYOADOHAVCNFSM6AAAAAA7ZNHN56VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQMJVHEZDKOJVHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
huh, then I'm confused. |
I added a test which is doing what you described, afaict, and it's working as expected. |
Any remaining semi-blockers have been cleared for this:
So I'll rebump review request from @A5rocks and/or @TeamSpen210. But unless @Zac-HD finds anything else on a re-review I think this can be merged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! Since this PR has been open and at times growing for two months, I'm going to merge, and open an issue for the docs followup mentioned above. Post-merge reviews entirely welcome; I suggest comments here if you want to discuss or just open an issue for some followup action if you spot something we should change further.
Huge thanks to @jakkdl for all your work on this, and to @A5rocks, @CoolCat467, @richardsheridan, and @TeamSpen210 for reviews 🤩 I'm really looking forward to getting this into production at work, and seeing the ecosystem improve handling of concurrent errors!
this commit ... is a monster. I could try to split it up some things into separate PRs, but a lot of it is hard to separate.
MultiError
and setstrict_exception_groups=True
by default #2785, which this fixes most of.pytest.raises
that handlesexceptiongroup
s. This should probably be added upstream, or in a separate package, or do a way less illegal way of interaction.mypy
also goes mad on it, as it should. Don't focus on reviewing_exceptiongroup_util.py
for now. See Allow pytest.raises cooperate with ExceptionGroups pytest-dev/pytest#11538strict_exception_groups=False
, but decided that embracing the future is much better in the long term and it unveiled a bunch of issues._assert_raises
has been removed. It seems to be a relic with no reason for existingExpectedExceptionGroup
, and have been simplified.MultiError
and changing nurseries etc to raiseExceptionGroup
s instead to a separate PR. But changed some tests to check for identity withExceptionGroup
instead ofMultiError
. (ignore the name of the branch >.>)This is a pretty big change, and touches on lots and lots of stuff, and should figure out a solution for
raises
/ExpectedExceptionGroup
, so I expect this to stay as a draft for quite a while - maybe with other PRs splitting out from it and implementing parts of it or fixing weird [non-]wrapping.