Merge pull request #2213 from python-trio/exceptiongroup

Replace MultiError with BaseExceptionGroup

docs-requirements.in

@@ -16,6 +16,7 @@ async_generator >= 1.9
 idna
 outcome
 sniffio
+exceptiongroup >= 1.0.0rc9
 
 # See note in test-requirements.in
 immutables >= 0.6

docs-requirements.txt

@@ -1,5 +1,5 @@
 #
-# This file is autogenerated by pip-compile with python 3.10
+# This file is autogenerated by pip-compile with python 3.7
 # To update, run:
 #
 #    pip-compile docs-requirements.in
@@ -28,6 +28,8 @@ docutils==0.17.1
     # via
     #   sphinx
     #   sphinx-rtd-theme
+exceptiongroup==1.0.0rc9
+    # via -r docs-requirements.in
 idna==3.4
     # via
     #   -r docs-requirements.in
@@ -36,6 +38,8 @@ imagesize==1.4.1
     # via sphinx
 immutables==0.18
     # via -r docs-requirements.in
+importlib-metadata==4.12.0
+    # via click
 incremental==21.3.0
     # via towncrier
 jinja2==3.0.3
@@ -88,8 +92,14 @@ tomli==2.0.1
     # via towncrier
 towncrier==22.8.0
     # via -r docs-requirements.in
+typing-extensions==4.3.0
+    # via
+    #   immutables
+    #   importlib-metadata
 urllib3==1.26.12
     # via requests
+zipp==3.8.1
+    # via importlib-metadata
 
 # The following packages are considered to be unsafe in a requirements file:
 # setuptools

docs/source/conf.py

@@ -84,8 +84,9 @@ def setup(app):
     'local_customization',
 ]
 
+# FIXME: change the "python" link back to /3 when Python 3.11 is released
 intersphinx_mapping = {
-    "python": ('https://docs.python.org/3', None),
+    "python": ('https://docs.python.org/3.11', None),
     "outcome": ('https://outcome.readthedocs.io/en/latest/', None),
     "pyopenssl": ('https://www.pyopenssl.org/en/stable/', None),
 }

docs/source/design.rst

@@ -461,15 +461,9 @@ of our public APIs without having to modify Trio internals.
 Inside ``trio._core``
 ~~~~~~~~~~~~~~~~~~~~~
 
-There are two notable sub-modules that are largely independent of
-the rest of Trio, and could (possibly should?) be extracted into their
-own independent packages:
-
-* ``_multierror.py``: Implements :class:`MultiError` and associated
-  infrastructure.
-
-* ``_ki.py``: Implements the core infrastructure for safe handling of
-  :class:`KeyboardInterrupt`.
+The ``_ki.py`` module implements the core infrastructure for safe handling
+of :class:`KeyboardInterrupt`.  It's largely independent of the rest of Trio,
+and could (possibly should?) be extracted into its own independent package.
 
 The most important submodule, where everything is integrated, is
 ``_run.py``. (This is also by far the largest submodule; it'd be nice

docs/source/history.rst

@@ -48,8 +48,8 @@ Features
 Bugfixes
 ~~~~~~~~
 
-- Trio now avoids creating cyclic garbage when a `MultiError` is generated and filtered,
-  including invisibly within the cancellation system.  This means errors raised
+- Trio now avoids creating cyclic garbage when a ``MultiError`` is generated and
+  filtered, including invisibly within the cancellation system. This means errors raised
   through nurseries and cancel scopes should result in less GC latency. (`#2063 <https://github.com/python-trio/trio/issues/2063>`__)
 - Trio now deterministically cleans up file descriptors that were opened before
   subprocess creation fails. Previously, they would remain open until the next run of
@@ -288,9 +288,9 @@ Bugfixes
 - On Ubuntu systems, the system Python includes a custom
   unhandled-exception hook to perform `crash reporting
   <https://wiki.ubuntu.com/Apport>`__. Unfortunately, Trio wants to use
-  the same hook to print nice `MultiError` tracebacks, causing a
+  the same hook to print nice ``MultiError`` tracebacks, causing a
   conflict. Previously, Trio would detect the conflict, print a warning,
-  and you just wouldn't get nice `MultiError` tracebacks. Now, Trio has
+  and you just wouldn't get nice ``MultiError`` tracebacks. Now, Trio has
   gotten clever enough to integrate its hook with Ubuntu's, so the two
   systems should Just Work together. (`#1065 <https://github.com/python-trio/trio/issues/1065>`__)
 - Fixed an over-strict test that caused failures on Alpine Linux.
@@ -492,7 +492,7 @@ Features
   violated. (One common source of such violations is an async generator
   that yields within a cancel scope.) The previous behavior was an
   inscrutable chain of TrioInternalErrors. (`#882 <https://github.com/python-trio/trio/issues/882>`__)
-- MultiError now defines its ``exceptions`` attribute in ``__init__()``
+- ``MultiError`` now defines its ``exceptions`` attribute in ``__init__()``
   to better support linters and code autocompletion. (`#1066 <https://github.com/python-trio/trio/issues/1066>`__)
 - Use ``__slots__`` in more places internally, which should make Trio slightly faster. (`#984 <https://github.com/python-trio/trio/issues/984>`__)
 
@@ -513,7 +513,7 @@ Bugfixes
   :meth:`~trio.Path.cwd`, are now async functions.  Previously, a bug
   in the forwarding logic meant :meth:`~trio.Path.cwd` was synchronous
   and :meth:`~trio.Path.home` didn't work at all. (`#960 <https://github.com/python-trio/trio/issues/960>`__)
-- An exception encapsulated within a :class:`MultiError` doesn't need to be
+- An exception encapsulated within a ``MultiError`` doesn't need to be
   hashable anymore.
 
   .. note::
@@ -1304,7 +1304,7 @@ Other changes
   interfering with direct use of
   :func:`~trio.testing.wait_all_tasks_blocked` in the same test.
 
-* :meth:`MultiError.catch` now correctly preserves ``__context__``,
+* ``MultiError.catch()`` now correctly preserves ``__context__``,
   despite Python's best attempts to stop us (`#165
   <https://github.com/python-trio/trio/issues/165>`__)
 

docs/source/reference-core.rst

@@ -641,7 +641,7 @@ crucial things to keep in mind:
 
   * Any unhandled exceptions are re-raised inside the parent task. If
     there are multiple exceptions, then they're collected up into a
-    single :exc:`MultiError` exception.
+    single :exc:`BaseExceptionGroup` or :exc:`ExceptionGroup` exception.
 
 Since all tasks are descendents of the initial task, one consequence
 of this is that :func:`run` can't finish until all tasks have
@@ -687,6 +687,8 @@ You might wonder why Trio can't just remember "this task should be cancelled in
 
 If you want a timeout to apply to one task but not another, then you need to put the cancel scope in that individual task's function -- ``child()``, in this example.
 
+.. _exceptiongroups:
+
 Errors in multiple child tasks
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -709,18 +711,102 @@ limitation. Consider code like::
 
 ``broken1`` raises ``KeyError``. ``broken2`` raises
 ``IndexError``. Obviously ``parent`` should raise some error, but
-what? In some sense, the answer should be "both of these at once", but
-in Python there can only be one exception at a time.
+what? The answer is that both exceptions are grouped in an :exc:`ExceptionGroup`.
+:exc:`ExceptionGroup` and its parent class :exc:`BaseExceptionGroup` are used to
+encapsulate multiple exceptions being raised at once.
+
+To catch individual exceptions encapsulated in an exception group, the ``except*``
+clause was introduced in Python 3.11 (:pep:`654`). Here's how it works::
+
+    try:
+        async with trio.open_nursery() as nursery:
+            nursery.start_soon(broken1)
+            nursery.start_soon(broken2)
+    except* KeyError as excgroup:
+        for exc in excgroup.exceptions:
+            ...  # handle each KeyError
+    except* IndexError as excgroup:
+        for exc in excgroup.exceptions:
+            ...  # handle each IndexError
+
+If you want to reraise exceptions, or raise new ones, you can do so, but be aware that
+exceptions raised in ``except*`` sections will be raised together in a new exception
+group.
+
+But what if you can't use ``except*`` just yet? Well, for that there is the handy
+exceptiongroup_ library which lets you approximate this behavior with exception handler
+callbacks::
+
+    from exceptiongroup import catch
+
+    def handle_keyerrors(excgroup):
+        for exc in excgroup.exceptions:
+            ...  # handle each KeyError
+
+    def handle_indexerrors(excgroup):
+        for exc in excgroup.exceptions:
+            ...  # handle each IndexError
+
+    with catch({
+        KeyError: handle_keyerror,
+        IndexError: handle_indexerror
+    }):
+        async with trio.open_nursery() as nursery:
+            nursery.start_soon(broken1)
+            nursery.start_soon(broken2)
 
-Trio's answer is that it raises a :exc:`MultiError` object. This is a
-special exception which encapsulates multiple exception objects –
-either regular exceptions or nested :exc:`MultiError`\s. To make these
-easier to work with, Trio installs a custom `sys.excepthook` that
-knows how to print nice tracebacks for unhandled :exc:`MultiError`\s,
-and it also provides some helpful utilities like
-:meth:`MultiError.catch`, which allows you to catch "part of" a
-:exc:`MultiError`.
+The semantics for the handler functions are equal to ``except*`` blocks, except for
+setting local variables. If you need to set local variables, you need to declare them
+inside the handler function(s) with the ``nonlocal`` keyword::
 
+    def handle_keyerrors(excgroup):
+        nonlocal myflag
+        myflag = True
+
+    myflag = False
+    with catch({KeyError: handle_keyerror}):
+        async with trio.open_nursery() as nursery:
+            nursery.start_soon(broken1)
+
+For reasons of backwards compatibility, nurseries raise ``trio.MultiError`` and
+``trio.NonBaseMultiError`` which inherit from :exc:`BaseExceptionGroup` and
+:exc:`ExceptionGroup`, respectively. Users should refrain from attempting to raise or
+catch the Trio specific exceptions themselves, and treat them as if they were standard
+:exc:`BaseExceptionGroup` or :exc:`ExceptionGroup` instances instead.
+
+"Strict" versus "loose" ExceptionGroup semantics
+++++++++++++++++++++++++++++++++++++++++++++++++
+
+Ideally, in some abstract sense we'd want everything that *can* raise an
+`ExceptionGroup` to *always* raise an `ExceptionGroup` (rather than, say, a single
+`ValueError`). Otherwise, it would be easy to accidentally write something like ``except
+ValueError:`` (not ``except*``), which works if a single exception is raised but fails to
+catch _anything_ in the case of multiple simultaneous exceptions (even if one of them is
+a ValueError). However, this is not how Trio worked in the past: as a concession to
+practicality when the ``except*`` syntax hadn't been dreamed up yet, the old
+``trio.MultiError`` was raised only when at least two exceptions occurred
+simultaneously. Adding a layer of `ExceptionGroup` around every nursery, while
+theoretically appealing, would probably break a lot of existing code in practice.
+
+Therefore, we've chosen to gate the newer, "stricter" behavior behind a parameter
+called ``strict_exception_groups``. This is accepted as a parameter to
+:func:`open_nursery`, to set the behavior for that nursery, and to :func:`trio.run`,
+to set the default behavior for any nursery in your program that doesn't override it.
+
+* With ``strict_exception_groups=True``, the exception(s) coming out of a nursery will
+  always be wrapped in an `ExceptionGroup`, so you'll know that if you're handling
+  single errors correctly, multiple simultaneous errors will work as well.
+
+* With ``strict_exception_groups=False``, a nursery in which only one task has failed
+  will raise that task's exception without an additional layer of `ExceptionGroup`
+  wrapping, so you'll get maximum compatibility with code that was written to
+  support older versions of Trio.
+
+To maintain backwards compatibility, the default is ``strict_exception_groups=False``.
+The default will eventually change to ``True`` in a future version of Trio, once
+Python 3.11 and later versions are in wide use.
+
+.. _exceptiongroup: https://pypi.org/project/exceptiongroup/
 
 Spawning tasks without becoming a parent
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -837,104 +923,6 @@ The nursery API
    See :meth:`~Nursery.start`.
 
 
-Working with :exc:`MultiError`\s
-++++++++++++++++++++++++++++++++
-
-.. autoexception:: MultiError
-
-   .. attribute:: exceptions
-
-      The list of exception objects that this :exc:`MultiError`
-      represents.
-
-   .. automethod:: filter
-
-   .. automethod:: catch
-      :with:
-
-Examples:
-
-Suppose we have a handler function that discards :exc:`ValueError`\s::
-
-    def handle_ValueError(exc):
-        if isinstance(exc, ValueError):
-            return None
-        else:
-            return exc
-
-Then these both raise :exc:`KeyError`::
-
-    with MultiError.catch(handle_ValueError):
-         raise MultiError([KeyError(), ValueError()])
-
-    with MultiError.catch(handle_ValueError):
-         raise MultiError([
-             ValueError(),
-             MultiError([KeyError(), ValueError()]),
-         ])
-
-And both of these raise nothing at all::
-
-    with MultiError.catch(handle_ValueError):
-         raise MultiError([ValueError(), ValueError()])
-
-    with MultiError.catch(handle_ValueError):
-         raise MultiError([
-             MultiError([ValueError(), ValueError()]),
-             ValueError(),
-         ])
-
-You can also return a new or modified exception, for example::
-
-    def convert_ValueError_to_MyCustomError(exc):
-        if isinstance(exc, ValueError):
-            # Similar to 'raise MyCustomError from exc'
-            new_exc = MyCustomError(...)
-            new_exc.__cause__ = exc
-            return new_exc
-        else:
-            return exc
-
-In the example above, we set ``__cause__`` as a form of explicit
-context chaining. :meth:`MultiError.filter` and
-:meth:`MultiError.catch` also perform implicit exception chaining – if
-you return a new exception object, then the new object's
-``__context__`` attribute will automatically be set to the original
-exception.
-
-We also monkey patch :class:`traceback.TracebackException` to be able
-to handle formatting :exc:`MultiError`\s. This means that anything that
-formats exception messages like :mod:`logging` will work out of the
-box::
-
-    import logging
-
-    logging.basicConfig()
-
-    try:
-        raise MultiError([ValueError("foo"), KeyError("bar")])
-    except:
-        logging.exception("Oh no!")
-        raise
-
-Will properly log the inner exceptions:
-
-.. code-block:: none
-
-    ERROR:root:Oh no!
-    Traceback (most recent call last):
-      File "<stdin>", line 2, in <module>
-    trio.MultiError: ValueError('foo',), KeyError('bar',)
-
-    Details of embedded exception 1:
-
-      ValueError: foo
-
-    Details of embedded exception 2:
-
-      KeyError: 'bar'
-
-
 .. _task-local-storage:
 
 Task-local storage

docs/source/tutorial.rst

@@ -13,14 +13,13 @@ Tutorial
    still probably read this, because Trio is different.)
 
    Trio turns Python into a concurrent language. It takes the core
-   async/await syntax introduced in 3.5, and uses it to add three
+   async/await syntax introduced in 3.5, and uses it to add two
    new pieces of semantics:
 
    - cancel scopes: a generic system for managing timeouts and
      cancellation
    - nurseries: which let your program do multiple things at the same
      time
-   - MultiErrors: for when multiple things go wrong at once
 
    Of course it also provides a complete suite of APIs for doing
    networking, file I/O, using worker threads,
@@ -57,8 +56,6 @@ Tutorial
    and demonstrate start()
    then point out that you can just use serve_tcp()
 
-   exceptions and MultiError
-
    example: catch-all logging in our echo server
 
    review of the three (or four) core language extensions
@@ -1149,9 +1146,6 @@ TODO: explain :exc:`Cancelled`
 TODO: explain how cancellation is also used when one child raises an
 exception
 
-TODO: show an example :exc:`MultiError` traceback and walk through its
-structure
-
 TODO: maybe a brief discussion of :exc:`KeyboardInterrupt` handling?
 
 ..