Run background tasks in a context manager

[joernheissler]

Hi,
many network protocols are multiplexed by nature, i.e. multiple logical connections share one physical connection. Examples include http2 (multiple requests in parallel), ssh (e.g. port forwardings), irc (chat with multiple persons).
Some background task is required which reads data from the physical connection and distributes it to the handler of the logical connection.

I'd like to see an easy way to create code like this:

async with Http2Client("server.example.net") as http:
    async with trio.open_nursery() as nursery:
        for path in {'/foo', '/bar', '/baz'}:
            nursery.start_soon(http.get, path)  # GET result is unused, for sake of brevity.

The Http2Client context manager would create a nursery and start a background task which reads data from the http server and handles it somehow.
The get method would send a request over the multiplexed connection and wait for the reply. The background task reads a reply from the server and sees e.g. "It's for the baz request" and then delivers it to the http.get foreground task for baz which will then return the result to the user.

As soon as the body of the context manager finishes, the background job is notified. In above example, it would cleanly shut down the http2 and TLS connections.

I wrote PoC code to create a simple IRC bot which talks to multiple people and echos everything they say back at them: https://gist.github.com/joernheissler/571f95e974ade2e5610dad9417ef8d49
(Needs python3.8; If you run it, please change the nicks)

Is that something which trio should have? And how should the API look like?

[belm0]

See also #467

[njsmith]

Yeah, this is definitely a common pattern. trio-websocket is a fairly mature example, and some relevant links are:

• https://stackoverflow.com/a/48300537
• Revisit the "optional nursery" pattern trio-websocket#31

I'm not sure whether it makes sense for Trio to include some kind of helpers for this pattern, or not. Mostly I'm just not sure what those helpers would look like, or if it's possible to make ones that are useful and general enough to include :-). A basic implementation is just like:

@asynccontextmanager
async def open_irc(host, port, *, use_ssl=True, other_option=...):
    if use_ssl:
        stream = await trio.open_ssl_over_tcp_stream(host, port)
    else:
        stream = await trio.open_tcp_stream(host, port)
    async with trio.open_nursery() as nursery:
        yield IRCClient(nursery, stream, other_option=other_option, ...)
        nursery.cancel_scope.cancel()

That's short, and has a pretty high-density of protocol-specific stuff.

[oremanj]

I think this is redundant with #467 in terms of next-actions, so closing.