asyncio: Re-reverse deprecation of set_event_loop?

[bdarnell]

Feature or enhancement

Proposal:

The asyncio policy system is deprecated in python 3.14 (#127949). As implemented, this includes the asyncio.set_event_loop() function. However, in 2022, it was decided that set_event_loop and get_event_loop (just the thread-local storage, not the broader policy system) were serving a useful and separate purpose and should be kept around. I relied on this decision in Tornado to adapt to various changes while staying on interfaces that I thought would be safe from deprecation.

I would like to briefly bring this up for reconsideration since this is a reversal of a decision from just a few years ago and it appears to have been lumped in with the rest of the policy system without consideration on its own.

If the decision to deprecate this function stands, I'll be able to adapt in Tornado, but it will be inconvenient: I think I'll have to use asyncio.Runner which was introduced in 3.11, while I'm still supporting 3.9 and 3.10 for another couple of years.

Background

Here's the history as I remember it. In Python 3.10, the policy system was deprecated and slated for removal in 3.12. When the 3.12 alphas were released with the policy methods removed, we found that there were code paths that relied on the deprecated methods without emitting suitable warnings, leading to surprise breakages. This caused everything to be reset (in 3.10.9 and 3.11.1, which is why those specific versions are cited in the docs). Deprecation notices were removed, but the understanding at the time was that this was just resetting the clock and policies would still be going away, just a few years later.

Separately, there was a debate about the set_event_loop function, although I can't find the right thread now. (#83710 (comment) and #98440 (comment) are related, but I can't find where the actual conclusion was reached). I think Jupyter was another project where set_event_loop turned out to be important?

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

[gvanrossum]

@kumaraditya303 I am fine with keeping these, assuming there are no major implementation hurdles or terrible flaws.

[kumaraditya303]

In previous versions asyncio.set_event_loop was a shorthand used to perform two things:

• set the current loop (may not be running!) in the policy's thread local state
• attach the current child watcher to the provided loop

In current Python 3.14, the child watchers have been removed entirely and policy is deprecated as such asyncio.set_event_loop is just a deprecated alias for policy's set_event_loop hence is deprecated.

FWIW, unlike asyncio.get_event_loop which in future would become alias to asyncio.get_running_loop, asyncio.set_event_loop is a highly misleading function because when policies will be removed, there will be no concept of current but not running event loop hence it was deprecated together with policies.

[gvanrossum]

Okay, then what does Tornado have to do? Just have its own per-thread datastructure to hold its preferred event loop?

[bdarnell]

there will be no concept of current but not running event loop hence it was deprecated together with policies.

This is where we differ - to me, policies concern the creation of an event loop, and the concept of a current-but-not-running event loop is something separate (which was located in the event loop policy as an implementation detail). I found a better thread where I laid all this out (and at the time @gvanrossum agreed): #100160 (comment)

Okay, then what does Tornado have to do? Just have its own per-thread datastructure to hold its preferred event loop?

That works as long as the application only uses Tornado's interfaces. Hybrid apps that mix Tornado with, for instance, Twisted are pretty rare (Glyph and I care about this but I'm not sure anyone else does). The problem is that because Tornado uses asyncio interfaces internally (and they are increasingly part of our documented best practices), mixing and matching is more of a concern.

Maybe this doesn't matter, though? I need to do some more testing but I don't currently have any examples of something we use that works today in this not-yet-running state but is likely to break in the future. The things that are important to us are not higher-level interfaces (which generally require an async context) but the low-level ones like call_soon, call_later, add_reader, add_writer, etc. As long as those continue to work on an event loop that is "current but not running" we may be OK with a thread-local on the tornado side (although I don't like the idea of going back to that and diverging from asyncio)

For more context, this comes up in two different places. First is at application startup time, when the Tornado convention was to start your server (adding an event handler to the listening socket) before starting the event loop. This part I'm not as worried about because even if it's an issue, it's by definition something that can be fixed once per app, and it's fairly straightforward. The trickier part is in tests. The still-predominant testing style in Tornado web apps doesn't use any coroutines in the tests, but it all runs in synchronous mode with the application server and HTTP client started before the event loop (and then we run the loop until the HTTP fetch is complete and stop it again). If this stopped working it would be very tedious to refactor all the tests that rely on it.

[kumaraditya303]

This is where we differ - to me, policies concern the creation of an event loop, and the concept of a current-but-not-running event loop is something separate (which was located in the event loop policy as an implementation detail).

Even if you treat is an implementation detail, it was always a part of policy system so it was deprecated with it.

The things that are important to us are not higher-level interfaces (which generally require an async context) but the low-level ones like call_soon, call_later, add_reader, add_writer, etc.

All of those functions would continue to work as they work currently, they are unaffected by deprecation of policy system.

In general, it would be better to use asyncio.Runner APIs for versions that support it as that would handle everything for you.

I am closing this as this is the intended behaviour here for reasons I described here #130322 (comment)

[gvanrossum]

I believe that the module-level get_event_loop and set_event_loop could easily be kept around, using module-level thread-local state to store the "current" event loop. This is what I take the '22 decision quoted by @bdarnell proposed to do. (Note that get_event_loop would no longer create an event loop, ever.) And we apparently never deprecated set_event_loop. I disagree that it is part of the policy system -- it just used the current policy when the policy wasn't fixed.

However, the plan to change get_event_loop to become an alias for get_running_loop would get in the way of such an implementation. Was this plan (to make it an alias) ever documented? I can't find it in the 3.13 or 3.14 docs for the module-level get_event_loop -- it still mentions get_event_loop_policy().get_event_loop().

The alternative would be for Tornado and other frameworks that have their own equivalent of asyncio.run() to write their own {get,set}_event_loop using thread-local storage inside Tornado, but this might not support all intended use cases (maybe Tornado's architecture allows sharing the current event loop with other frameworks?).

Let's keep this issue open until we have more of an agreement on the way forward.

[bdarnell]

I believe that the module-level get_event_loop and set_event_loop could easily be kept around, using module-level thread-local state to store the "current" event loop. This is what I take the '22 decision quoted by @bdarnell proposed to do.

That is how I understood it as well.

(Note that get_event_loop would no longer create an event loop, ever.)

Yes, and I support this part of the changes.

However, the plan to change get_event_loop to become an alias for get_running_loop would get in the way of such an implementation. Was this plan (to make it an alias) ever documented?

I don't think it was ever explicitly documented because it's kind of implied by the removal of set_event_loop. Without either set_event_loop or policies, there's no reason for get_event_loop and get_running_loop to differ. (You and I) have each commented on merging get_event_loop with get_running_loop in a world without set_event_loop).

If we keep set_event_loop, we obviously need some way to get that saved event loop. This means either A) get_event_loop is not an alias for get_running_loop (it should return the running loop if there is one, or the saved loop if none is running) or B) get_event_loop is an alias for get_running_loop and a separate get_saved_loop (or some other name) gives you the saved loop.

write their own {get,set}_event_loop using thread-local storage inside Tornado, but this might not support all intended use cases (maybe Tornado's architecture allows sharing the current event loop with other frameworks?).

Yes, the ability to share the saved event loop across frameworks is why we'd prefer this to be asyncio.set_event_loop and not tornado.set_event_loop. Mixing different frameworks may sound like a theoretical concern (Glyph and I sometimes build Tornado/Twisted hybrids but we're not representative of the broader community), but this turns out to be important because every Tornado app is effectively a Tornado/asyncio hybrid due to Tornado's internal use of asyncio primitives (and, sometimes, due to code in the application layer using asyncio directly).

We need to be able to construct asyncio objects while no event loop is running. In particular, that includes asyncio.Future which is bound to an event loop at init time. Today that works with a one-time call to asyncio.set_event_loop. If we lost that facility at the asyncio level (and had an equivalent thread-local in Tornado), we'd have to introduce a wrapper function and replace all occurrences of asyncio.Future() with a function that does `return asyncio.Future(loop=tornado.thread_local_event_loop())

[gvanrossum]

It seems to me that Ben and I are in complete agreement on this issue. @kumaraditya303, are you willing to reinstate get/set_event_loop with the semantics of (eventually) storing the loop in an asyncio-wide thread-local variable? We wouldn't have to do that until policies are actually ripped out. It would be a simple change, and there's time to undeprecate the parts that will remain public APIs before 3.14 beta1 rolls around.

[kumaraditya303]

However, the plan to change get_event_loop to become an alias for get_running_loop would get in the way of such an implementation. Was this plan (to make it an alias) ever documented? I can't find it in the 3.13 or 3.14 docs for the module-level get_event_loop -- it still mentions get_event_loop_policy().get_event_loop().

@gvanrossum:
See https://docs.python.org/3.14/library/asyncio-eventloop.html#asyncio.get_event_loop
There's a note at the end of the docs of this function which mentions this.

We need to be able to construct asyncio objects while no event loop is running. In particular, that includes asyncio.Future which is bound to an event loop at init time. Today that works with a one-time call to asyncio.set_event_loop. If we lost that facility at the asyncio level (and had an equivalent thread-local in Tornado), we'd have to introduce a wrapper function and replace all occurrences of asyncio.Future() with a function that does `return asyncio.Future(loop=tornado.thread_local_event_loop())

That will not work, even if set_event_loop is changed to store loop in thread local, such use of asyncio.Future will not work because once the loop of an Future is set it cannot be changed i.e. asyncio.Future's loop cannot be changed by just calling set_event_loop, it will continue to use the old set loop. Creating futures and tasks by directly calling asyncio.Future or asyncio.Task is a bad idea, it has always been recommended to use instead loop.create_future

If the decision to deprecate this function stands, I'll be able to adapt in Tornado, but it will be inconvenient: I think I'll have to use asyncio.Runner which was introduced in 3.11, while I'm still supporting 3.9 and 3.10 for another couple of years.

I would suggest you to try to adapt tornado to use runner APIs first, I would like to see concrete code which you cannot migrate to runner API if any before proceeding with any rolling back of deprecation.

[kumaraditya303]

are you willing to reinstate get/set_event_loop with the semantics of (eventually) storing the loop in an asyncio-wide thread-local variable? We wouldn't have to do that until policies are actually ripped out. It would be a simple change, and there's time to undeprecate the parts that will remain public APIs before 3.14 beta1 rolls around.

I am not a fan of adding more thread local storage and continuing this weird state of "set loop but not running thing" even if it is thread specific in asyncio. I am -1 on that for now unless we have no other way to fix this.

This can be solved in tornado itself by relying on the use of current event loop and not calling asyncio.set_event_loop at all.
If it needs to override the event loop or attach some data to maintain thread local mapping loop -> tornado loop state, they can use the loop_factory to do that without resorting to policy and set_event_loop.

Sample code:

loop_data = {}
def loop_factory():
    loop = asyncio.EventLoop()
    tornado_data = ...
    loop_data[loop] = tornado_data
    return data
    

asyncio.run(coro, loop_factory=tornado.loop_factory)

[bdarnell]

That will not work, even if set_event_loop is changed to store loop in thread local, such use of asyncio.Future will not work because once the loop of an Future is set it cannot be changed i.e. asyncio.Future's loop cannot be changed by just calling set_event_loop, it will continue to use the old set loop.

Correct, but what I meant was to call set_event_loop early, before any Futures are created.

Creating futures and tasks by directly calling asyncio.Future or asyncio.Task is a bad idea, it has always been recommended to use instead loop.create_future

Not always - loop.create_future was new in 3.5.2. And the Tornado patterns we're talking about long predate asyncio - we had our own Future class that was not bound to the event loop in this way, and merged it into asyncio's Future with the expectation that its interfaces and behavior would be stable.

But acknowledged that if we go this route the wrapper should call get_event_loop().create_future() instead of Future(loop=...).

I would suggest you to try to adapt tornado to use runner APIs first

There are two problems with runner from my perspective: First is just that it's too new - I still support versions of python that don't have it. But the bigger problem is the design of Tornado's AsyncHTTPTestCase. This is ancient code that predates coroutines but was never deprecated and is still the recommended way to write tests of Tornado apps. It simulates coroutines with synchronous methods that run the event loop for you. For example:

def test_search(self):
    do_some_setup()
    response = self.fetch("/search?q=foo")
    assert_stuff(response)

self.fetch is basically equivalent to return self.loop.run_until_complete(self.http_client.fetch(url)). This requires that the event loop is not currently running. But do_some_setup may do some things that depend on the existence of an event loop. As long as do_some_setup is going through Tornado interfaces, we can make it work with a tornado-specific thread-local event loop. But when it goes through asyncio interfaces like asyncio.Future(), it'll break if we don't keep set_event_loop.

So if we lose asyncio.set_event_loop, Tornado's plan would be:

1. Introduce a tornado-specific threading.local for the event loop and use it wherever we currently access the event loop (not that hard, we already have all of this fairly centralized from the pre-asyncio days)
2. Change all Future creation in Tornado itself to use loop.create_future() on the event loop (which can be the thread-local one)
3. Some unknown amount of application code will be broken by this too and they'll need to adapt themselves when they upgrade python.

This can be solved in tornado itself by relying on the use of current event loop and not calling asyncio.set_event_loop at all.

This is a bit of a digression since our issue does not involve attaching information to the event loop, but I just want to emphasize the limitations of solving things "in tornado itself". Tornado is a framework; it does not call asyncio.run() on behalf of applications. So even if this loop_factory were a solution to our problem, this would mean a change to every tornado application, not a once-and-for-all solution in tornado itself. This is why I'm pushing for backwards-compatibility even at the expense of maintaining an arguably impure interface indefinitely.