Coroutine with asyncio.sleep() hangs when run in a different thread

[mikeyhew]

Here is my code:

import asyncio
from threading import Thread


async def onRequest(data):
    print('got request: {}'.format(data))
    await asyncio.sleep(1)
    print('finished processing request {}'.format(data))

loop = asyncio.get_event_loop()
loop.set_debug(True)

def child_executor():
    loop.run_forever()

child = Thread(target=child_executor, name="child")

def request(data):
    asyncio.run_coroutine_threadsafe(onRequest(data), loop)

child.start()

request(4)

This produces the following output:

got request: 4

and then it just hangs, without the 'finished processing request' message. If I press ctrl^C, it outputs the following message, which doesn't look very interesting:

Exception ignored in: <module 'threading' from '/home/ubuntu/lib/python3.5/threading.py'>
Traceback (most recent call last):
  File "/home/ubuntu/lib/python3.5/threading.py", line 1288, in _shutdown
    t.join()
  File "/home/ubuntu/lib/python3.5/threading.py", line 1054, in join
    self._wait_for_tstate_lock()
  File "/home/ubuntu/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

[asvetlov]

Event loop is coupled to thread. You cannot share the loop between different threads, except call_soon_threadsafe()/run_coroutine_threadsafe() calls.

See https://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading

You can have a loop per thread though.

[mikeyhew]

Oh right, I think I remember reading a while back that asyncio.get_event_loop() returned the main event loop, which can only be run from the main thread. If I want to run an event loop in a child thread, do I have to create it with asyncio.new_event_loop() from the child thread, or can I create it in the main thread first and run it in the child thread?

[mikeyhew]

Assuming that the child thread has to create the event loop, here is updated code that creates a new event loop and runs it in the child thread. It still has the same hanging problem as before.

import asyncio
from threading import Thread, Semaphore

async def onRequest(data):
    print('got request: {}'.format(data))
    await asyncio.sleep(1)
    print('finished processing request: {}'.format(data))

loop_created_sem = Semaphore(0)

def child_executor():
    global loop
    loop = asyncio.new_event_loop()

    # signal to the main thread that we've created the event loop
    loop_created_sem.release()

    loop.run_forever()

child = Thread(target=child_executor, name="child")

def request(data):
    global loop
    asyncio.run_coroutine_threadsafe(onRequest(data), loop)

child.start()

# wait for the child thread to create its event loop
# before we advance
loop_created_sem.acquire()

request(4)

Expected Output:

got request: 4
finished processing request: 4

Output (control^C pressed after 10 seconds)

got request: 4
^CException ignored in: <module 'threading' from '/home/ubuntu/lib/python3.5/threading.py'>
Traceback (most recent call last):
  File "/home/ubuntu/lib/python3.5/threading.py", line 1288, in _shutdown
    t.join()
  File "/home/ubuntu/lib/python3.5/threading.py", line 1054, in join
    self._wait_for_tstate_lock()
  File "/home/ubuntu/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

[asvetlov]

That's because you don't pass explicit event loop to asyncio.sleep(1, loop=loop).

As an option you may setup thread-local loop:

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

[rudyryk]

I think it's one of most unclear things about asyncio :) Just started a discussion issue #332 related to that.

[mpaolini]

see also bugs.python.org/issue26969

[mpaolini]

py.test asyncio plugin has a feature that detects the frowned-upon use of the default event loop

[asvetlov]

Honestly I recommend to newer use implicit event loop in your code but always pass loop=loop into every asyncio API call which accepts the loop.

Moreover global implicit loop should be disabled by asyncio.set_event_loop(None) call.
This is the only way to write 100% correct code.

[gjcarneiro]

On 18 July 2016 at 22:46, Andrew Svetlov notifications@github.com wrote:

Honestly I recommend to newer use implicit event loop in your code but
always pass loop=loop into every asyncio API call which accepts the loop.

Moreover global implicit loop should be disabled by
asyncio.set_event_loop(None) call.
This is the only way to write 100% correct code.

That seems an exaggeration. So if you don't add loop=loop parameters
everywhere, that code still works with the default loop. It just means
that asyncio code does not support being run in a thread, it's not the end
of the world. Just document it.

Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert

[Martiusweb]

As I understand it, one should set a customized event loop policy globally,
which should be able to return the right loop according to the context. For
instance, in a multithreaded context, get_event_loop() can check the thread
id and return a loop created for that thread. Passing explicitly the loop
is useful when something is expected to run on a distant loop : for
instance, one can create Task that will run in a sub-thread loop (but
loop.create_task() does this) or to passe a Future() that will be awaited
on the sub-thread loop.

The thing is that the documentation is (or used to be) light on the topic,
and many people (including myself) started to write code as asyncio is
itself written : using explicit loop passing and only use get_event_loop()
as a fallback, making the the latter solution look like a sloppy
practice... and thus avoided.

2016-07-19 11:44 GMT+02:00 Gustavo J. A. M. Carneiro <
notifications@github.com>:

On 18 July 2016 at 22:46, Andrew Svetlov notifications@github.com wrote:

Honestly I recommend to newer use implicit event loop in your code but
always pass loop=loop into every asyncio API call which accepts the loop.

Moreover global implicit loop should be disabled by
asyncio.set_event_loop(None) call.
This is the only way to write 100% correct code.

That seems an exaggeration. So if you don't add loop=loop parameters
everywhere, that code still works with the default loop. It just means
that asyncio code does not support being run in a thread, it's not the end
of the world. Just document it.

Gustavo J. A. M. Carneiro
Gambit Research
"The universe is always one step beyond logic." -- Frank Herbert

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#312 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAL7GMB4ycVgXckOapVv_IneOAkww7xRks5qXJyWgaJpZM4HFw8N
.

Martin http://www.martiusweb.net Richard
www.martiusweb.net

[vxgmichel]

I wanted to mention that in OP's code, the actual exception is caught and set in the concurrent.Future returned by run_coroutine_threadsafe. This exception is actually quite explicit about what's going on. It is possible to modify the first example to get it raised:

def request(data):
    concurrent_future = asyncio.run_coroutine_threadsafe(onRequest(data), loop)
    # Raises: RuntimeError: There is no current event loop in thread 'child'.
    return concurrent_future.result()

Regarding the proper way to handle the event loop, my recommendation for users would be to never pass it explicitly and always rely on get_event_loop. This is perfectly compatible with threading, as long as asyncio.set_event_loop is called in the proper thread before the sub-loop is run. Using OP's example:

loop = asyncio.new_event_loop()
loop.set_debug(True)

def child_executor():
    asyncio.set_event_loop(loop)
    loop.run_forever()

My opinion is that passing an event loop around might make sense for libraries to be unit-testable, but it is annoying and unnecessary for user code.

[asvetlov]

@vxgmichel do you mean that non-libraries should not be unit-testable?

[vxgmichel]

@asvetlov My mistake, that's not what I meant.

Relying on get_event_loop is not a problem for unit-tests: using a TestCase.new_test_loop() as the current loop works just fine:

class SomeTests(test_utils.TestCase):

    def setUp(self):
        self.loop = self.new_test_loop()
        asyncio.set_event_loop(self.loop)

    def test_something(self):
        [...]

Even asyncio does this kind of test in some cases. To quote Guido (see this message):

I also think it's fine for a particular library or app to request that
get_event_loop() not return None. User code is simpler that way.

I can only see two issues with a library not supporting explicit loop passing:

• Everything built on top of the library will have to rely on get_event_loop as well.
• Objects similar to Lock or Queue can be created outside of the event loop so it makes sense for them to have the loop=None argument anyway. If the library provides such objects, separate tests will be required in order to handle this particular case.

For small libraries, those two points are probably not worth the trouble of passing the event loop around. But I guess it is also a matter of taste.

[1st1]

Thanks to the updated behaviour of get_event_loop() this will be non-issue in the next release of 3.5 and in 3.6. See also #452.