pytest-asyncio doesn't catch error with call_later functions

[sveinse]

If an error occurs in a function which is called by loop.call_soon() or loop.call_soon_threadsafe(), the exception is not printed and the pytest run is marked as successful.

Consider the code below. The given example test is passed, despite the exception, and no output of the failure is printed. If, however, the code is run directly in python without pytest, it can be seen that the exception and the traceback is indeed shown stderr. I'm not sure if I'm missing an option to pytest in the former case and that's why I don't see the missing traceback.

Please note that in either cases, the test function execution is not aborted by the exception. Evidently the event loop does not stop if errors are encountered on call_soon/call_soon_threadsafe. However, these can be intercepted by installing custom handlers using loop.set_exception_handler().

I'd like to propose that pytest-asyncio should intercept this handler to ensure it catch any errors that might happen in the lower layers of asyncio. It is unexpected that pytest gives green pass on hidden failures like this.

This way users wouldn't need to spend hours on figuring out why the test pass, but fails, and where the output goes ;)

import asyncio

def fail():
    raise Exception("FOOBAR")

async def test_loop_failure():
    await asyncio.sleep(1)
    asyncio.get_event_loop().call_soon(fail)
    await asyncio.sleep(5)

if __name__ == "__main__":
    asyncio.run(test_loop_failure())

[sveinse]

I learned that the anyio pytest integration (using asyncio backend) is able to handle these errors. It makes a separate error section in the output where it prints the results. Could be useful for inspiration if this is considered.

=================================== FAILURES ==================================== 
_______________________________ test_loop_failure _______________________________ 

    def fail():
>       raise Exception("FOOBAR")
E       Exception: FOOBAR

tests\test_foobar.py:7: Exception
============================ short test summary info ============================ 
FAILED tests/test_foobar.py::test_loop_failure - Exception: FOOBAR

PS! To reproduce, you need to install anyio and add the following code to the tests:

# Add this on top of the test file
pytestmark = pytest.mark.anyio

# Add this to conftest.py
@pytest.fixture
def anyio_backend():
    return 'asyncio'

[seifertm]

Thanks for the good writeup and explanation of the issue. I agree that pytest-asyncio should absolutely fail a test, if an exception in call_soon and similar methods occurs.

#205 describes the same issue. Since it's the older one, I'll close this ticket on favor of #205. That doesn't make your request less relevant, though.