Add a warning when setting up async fixtures

[seifertm]

What's the problem this feature will solve?

Pytest cannot evaluate fixture that are async functions or async generators. Users of async pytest plugins may unintentionally annotate those functions with @pytest.fixture rather than the async plugin's fixture function.

The fixture result ends up to be an unawaited async generator or coroutine, which is not what the user expected. Most of the time, the corresponding tests will fail. However, when the user defines an autouse fixture to perform setup or teardown, the fixture can silently fail.

Fixture results in async generator

import pytest

@pytest.fixture
async def async_fixture():
    yield 42

def test_this(async_fixture):
    assert async_fixture == 42  # fails

Fixture results in coroutine

import pytest

@pytest.fixture
async def async_fixture():
    return 42

def test_this(async_fixture):
    assert async_fixture == 42  # fails

Fixture silently does nothing

import pytest

@pytest.fixture(autouse=True)
async def async_fixture():
    yield 42
   # perform teardown

def test_this():
    assert True  # succeeds but should fail

Describe the solution you'd like

Pytest emits the following warning when running async test functions without an async plugin (see #2224):

PytestUnhandledCoroutineWarning: async def functions are not natively supported and have been skipped.
  You need to install a suitable plugin for your async framework, for example:
    - anyio
    - pytest-asyncio
    - pytest-tornasync
    - pytest-trio
    - pytest-twisted

The warning is emitted as part of a trylast hook to pytest_pyfunc_call. That means async pytest plugins can create hook wrappers to synchronize the test function and prevent the warning from being emitted.

I suggest to add the same behavior to the use of @pytest.fixture on async functions or async generators.

Examples

• Async fixture with autouse=True not executing properly in test pytest-asyncio#518
• v0.18.0 doesn't find some of my async fixtures pytest-asyncio#286
• 0.19.0: Fixture misses attribute pytest-asyncio#390
• pytest async fixture seems not work on linux-py3.7 pytest-asyncio#386

Alternative Solutions

#9962 proposes a mechanism to await results of awaitable tests and fixtures. It can potentially address the issue described here.

Additional context

From the above list of async plugins, only anyio, pytest-asyncio, and pytest-trio make use of pytest_fixture_setup.

[seifertm]

This approach may collide with anyio which uses the standard @pytest.fixture decorator for coroutines and async generators.

https://anyio.readthedocs.io/en/stable/testing.html#asynchronous-fixtures

[jakkdl]

Okay, so there's several things going on here:

1. sync test function relies on async fixture
   • pytest-trio raises an error when attempting this
   • pytest-anyio and pytest-asyncio don't do anything
   • adding a check for this in pytest_fixture_setup seems trivial
   • but this is always a user error, so pytest can raise an error without any problem - and whether that happens before or after a plugin does so doesn't matter
2. async test function relies on async @pytest.fixture
   • pytest-anyio and pytest-trio handles this seemlessly
   • I don't see why pytest-asyncio couldn't inspect fixturedef in pytest_fixture_setup and raise an error. somewhat more complicated, but see fail if asyncio test requests async pytest fixture in strict mode pytest-asyncio#979
   • I don't think this can/should be handled by pytest, since it looks like it'd break some plugins.
3. async test function with an async @pytest.fixture(autouse=True)
   • pytest-anyio handles this seemlessly
   • pytest-trio ignores the fixture Async fixtures do not support autouse=True python-trio/pytest-trio#123
4. sync test function with an async @pytest.fixture(autouse=True)
   • pytest-trio raises an error
   • pytest-anyio ignores the fixture pytest-anyio should raise error if sync test relies on async fixture agronholm/anyio#789
   • I can see an argument for silently ignoring it, but the best thing to do is probably to raise an error and force developers to refactor their tests and not combine sync test+autouse async fixture. As with 1. I think this can be handled by pytest

[euri10]

sync test function with an async @pytest.fixture(autouse=True)
pytest-trio raises an error
pytest-anyio ignores the fixture agronholm/anyio#789

great summary @jakkdl , long story short I introduced pytest-random-order to a pretty large suite, it has only one autouse async fixture, and as soon as you get a sync test executed first, then the autouse is silently ignored, and it was a real pain to figure out why.

see https://github.com/euri10/pytest_autouse for a reproduction

a warning of some sort would be highly appreciated 💌

[jakkdl]

Okay "but this is always a user error, so pytest can raise an error without any problem - and whether that happens before or after a plugin does so doesn't matter" was an overstatement, after playing around in #12930 I couldn't come up with an easy way to have pytest error without breaking any plugins or end users code. There are plugins (e.g. hypothesis) that wrap async tests to look sync to pytest, but still handles async fixtures; and end users can write

@pytest.fixture
async def fix():
  return 1

def test_fix(fix):
  assert 1 == asyncio.run(fix)

but I also don't love punting this off to all the various plugins to handle, in large part since you can encounter this without having any plugins installed.

The only reasonable solution I came up with in #12930 is to require a @pytest.mark.allow_async_fixture on any sync test that requests an async fixture. This would require several pytest plugins to apply that mark (and define it if running against old pytest versions) to silence the warning/[future] error, but once that's done most end users would rarely encounter that mark.

But perhaps cleaner would be to robustly resolve #10404, although that can still confuse users due to caching

import pytest

@pytest.fixture(scope='session')
async def my_fixture():
    return 5

@pytest.mark.anyio
async def test_foo_async(my_fixture):
    assert my_fixture == 5

# only works if test_foo_async is run before it, and raises no warnings
def test_foo(my_fixture):
    assert my_fixture == 5