Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a warning when setting up async fixtures #10839

Closed
seifertm opened this issue Mar 26, 2023 · 4 comments · Fixed by #12930 or pytest-dev/pytest-asyncio#979
Closed

Add a warning when setting up async fixtures #10839

seifertm opened this issue Mar 26, 2023 · 4 comments · Fixed by #12930 or pytest-dev/pytest-asyncio#979
Labels
topic: fixtures anything involving fixtures directly or indirectly type: enhancement new feature or API change, should be merged into features branch

Comments

@seifertm
Copy link

What's the problem this feature will solve?

Pytest cannot evaluate fixture that are async functions or async generators. Users of async pytest plugins may unintentionally annotate those functions with @pytest.fixture rather than the async plugin's fixture function.

The fixture result ends up to be an unawaited async generator or coroutine, which is not what the user expected. Most of the time, the corresponding tests will fail. However, when the user defines an autouse fixture to perform setup or teardown, the fixture can silently fail.

Fixture results in async generator
import pytest

@pytest.fixture
async def async_fixture():
    yield 42

def test_this(async_fixture):
    assert async_fixture == 42  # fails
Fixture results in coroutine
import pytest

@pytest.fixture
async def async_fixture():
    return 42

def test_this(async_fixture):
    assert async_fixture == 42  # fails
Fixture silently does nothing
import pytest

@pytest.fixture(autouse=True)
async def async_fixture():
    yield 42
   # perform teardown

def test_this():
    assert True  # succeeds but should fail

Describe the solution you'd like

Pytest emits the following warning when running async test functions without an async plugin (see #2224):

PytestUnhandledCoroutineWarning: async def functions are not natively supported and have been skipped.
  You need to install a suitable plugin for your async framework, for example:
    - anyio
    - pytest-asyncio
    - pytest-tornasync
    - pytest-trio
    - pytest-twisted

The warning is emitted as part of a trylast hook to pytest_pyfunc_call. That means async pytest plugins can create hook wrappers to synchronize the test function and prevent the warning from being emitted.

I suggest to add the same behavior to the use of @pytest.fixture on async functions or async generators.

Examples

Alternative Solutions

#9962 proposes a mechanism to await results of awaitable tests and fixtures. It can potentially address the issue described here.

Additional context

From the above list of async plugins, only anyio, pytest-asyncio, and pytest-trio make use of pytest_fixture_setup.

@nicoddemus nicoddemus added type: enhancement new feature or API change, should be merged into features branch topic: fixtures anything involving fixtures directly or indirectly labels Mar 26, 2023
@seifertm
Copy link
Author

This approach may collide with anyio which uses the standard @pytest.fixture decorator for coroutines and async generators.

https://anyio.readthedocs.io/en/stable/testing.html#asynchronous-fixtures

@jakkdl
Copy link
Member

jakkdl commented Sep 17, 2024

Okay, so there's several things going on here:

  1. sync test function relies on async fixture
    • pytest-trio raises an error when attempting this
    • pytest-anyio and pytest-asyncio don't do anything
    • adding a check for this in pytest_fixture_setup seems trivial
    • but this is always a user error, so pytest can raise an error without any problem - and whether that happens before or after a plugin does so doesn't matter
  2. async test function relies on async @pytest.fixture
  3. async test function with an async @pytest.fixture(autouse=True)
  4. sync test function with an async @pytest.fixture(autouse=True)

@euri10
Copy link

euri10 commented Oct 7, 2024

sync test function with an async @pytest.fixture(autouse=True)
pytest-trio raises an error
pytest-anyio ignores the fixture agronholm/anyio#789

great summary @jakkdl , long story short I introduced pytest-random-order to a pretty large suite, it has only one autouse async fixture, and as soon as you get a sync test executed first, then the autouse is silently ignored, and it was a real pain to figure out why.

see https://github.com/euri10/pytest_autouse for a reproduction

a warning of some sort would be highly appreciated 💌

@jakkdl
Copy link
Member

jakkdl commented Nov 5, 2024

EDIT: I ultimately came up with a clean solution in #12930, this comment can be disregarded

Okay "but this is always a user error, so pytest can raise an error without any problem - and whether that happens before or after a plugin does so doesn't matter" was an overstatement, after playing around in #12930 I couldn't come up with an easy way to have pytest error without breaking any plugins or end users code. There are plugins (e.g. hypothesis) that wrap async tests to look sync to pytest, but still handles async fixtures; and end users can write

@pytest.fixture
async def fix():
  return 1

def test_fix(fix):
  assert 1 == asyncio.run(fix)

but I also don't love punting this off to all the various plugins to handle, in large part since you can encounter this without having any plugins installed.

The only reasonable solution I came up with in #12930 is to require a @pytest.mark.allow_async_fixture on any sync test that requests an async fixture. This would require several pytest plugins to apply that mark (and define it if running against old pytest versions) to silence the warning/[future] error, but once that's done most end users would rarely encounter that mark.

But perhaps cleaner would be to robustly resolve #10404, although that can still confuse users due to caching

import pytest

@pytest.fixture(scope='session')
async def my_fixture():
    return 5

@pytest.mark.anyio
async def test_foo_async(my_fixture):
    assert my_fixture == 5

# only works if test_foo_async is run before it, and raises no warnings
def test_foo(my_fixture):
    assert my_fixture == 5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: fixtures anything involving fixtures directly or indirectly type: enhancement new feature or API change, should be merged into features branch
Projects
None yet
4 participants