How to use django_db mark at session fixture?

[ludbek]

I have a situation where I need to load huge db fixture which is created by a function. The fixture is needed in all api tests. So I made a session fixture at conftest.py which would do it. But the problem is pytest throws following exception even though I have marked django_db:

E Failed: Database access not allowed, use the "django_db" mark to enable it.
Below is my code snippet.

from permission.helpers import update_permissions

pytestmark = [
        pytest.mark.django_db(transaction = True),]

@pytest.fixture(scope="session", autouse = True)
def permission(request):
        load_time_consuming_db_fixture()

[cybergrind]

@ludbek we've also missed such feature and created plugin for this feature:
https://github.com/tipsi/pytest-tipsi-django (usage: https://github.com/tipsi/pytest-tipsi-django/blob/master/test_django_plugin/app/tests/test_transactions.py)
In conjunction with:
https://github.com/tipsi/pytest-tipsi-testing
It gives you the ability to nest transactions and correct execution/shutdown order.

[ludbek]

@cybergrind Thanks for replying. I will definitely check it out and let you know how it went.

[paultiplady]

This has been the biggest pain point for me coming from Django's UnitTest class-based tests, to pytest-django -- in Django we use setUpTestData to run expensive DB operations once (equivalent to session-scoped pytest fixtures), and then there's a cunning trick to run obj.refresh_from_db() in the setUp to refresh the class references.

Even if I'm just creating one DB model instance, and reloading it every TC, this is almost always faster than creating in each test case.

It would be great if we could get the session-scoped fixtures from pytest-tipsi-django merged upstream, if that's a possibility; it took a bit of digging to find this issue and solution.

[cybergrind]

hey @paultiplady

I'm not sure that approach from pytest-tipsi-djanjo fits the usual testing model for pytest. The most noticeable difference is the finishing on fixtures: currently pytest doesn't explicitly finish unnecessary fixtures with a wider scope. So you need explicitly finish transactions in particular order and in general, this may cause very different effects (currently pytest may keep fixtures active even if active test and its fixtures don't require it at all).

We had to change tests in our project to this approach because we need to test some big scenarios sometimes and we've replaced existing manual transaction management in huge tests with slightly better fixtures, but it still requires attention on order tests.

Right now I can see only one solution for that: putting some kind of FAQ into documentation.

[paultiplady]

Thanks for the additional detail @cybergrind. I've dug into this a little bit more, but have run out of time for today -- here's where I've got to, I'd appreciate a sanity-check on whether this approach is useful or not, since I'm not that familiar with the pytest internals.

Also I don't understand what you mean by "pytest doesn't explicitly finish unnecessary fixtures with a wider scope", could you expand on that a bit more please? Is that referring to finalizers? That might affects what I've written below.

The pytest-django plugin uses the django_db mark, which gets handled in _django_db_marker in plugin.py (https://github.com/pytest-dev/pytest-django/blob/master/pytest_django/plugin.py#L375), calling in to the function-scoped db fixture (https://github.com/pytest-dev/pytest-django/blob/master/pytest_django/fixtures.py#L142). This fixture instantiates a Django TestCase, and calls its _pre_setup function (and enqueues the _post_teardown).

I can see a couple of options:

I'm wondering if we could extend _django_db_marker to optionally do class- or session-
scoped setup as well, which would do essentially the same thing as db, but calling the equivalent of a cls.setUpTestData or a function passed in the mark kwargs instead.

For class-scoped, and I'm assuming for session-scoped as well, the expected behaviour would be to roll back the DB afterwards, so we basically need scoped fixtures that trigger in the right order, and that each set up their own atomic transaction. I believe that means this needs to be a modification to the db fixture, rather than a separate fixture that runs beside it.

That way we'd correctly trigger any class-/session-level setup that is specified, and that setup would get called once per class/session. I believe a modification to the mark itself is required because if you just set up a session-scoped function that manually triggers django_db_blocker.unblock(), that seems to happen after the django_db mark has set up the first transaction.

This might look something like this (in plugin.py _django_db_marker()):

    if marker:
        if marker.session_db:
            getfixturevalue(request, 'db_session')
            
        if marker.class_db:
            getfixturevalue(request, 'db_class')
            
        validate_django_db(marker)
        if marker.transaction:
            getfixturevalue(request, 'transactional_db')
        else:
            getfixturevalue(request, 'db')

Is this crazy talk, or is this thread worth exploring further?

[cybergrind]

Regarding finalization: https://github.com/tipsi/pytest-tipsi-testing/blob/master/tests/test_finalization.py

This test doesn't work without explicit finalization, same as non-function level database fixtures. And this is about pytest implementation, so there is nothing to do in pytest-django to fix it.

[MRigal]

Duplicate of #388 and #33

[blueyed]

Thanks, closing.

[paultiplady]

#33 is closed, and #388 contains no meaningful discussion (unlike this one). Seems odd to close this one @blueyed , if anything I'd suggest closing #388 and make this one the canonical ticket for this problem.

[paultiplady]

👍 thanks!