Test collection is wrong with parametrized tests

[maratumba]

Issue

Pytest collection is not deterministic when there are parametrized fixtures involved, which will make test collection unpredictable and result in some tests being skipped. I wrote a shell script that does exactly this before I found out about the plugin. I solved it by sorting the tests after collecting them, hopefully it will help to you too.

Reproduction

Create a test with parameters in django-cookiecutter template. Note that I haven't tried with this exact setup but I'm 90% sure it will have the same issue.

# conftest.py
import pytest
from django.utils.dateparse import parse_datetime
from faker import Faker

from cpde.users.models import User
from code.users.tests.factories import UserFactory

faker_uk = Faker(locale="en_GB")
Faker.seed()

naive_datetime_fuzz = [
    pytest.param(
        parse_datetime("2021-11-07T01:30:00"), marks=pytest.mark.fuzz
    ),  # Naive datetime during US DST end
    pytest.param(
        parse_datetime("2020-02-29T09:30:00"), marks=pytest.mark.fuzz
    ),  # Leap Year Feb29th
    pytest.param(
        parse_datetime("2021-11-07T00:00:00.000"), marks=pytest.mark.fuzz
    ),  # Exactly midnight
    pytest.param(faker_uk.past_datetime()),  # Recent time
]

@pytest.fixture(params=naive_datetime_fuzz)
def naive_datetime(request):
    return request.param

@pytest.fixture
def user(naive_datetime) -> User:
    return UserFactory(date_joined=naive_datetime)

Run the same portion a few times and note the different number of collected tests:

$ pytest --portion 1/3
Test session starts (platform: linux, Python 3.9.6, pytest 6.2.5, pytest-sugar 0.9.4)
django: settings: config.settings.test (from option)
Using --randomly-seed=1638466916
rootdir: /home/user/devel/code, configfile: pytest.ini
plugins: Faker-4.18.0, django-constance-2.8.0, django-test-migrations-1.0.0, hypothesis-5.19.0, postmarker-0.15.0, cov-2.10.1, deadfixtures-2.2.1, django-4.4.0, forked-1.3.0, mock-3.5.1, portion-0.1.0, randomly-3.4.0, sugar-0.9.4, testmon-1.0.3, timeout-1.4.0, xdist-2.4.0, requests-mock-1.8.0
collecting ... 
 code/users/tests/test_models_unit.py ✓✓                                                                                                                                                                                                           17% █▋        
 code/users/tests/test_models.py ✓                                                                                                                                                                                                                 25% ██▌       
 code/users/tests/test_schema_views.py ✓✓                                                                                                                                                                                                          42% ████▎     
 code/users/tests/test_tasks.py ✓                                                                                                                                                                                                                  50% █████     
 code/users/tests/test_admin.py ✓✓✓✓                                                                                                                                                                                                               83% ████████▍ 
 code/users/tests/test_views.py ✓✓                                                                                                                                                                                                                100% ██████████
======================================================================================================================= warnings summary ========================================================================================================================
code/users/tests/test_models.py::test_user_get_absolute_url[naive_datetime3]
code/users/tests/test_views.py::TestUserDetailView::test_authenticated[naive_datetime3]
code/users/tests/test_views.py::TestUserDetailView::test_not_authenticated[naive_datetime3]
  /home/user/.pyenv/versions/3.9.6/envs/code/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1367: RuntimeWarning: DateTimeField User.date_joined received a naive datetime (2021-11-05 08:39:16) while time zone support is active.
    warnings.warn("DateTimeField %s received a naive datetime (%s)"

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (5.45s):
      12 passed
      49 deselected
# Run again:
$ pytest --portion 1/3
Test session starts (platform: linux, Python 3.9.6, pytest 6.2.5, pytest-sugar 0.9.4)
django: settings: config.settings.test (from option)
Using --randomly-seed=1638466930
rootdir: /home/user/devel/code, configfile: pytest.ini
plugins: Faker-4.18.0, django-constance-2.8.0, django-test-migrations-1.0.0, hypothesis-5.19.0, postmarker-0.15.0, cov-2.10.1, deadfixtures-2.2.1, django-4.4.0, forked-1.3.0, mock-3.5.1, portion-0.1.0, randomly-3.4.0, sugar-0.9.4, testmon-1.0.3, timeout-1.4.0, xdist-2.4.0, requests-mock-1.8.0
collecting ... 
 code/users/tests/test_schema_views.py ✓✓                                                                                                                                                                                                          18% █▊        
 code/users/tests/test_drf_views.py ✓✓                                                                                                                                                                                                             36% ███▋      
 code/users/tests/test_admin.py ✓✓✓✓                                                                                                                                                                                                               73% ███████▍  
 code/users/tests/test_drf_urls.py ✓✓✓                                                                                                                                                                                                            100% ██████████
======================================================================================================================= warnings summary ========================================================================================================================
code/users/tests/test_drf_views.py::TestUserViewSet::test_get_queryset[naive_datetime3]
code/users/tests/test_drf_views.py::TestUserViewSet::test_me[naive_datetime3]
code/users/tests/test_drf_urls.py::test_user_detail[naive_datetime3]
  /home/user/.pyenv/versions/3.9.6/envs/code/lib/python3.9/site-packages/django/db/models/fields/__init__.py:1367: RuntimeWarning: DateTimeField User.date_joined received a naive datetime (2021-11-21 23:51:20) while time zone support is active.
    warnings.warn("DateTimeField %s received a naive datetime (%s)"

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (2.08s):
      11 passed
      50 deselected