Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tests] linkcheck builder unit tests fail sporadically when run using unpredictable orderings. #12122

Closed
jayaddison opened this issue Mar 17, 2024 · 0 comments · Fixed by #12126

Comments

@jayaddison
Copy link
Contributor

Describe the bug

Although the linkcheck builder unit tests pass fairly reliably in the GitHub Actions continuous integration we have configured, they are less reliable when run using a random unit test ordering.

Ideally unit tests should be able to run independently of each other; as reported in #11285, we do have some side-effects within our test suite, and it is possible that side-effects are a factor here too.

How to Reproduce

Using the pytest-randomly plugin is a straightforward way to replicate this problem.

From a checkout of bf0bec3 I'm able to see the problem occur fairly reliably with pytest==8.1.1 and pytest-randomly==3.15.0 by running:

$ pytest --randomly-seed=16 tests/test_builders/test_build_linkcheck.py
...
=========================== short test summary info ============================
FAILED tests/test_builders/test_build_linkcheck.py::test_too_many_requests_retry_after_HTTP_date[GMT-3] - assert '127.0.0.1 - .../1.1" 200 -\n' == '127.0.0.1 - .../1.1" 200 -\n'
========================= 1 failed, 38 passed in 1.59s =========================

Environment Information

Platform:              linux; (Linux-6.6.15-rt-amd64-x86_64-with-glibc2.37)
Python version:        3.11.8 (main, Feb  7 2024, 21:52:08) [GCC 13.2.0])
Python implementation: CPython
Sphinx version:        7.3.0+/bf0bec3b4
Docutils version:      0.20.1
Jinja2 version:        3.1.3
Pygments version:      2.17.2

Sphinx extensions

N/A

Additional context

No response

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant