flaky tests

[rosesyrett]

Sometimes CI tests fail. Flaky tests are bad and we should aim to not have any. Here is an example of such a test failing on one instance, but passing when rerun:

https://github.com/bluesky/ophyd-async/actions/runs/10741400139/job/29791703029
FAILED tests/epics/adsimdetector/test_sim.py::test_detector_writes_to_file - bluesky.utils.FailedStatus: <AsyncStatus, task: <coroutine object StandardDetector.trigger at 0x7fecaa70fe60>, errored: TimeoutError()>

[callumforrester]

I haven't seen flaky tests for several months so going to close this. If they come back, I've wondered about detecting them with infrequent use of pytest-repeat

[DiamondJoseph]

Flaky tests have re-appeared, I'm going to re-open this ticket.

[DiamondJoseph]

If you observe flaky tests, please add them as a comment on this issue.

[callumforrester]

Github actions tool for finding flaky tests, may be worth a look: https://github.com/ctrf-io/github-actions-test-reporter-ctrf

[DiamondJoseph]

pytest plugin for generating CTRF report for the above https://pypi.org/project/pytest-json-ctrf/

Do we want a table of all runs or just of runs on main/after PRs are merged?

[callumforrester]

I think all runs, why not?

[DiamondJoseph]

Identifying flaky tests vs. tests that were temporarily broken by unfinished PRs?

[DiamondJoseph]

@callumforrester I've enabled on this draft PR (scroll down to summary section): it hasn't marked any tests as flaky yet, and doesn't provide any easier method of seeing what's flaky/broken.

@coretl for comment too

https://github.com/bluesky/ophyd-async/actions/runs/10809455998?pr=567

[coretl]

Tango tests are apparently flaky because of Tango test infrastructure and there is nothing we can do to fix that. I suggest we report test failures on this ticket and make PRs that @pytest.mark.flaky(reruns=2) individual flaky tests until the CI is reliable again.

The first PR that does this needs to add pytest-rerunfailures to the dev dependencies to get the marker.