Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Group similar test failures in test dashboard? #6778

Open
gjoseph92 opened this issue Jul 21, 2022 · 0 comments
Open

Group similar test failures in test dashboard? #6778

gjoseph92 opened this issue Jul 21, 2022 · 0 comments
Labels
tests Unit tests and/or continuous integration

Comments

@gjoseph92
Copy link
Collaborator

gjoseph92 commented Jul 21, 2022

In #6774, I manually went through the CI dashboard and identified taxonomies of test failures that were similar/the same across many tests. I have a feeling it would be valuable (and reasonable effort) to create a view that does this automatically, helping identify high-impact issues affecting CI.

Generally, there are probably 2 reasons for flaky tests:

  1. An individual test is written in a way that's unreliable (too reliant on timing, actually causes a deadlock sometimes, etc.).
  2. A bug in dask is causing something unrelated to the test to fail (timeout connecting to the cluster, asyncio error during cluster teardown, etc.). These tend to pop up in many unrelated tests. Because they can happen anywhere, they tend to blow up CI and the flaky test dashboard, and are probably responsible for the majority of failing tests.

I think we could identify #2 in a more automated way, just by creating another view on the test dashboard that groups failures by the failure message (like how OSError: Timed out trying to connect to tcp://127.0.0.1:8786 after 5 s shows up in 13 different tests). There would need to be some fuzziness to this (an exact string match wouldn't work). But that visibility might help us to identify, prioritize, and fix the problems faster. It's also possible that these systematic problems would be more likely to affect users?

cc @ian-r-rose

@gjoseph92 gjoseph92 added the tests Unit tests and/or continuous integration label Jul 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Unit tests and/or continuous integration
Projects
None yet
Development

No branches or pull requests

1 participant