Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve awareness of frequently flaking tests #6500

Open
hendrikmakait opened this issue Jun 3, 2022 · 0 comments
Open

Improve awareness of frequently flaking tests #6500

hendrikmakait opened this issue Jun 3, 2022 · 0 comments
Labels
discussion Discussing a topic with no specific actions yet flaky test Intermittent failures on CI. stability Issue or feature related to cluster stability (e.g. deadlock)

Comments

@hendrikmakait
Copy link
Member

Problem:
We have tests that are failing rather frequently. The test report is a good resource to get an overview, but you have to actively take a look at it to identify those frequently failing tests. At the same time, #6452 has demonstrated that these flakes can point toward general issues that affect users. Had we taken a look earlier, we would have caught #6494 earlier as well.
image

Question:
How can we improve our awareness of and response time to tests that start flaking frequently?

Possible Solution:
One possible solution would be to implement a bot that files a new ticket for every test that starts flaking at least x times in the last y CI runs on main. The extreme being a ticket for any failing test on main.

There also exist (paid) tools that allow tracking flaking tests in more detail.

cc @fjetter

@hendrikmakait hendrikmakait added flaky test Intermittent failures on CI. discussion Discussing a topic with no specific actions yet stability Issue or feature related to cluster stability (e.g. deadlock) labels Jun 3, 2022
@hendrikmakait hendrikmakait changed the title Improve awareness for frequently flaking tests Improve awareness of frequently flaking tests Jun 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion Discussing a topic with no specific actions yet flaky test Intermittent failures on CI. stability Issue or feature related to cluster stability (e.g. deadlock)
Projects
None yet
Development

No branches or pull requests

1 participant