-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle flasky tests #1297
Comments
How about fixing the flacky test? When we encoutered an unrelated failing test we just open a ticket about it investigate it and fix it.
I'm against it. That would be hiding issues that would only grow worse. Let's rather fix them. |
Question is: is it always possible to fix a flasky test.
+1 Could be useful:
The plugin would also categorize a test as flasky if it fails and succeeds on rerun so it can be used to detect flasky tests too. |
Yes |
A broken build is IMO a better incetive to fix stuff |
Description
Here's a typical scenario: I open a pull request, the build kicks off, and about an hour and a half later, the pipeline fails. When I check the failed tests, it’s something completely unrelated to my changes—a classic flaky test problem.
Here are some suggestions:
Run flaky tests in a separate pipeline
If flaky tests have their own pipeline, we can simply rerun that part when it fails instead of rerunning everything.
Disable flaky tests (not ideal)
Try the Jenkins Flaky Test Handler plugin
Here’s the plugin. It’s designed to identify and rerun flaky tests automatically, which could help in our situation.
Why This Matters:
The text was updated successfully, but these errors were encountered: