-
Notifications
You must be signed in to change notification settings - Fork 66
🌱 Increase timeout for upgrade-e2e test to avoid flakes #1548
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🌱 Increase timeout for upgrade-e2e test to avoid flakes #1548
Conversation
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1548 +/- ##
=======================================
Coverage 74.25% 74.25%
=======================================
Files 42 42
Lines 3329 3329
=======================================
Hits 2472 2472
Misses 676 676
Partials 181 181
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
One thing I think all reviewers should ask when a PR comes in for bumping a timeout (so I'll ask): what exactly is taking more than a minute when the flakes happen? Is there a different change we could make such that the test would execute within the existing time budget? To me, there's a bit of a smell if it takes 3 minutes for an upgrade to happen. Related question: should we set some SLO's for "how long do we expect it to take for an install/upgrade to happen?" And then start measuring that, and then make sure we don't regress. |
Bumping timeouts is a race you can't win... We should be figuring out why it's taking so long and correcting that. e.g. We had flakes where the problem was that the older test resources hadn't been fully cleaned up. Bumping the timeouts on subsequent tests was not the answer. Waiting on those older resources to be cleaned up made subsequent test runs work in a reasonable amount of time. |
IMO, 1 minute is too low for this test. |
Done : #1550 Closing this one in favor of the issue |
Increasing the timeout to avoid flakes
PS.: I notice that it sometimes fails in GitHub action; therefore, I am just increasing to avoid flakes.
Example: https://github.com/operator-framework/operator-controller/actions/runs/12653708386/job/35259771016?pr=1542
(I think it occurs when the same action is running and we push more changes )