Retry get pull request calls to github #1131
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
These calls have been 404'ing lately, likely due to eventual consistency issues on GitHub's side where they send the PR created webhook but it's not yet available on their API. To work around this, we will retry up to 3 times with a 1s delay.
Codecov Report
@@ Coverage Diff @@
## master #1131 +/- ##
=======================================
Coverage 71.17% 71.17%
=======================================
Files 71 71
Lines 5876 5887 +11
=======================================
+ Hits 4182 4190 +8
- Misses 1346 1348 +2
- Partials 348 349 +1
Continue to review full report at Codecov.
|
| // They've got some eventual consistency issues going on so we're just going | ||
| // to retry up to 3 times with a 1s sleep. | ||
| numRetries := 3 | ||
| retryDelay := 1 * time.Second |
There was a problem hiding this comment.
Thank you for implementing this retries! 🤟
could you also please implement exponential backoff to space out the retries?
- on first retry you sleep for 1 sec
- on second retry you sleep for 2 sec
- on third retry you sleep for 4 sec.
jamengual
pushed a commit
that referenced
this pull request
Oct 4, 2021
* Improve github pull request call retries Retry with fixed 1 second backoff up to 3 retries was added by #1131 to address #1019, but the issue continued to show up (#1453). Increase max attempts to 5 and use exponential backoff for a maximum total retry time of (2^n - n - 1) seconds, which is roughly 30 seconds for current max attempts n = 5. Also move the sleep to the top of the loop so that we never sleep without sending the request again on the last iteration. * Fix style with gofmt -s
iainlane
added a commit
to iainlane/atlantis
that referenced
this pull request
Jan 17, 2022
Similar to runatlantis#1131, we see this for /files/ too, resulting in a plan error. Closes runatlantis#1905
iainlane
added a commit
to iainlane/atlantis
that referenced
this pull request
Jan 17, 2022
Similar to runatlantis#1131, we see this for /files/ too, resulting in a plan error. Closes runatlantis#1905
elementalvoid
added a commit
to elementalvoid/atlantis
that referenced
this pull request
Jan 21, 2022
This is a follow on to resolve similar issues to runatlantis#1019. In runatlantis#1131 retries were added to GetPullRequest. And in runatlantis#1810 a backoff was included. However, those only resolve one potential request at the very beginning of a PR creation. The other request that happens early on during auto-plan is one to ListFiles to detect the modified files. This too can sometimes result in a 404 due to async updates on the GitHub side.
chenrui333
pushed a commit
that referenced
this pull request
Jan 25, 2022
krrrr38
pushed a commit
to krrrr38/atlantis
that referenced
this pull request
Dec 16, 2022
* Improve github pull request call retries Retry with fixed 1 second backoff up to 3 retries was added by runatlantis#1131 to address runatlantis#1019, but the issue continued to show up (runatlantis#1453). Increase max attempts to 5 and use exponential backoff for a maximum total retry time of (2^n - n - 1) seconds, which is roughly 30 seconds for current max attempts n = 5. Also move the sleep to the top of the loop so that we never sleep without sending the request again on the last iteration. * Fix style with gofmt -s
krrrr38
pushed a commit
to krrrr38/atlantis
that referenced
this pull request
Dec 16, 2022
Similar to runatlantis#1131, we see this for /files/ too, resulting in a plan error. Closes runatlantis#1905
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These calls have been 404'ing lately, likely due to eventual consistency
issues on GitHub's side where they send the PR created webhook but it's
not yet available on their API.
To work around this, we will retry up to 3 times with a 1s delay.
Fixes #1019