Retry requests when status code is 429 #410
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This code is what HTTP uses to indicate the client has sent too many requests in a given amount of time. It may sound counter-intuitive to retry in this situation, but that's why we use an exponential backoff mechanism. It gives the server the opportunity to recover, without failing the requests immediately. The retry mechanism already works in object stores like S3 because they return a server error. But without this change, we are not handling GCS properly. GCS returns a client error `429 Too Many Requests` instead. This change enables retries on this response too. A more advanced retry mechanism would use the optional response header `Retry-After`, but that is beyond the scope of this PR. Closes: apache#309
alamb
approved these changes
Jul 1, 2025
Contributor
Author
|
Is the unrelated Clippy error a blocker? |
Member
|
you should be able to merge in main #418 |
Contributor
Author
done |
mbrobbel
approved these changes
Jul 2, 2025
4 tasks
This was referenced Jul 7, 2025
crepererum
pushed a commit
to crepererum/arrow-rs-object-store
that referenced
this pull request
Jul 10, 2025
This code is what HTTP uses to indicate the client has sent too many requests in a given amount of time. It may sound counter-intuitive to retry in this situation, but that's why we use an exponential backoff mechanism. It gives the server the opportunity to recover, without failing the requests immediately. The retry mechanism already works in object stores like S3 because they return a server error. But without this change, we are not handling GCS properly. GCS returns a client error `429 Too Many Requests` instead. This change enables retries on this response too. A more advanced retry mechanism would use the optional response header `Retry-After`, but that is beyond the scope of this PR. Closes: apache#309
Merged
dcherian
added a commit
to earth-mover/icechunk
that referenced
this pull request
Nov 12, 2025
Looks like the upstream PR was closed and released in object-store-rs 0.12.4 apache/arrow-rs-object-store#410
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #309
Closes #425
Rationale for this change
Status code 429 is what HTTP uses to indicate the client has sent too many requests in a given amount of time. It may sound counter-intuitive to retry in this situation, but that's why we use an exponential backoff mechanism. It gives the server the opportunity to recover, without failing the requests immediately.
The retry mechanism already works in object stores like S3 because they return a server error. But without this change, we are not handling GCS properly. GCS returns a client error
429 Too Many Requestsinstead. This change enables retries on this response too.A more advanced retry mechanism would use the optional response header
Retry-After, but that is beyond the scope of this PR.