-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
storage: Initial request for resumable uploads larger than ChunkSize
not retried
#6652
Closed
jamesl33 opened this issue
Sep 12, 2022
· 2 comments
· Fixed by googleapis/google-api-go-client#1690
Closed
storage: Initial request for resumable uploads larger than ChunkSize
not retried
#6652
jamesl33 opened this issue
Sep 12, 2022
· 2 comments
· Fixed by googleapis/google-api-go-client#1690
Labels
api: storage
Issues related to the Cloud Storage API.
priority: p2
Moderately-important priority. Fix may not be included in next release.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Comments
product-auto-label
bot
added
the
api: storage
Issues related to the Cloud Storage API.
label
Sep 12, 2022
shaffeeullah
added
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
priority: p2
Moderately-important priority. Fix may not be included in next release.
and removed
triage me
I really want to be triaged.
labels
Sep 12, 2022
Thanks for flagging this, we were able to reproduce the problem and you're entirely correct. We are working on a fix in the library that handles resumable upload logic, google-api-go-client. Follow along at googleapis/google-api-go-client#1690 |
tritone
added a commit
to googleapis/google-api-go-client
that referenced
this issue
Sep 21, 2022
… w/ non-nil getBody (#1690) This allows retries when initiating a resumable upload session. Add support in media.go to return a getBody function so the request body can be recreated during retry attempts. Tests pass locally in google-cloud-go/storage Updates googleapis/google-cloud-go#6652 Co-authored-by: Chris Cotter <cjcotter@google.com>
The fix has been released in google-api-go-client; we will bump the dependency in this library as well and release tomorrow. |
tritone
added a commit
to tritone/google-cloud-go
that referenced
this issue
Sep 21, 2022
Bump apiary dependency to pick up the fix for googleapis#6652
gcf-merge-on-green bot
pushed a commit
that referenced
this issue
Sep 22, 2022
Bump apiary dependency for storage to pick up the fix for #6652
rhu713
pushed a commit
to rhu713/google-api-go-client
that referenced
this issue
Nov 4, 2022
… w/ non-nil getBody (googleapis#1690) This allows retries when initiating a resumable upload session. Add support in media.go to return a getBody function so the request body can be recreated during retry attempts. Tests pass locally in google-cloud-go/storage Updates googleapis/google-cloud-go#6652 Co-authored-by: Chris Cotter <cjcotter@google.com>
rhu713
pushed a commit
to rhu713/google-api-go-client
that referenced
this issue
Nov 4, 2022
… w/ non-nil getBody (googleapis#1690) This allows retries when initiating a resumable upload session. Add support in media.go to return a getBody function so the request body can be recreated during retry attempts. Tests pass locally in google-cloud-go/storage Updates googleapis/google-cloud-go#6652 Co-authored-by: Chris Cotter <cjcotter@google.com>
rhu713
pushed a commit
to rhu713/google-api-go-client
that referenced
this issue
Nov 4, 2022
… w/ non-nil getBody (googleapis#1690) This allows retries when initiating a resumable upload session. Add support in media.go to return a getBody function so the request body can be recreated during retry attempts. Tests pass locally in google-cloud-go/storage Updates googleapis/google-cloud-go#6652 Co-authored-by: Chris Cotter <cjcotter@google.com>
rhu713
pushed a commit
to cockroachdb/google-api-go-client
that referenced
this issue
Nov 4, 2022
… w/ non-nil getBody (googleapis#1690) This allows retries when initiating a resumable upload session. Add support in media.go to return a getBody function so the request body can be recreated during retry attempts. Tests pass locally in google-cloud-go/storage Updates googleapis/google-cloud-go#6652 Co-authored-by: Chris Cotter <cjcotter@google.com>
rhu713
pushed a commit
to cockroachdb/google-api-go-client
that referenced
this issue
Nov 7, 2022
… w/ non-nil getBody (googleapis#1690) This allows retries when initiating a resumable upload session. Add support in media.go to return a getBody function so the request body can be recreated during retry attempts. Tests pass locally in google-cloud-go/storage Updates googleapis/google-cloud-go#6652 Co-authored-by: Chris Cotter <cjcotter@google.com>
rhu713
pushed a commit
to cockroachdb/google-api-go-client
that referenced
this issue
Nov 7, 2022
… w/ non-nil getBody (googleapis#1690) This allows retries when initiating a resumable upload session. Add support in media.go to return a getBody function so the request body can be recreated during retry attempts. Tests pass locally in google-cloud-go/storage Updates googleapis/google-cloud-go#6652 Co-authored-by: Chris Cotter <cjcotter@google.com>
rhu713
pushed a commit
to cockroachdb/google-api-go-client
that referenced
this issue
Nov 17, 2022
… w/ non-nil getBody (googleapis#1690) This allows retries when initiating a resumable upload session. Add support in media.go to return a getBody function so the request body can be recreated during retry attempts. Tests pass locally in google-cloud-go/storage Updates googleapis/google-cloud-go#6652 Co-authored-by: Chris Cotter <cjcotter@google.com>
rhu713
pushed a commit
to cockroachdb/google-api-go-client
that referenced
this issue
Jan 18, 2023
… w/ non-nil getBody (googleapis#1690) This allows retries when initiating a resumable upload session. Add support in media.go to return a getBody function so the request body can be recreated during retry attempts. Tests pass locally in google-cloud-go/storage Updates googleapis/google-cloud-go#6652 Co-authored-by: Chris Cotter <cjcotter@google.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
api: storage
Issues related to the Cloud Storage API.
priority: p2
Moderately-important priority. Fix may not be included in next release.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Description
I'm currently investigating a case where we're seeing a large number of failures due to 503 status codes
(
ServiceUnavailable
). My understanding was that these errors are retried by default, however, I've observed a casewhere that doesn't appear to be true.
Walking through the code, it appears this is because the first request for an upload has
req.GetBody
set tonil
:This results in
gensupport.SendRequestWithRetry
not retrying the request because it can't retrieve the request bodyfor the retry.
I suspect this may be why we've seen so many 503 status codes surface as errors where I'd expect them to be
retried.
Client
Storage
Environment
Go Environment
Code
Steps to Reproduce
go mod init github.com/<username>/gcs-retries && go mod tidy
go run server.go
go run client.go
(we should see multiple requests logged in the server)go run client.go
(we should an immediate failure)Expected behavior
I would expect to see this 503 error retried, however, there may be a good reason for not retrying the first request
(e.g. forced non-idempotency for requests that setup resumable uploads).
Actual behavior
I observe that the first request for an upload larger than
ChunkSize
is not retried.The text was updated successfully, but these errors were encountered: