Refactor SDK's request retry, and error handling behavior #487

jasdel · 2020-02-06T19:30:49Z

This update refactors the SDK's request retry behavior by updating the SDK's implementation to a model that will be the standard across the SDKs. This model simplifies retry behavior, and will retry all connection errors. Not just write connection reset errors.

This update also refactors the SDK's error wrapping and handling. The SDK's error types such as awserr.Error implement the Unwrap method allowing the nested errors to be more easily extracted using the Go 1.13 errors.As, Is, and Unwrap functions.

This update also provides more information about why a request failed, and was retried, but exhausted attempts. The aws.MaxAttemptsError will wrap the connection or service API error.

TODO:

Update SDK's retry and error tests
Add changelog
Add missing/commented out tests to retry package

This update refactors the SDK's request retry behavior by updating the SDK's implementation to a model that will be the standard across the SDKs. This model simplifies retry behavior, and will retry all connection errors. Not just write connection reset errors. This update also refactors the SDK's error wrapping and handling. The SDK's error types such as awserr.Error implement the Unwrap method allowing the nested errors to be more easily extracted using the Go 1.13 errors.As, Is, and Unwrap functions. This update also provides more information about why a request failed, and was retried, but exhausted attempts. The aws.MaxAttemptsError will wrap the connection or service API error.

jasdel · 2020-03-07T23:29:15Z

Two commits, first includes the hand written code changes, second commit is all code generation.

jasdel · 2020-03-08T18:37:02Z

Expect Go 1.11 and Go 1.12 builds to fail due to errors.As and other functions added in Go 1.13

aws/defaults/handlers.go

aws/ec2metadata/token_provider.go

aws/ratelimit/token_bucket.go

aws/ratelimit/token_rate_limit.go

aws/retry/retry.go

aws/retry/retryable_error.go

aws/retry/retryable_error_test.go

aws/retry/standard.go

aws/signer/v4/functional_test.go

aws/request.go

Fixes a broken unit test missed when implementing aws#487.

Fixes a broken unit test missed when implementing #487.

Breaking Change --- * Update SDK retry behavior * Significant updates were made the SDK's retry behavior. The SDK will now retry all connections error. In addition, to changing what errors are retried the SDK's retry behavior not distinguish the difference between throttling errors, and regular retryable errors. All errors will be retried with the same backoff jitter delay scaling. * The SDK will attempt an operation request 3 times by default. This is one less than the previous initial request with 3 retires. * New helper functions in the new `aws/retry` package allow wrapping a `Retrier` with custom behavior, overriding the base `Retrier`, (e.g. `AddWithErrorCodes`, and `AddWithMaxAttempts`) * Update SDK error handling * Updates the SDK's handling of errors to take advantage of Go 1.13's new `errors.As`, `Is`, and `Unwrap`. The SDK's errors were updated to satisfy the `Unwrap` interface, returning the underlying error. * With this update, you can now more easily access the SDK's layered errors, and meaningful state such as, `Timeout`, `Temporary`, and other states added to the SDK such as `CanceledError`. * Bump SDK minimum supported version from Go 1.12 to Go 1.13 * The SDK's minimum supported version is bumped to take advantage of Go 1.13's updated `errors` package. Services --- * Synced the V2 SDK with latest AWS service API definitions. SDK Features --- * `aws`: Add Support for additional credential providers and credential configuration chaining ([aws#488](aws#488)) * `aws/processcreds`: Adds Support for the Process Credential Provider * Fixes [aws#249](aws#249) * `aws/stscreds`: Adds Support for the Web Identity Credential Provider * Fixes [aws#475](aws#475) * Fixes [aws#338](aws#338) * Adds Support for `credential_source` * Fixes [aws#274](aws#274) * `aws/awserr`: Adds support for Go 1.13's `errors.Unwrap` ([aws#487](aws#487)) * `aws`: Updates SDK retry behavior ([aws#487](aws#487)) * `aws/retry`: New package defining logic to determine if a request should be retried, and backoff. * `aws/ratelimit`: New package defining rate limit logic such as token bucket used by the `retry.Standard` retrier. SDK Enhancements --- * `aws`: Add grouping of concurrent refresh of credentials ([aws#503](aws#503) * Concurrent calls to `Retrieve` are now grouped in order to prevent numerous synchronous calls to refresh the credentials. Replacing the mutex with a singleflight reduces the overall amount of time request signatures need to wait while retrieving credentials. This is improvement becomes pronounced when many requests are made concurrently. * `service/s3/s3manager`: Improve memory allocation behavior by replacing sync.Pool with custom pool implementation * Improves memory allocations that occur when the provided `io.Reader` to upload does not satisfy both the `io.ReaderAt` and `io.ReadSeeker` interfaces. SDK Bugs --- * `service/s3/s3manager`: Fix resource leaks when the following occurred: * Failed CreateMultipartUpload call * Failed UploadPart call

Breaking Change --- * Update SDK retry behavior * Significant updates were made the SDK's retry behavior. The SDK will now retry all connections error. In addition, to changing what errors are retried the SDK's retry behavior not distinguish the difference between throttling errors, and regular retryable errors. All errors will be retried with the same backoff jitter delay scaling. * The SDK will attempt an operation request 3 times by default. This is one less than the previous initial request with 3 retires. * New helper functions in the new `aws/retry` package allow wrapping a `Retrier` with custom behavior, overriding the base `Retrier`, (e.g. `AddWithErrorCodes`, and `AddWithMaxAttempts`) * Update SDK error handling * Updates the SDK's handling of errors to take advantage of Go 1.13's new `errors.As`, `Is`, and `Unwrap`. The SDK's errors were updated to satisfy the `Unwrap` interface, returning the underlying error. * With this update, you can now more easily access the SDK's layered errors, and meaningful state such as, `Timeout`, `Temporary`, and other states added to the SDK such as `CanceledError`. * Bump SDK minimum supported version from Go 1.12 to Go 1.13 * The SDK's minimum supported version is bumped to take advantage of Go 1.13's updated `errors` package. Services --- * Synced the V2 SDK with latest AWS service API definitions. SDK Features --- * `aws`: Add Support for additional credential providers and credential configuration chaining ([aws#488](aws#488)) * `aws/processcreds`: Adds Support for the Process Credential Provider * Fixes [aws#249](aws#249) * `aws/stscreds`: Adds Support for the Web Identity Credential Provider * Fixes [aws#475](aws#475) * Fixes [aws#338](aws#338) * Adds Support for `credential_source` * Fixes [aws#274](aws#274) * `aws/awserr`: Adds support for Go 1.13's `errors.Unwrap` ([aws#487](aws#487)) * `aws`: Updates SDK retry behavior ([aws#487](aws#487)) * `aws/retry`: New package defining logic to determine if a request should be retried, and backoff. * `aws/ratelimit`: New package defining rate limit logic such as token bucket used by the `retry.Standard` retrier. SDK Enhancements --- * `aws`: Add grouping of concurrent refresh of credentials ([aws#503](aws#503)) * Concurrent calls to `Retrieve` are now grouped in order to prevent numerous synchronous calls to refresh the credentials. Replacing the mutex with a singleflight reduces the overall amount of time request signatures need to wait while retrieving credentials. This is improvement becomes pronounced when many requests are made concurrently. * `service/s3/s3manager`: Improve memory allocation behavior by replacing sync.Pool with custom pool implementation * Improves memory allocations that occur when the provided `io.Reader` to upload does not satisfy both the `io.ReaderAt` and `io.ReadSeeker` interfaces. SDK Bugs --- * `service/s3/s3manager`: Fix resource leaks when the following occurred: * Failed CreateMultipartUpload call * Failed UploadPart call

Breaking Change --- * Update SDK retry behavior * Significant updates were made the SDK's retry behavior. The SDK will now retry all connections error. In addition, to changing what errors are retried the SDK's retry behavior not distinguish the difference between throttling errors, and regular retryable errors. All errors will be retried with the same backoff jitter delay scaling. * The SDK will attempt an operation request 3 times by default. This is one less than the previous initial request with 3 retires. * New helper functions in the new `aws/retry` package allow wrapping a `Retrier` with custom behavior, overriding the base `Retrier`, (e.g. `AddWithErrorCodes`, and `AddWithMaxAttempts`) * Update SDK error handling * Updates the SDK's handling of errors to take advantage of Go 1.13's new `errors.As`, `Is`, and `Unwrap`. The SDK's errors were updated to satisfy the `Unwrap` interface, returning the underlying error. * With this update, you can now more easily access the SDK's layered errors, and meaningful state such as, `Timeout`, `Temporary`, and other states added to the SDK such as `CanceledError`. * Bump SDK minimum supported version from Go 1.12 to Go 1.13 * The SDK's minimum supported version is bumped to take advantage of Go 1.13's updated `errors` package. Services --- * Synced the V2 SDK with latest AWS service API definitions. SDK Features --- * `aws`: Add Support for additional credential providers and credential configuration chaining ([#488](#488)) * `aws/processcreds`: Adds Support for the Process Credential Provider * Fixes [#249](#249) * `aws/stscreds`: Adds Support for the Web Identity Credential Provider * Fixes [#475](#475) * Fixes [#338](#338) * Adds Support for `credential_source` * Fixes [#274](#274) * `aws/awserr`: Adds support for Go 1.13's `errors.Unwrap` ([#487](#487)) * `aws`: Updates SDK retry behavior ([#487](#487)) * `aws/retry`: New package defining logic to determine if a request should be retried, and backoff. * `aws/ratelimit`: New package defining rate limit logic such as token bucket used by the `retry.Standard` retrier. SDK Enhancements --- * `aws`: Add grouping of concurrent refresh of credentials ([#503](#503)) * Concurrent calls to `Retrieve` are now grouped in order to prevent numerous synchronous calls to refresh the credentials. Replacing the mutex with a singleflight reduces the overall amount of time request signatures need to wait while retrieving credentials. This is improvement becomes pronounced when many requests are made concurrently. * `service/s3/s3manager`: Improve memory allocation behavior by replacing sync.Pool with custom pool implementation * Improves memory allocations that occur when the provided `io.Reader` to upload does not satisfy both the `io.ReaderAt` and `io.ReadSeeker` interfaces. SDK Bugs --- * `service/s3/s3manager`: Fix resource leaks when the following occurred: * Failed CreateMultipartUpload call * Failed UploadPart call

…#487

jasdel added the pr/work-in-progress This PR is a draft and needs further work. label Feb 6, 2020

jasdel force-pushed the proto/StandardRetries branch from 667653e to f7c0f3c Compare February 27, 2020 00:21

jasdel changed the title ~~Introduces new retry model and transport client to the SDK~~ Refactor SDK's request retry behavior adding rate limiting Feb 27, 2020

jasdel force-pushed the proto/StandardRetries branch 2 times, most recently from 778842e to 3fb5834 Compare March 4, 2020 20:41

jasdel force-pushed the proto/StandardRetries branch from 710f891 to 97aa0b8 Compare March 7, 2020 23:26

jasdel changed the title ~~Refactor SDK's request retry behavior adding rate limiting~~ Refactor SDK's request retry, and error handling behavior Mar 7, 2020

jasdel marked this pull request as ready for review March 7, 2020 23:28

jasdel removed the pr/work-in-progress This PR is a draft and needs further work. label Mar 7, 2020

jasdel added 2 commits March 7, 2020 15:29

regenerate API clients

49924a0

increment minimum supported version

b4dba04

add request invocation and metric headers

8d072ae

jasdel force-pushed the proto/StandardRetries branch from 97aa0b8 to 8d072ae Compare March 8, 2020 20:25

skotambkar self-requested a review March 9, 2020 16:25

skotambkar reviewed Mar 9, 2020

View reviewed changes

aws/defaults/handlers.go Show resolved Hide resolved

skotambkar reviewed Mar 9, 2020

View reviewed changes

aws/defaults/handlers.go Show resolved Hide resolved

skotambkar reviewed Mar 9, 2020

View reviewed changes

aws/defaults/handlers.go Show resolved Hide resolved