Investigate AWS SDK for Go v2 default retry configuration

[ewbankkit]

Background

• https://aws.amazon.com/blogs/mt/managing-monitoring-api-throttling-in-workloads/
• https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html

AWS services maintain API rate limits which are used to protect the performance and availability of the service. If a service’s API rate limit is exceeded then the caller will receive a throttling error (usually RequestLimitExceeded or ThrottlingException) and subsequent API calls are throttled. AWS SDKs implement techniques to limit the impact of throttling on applications, including:

• Retry - SDKs implement configurable, automatic retry logic.
• Backoff - SDKs implement an exponential backoff algorithm, using increasingly longer waits between retries when consecutive throttling errors are encountered.
• Jitter - A random delay is added to each retry waiting period to help prevent large numbers of clients retrying at the same time (a variant of the thundering herd problem).

AWS SDK for Go v1

The Terraform AWS Provider’s max_retries configuration attribute populates the SDK’s Config.MaxRetries field. The default value is 25.
The DefaultRetryer is used, with only NumMaxRetries modified from its default value.
Retry logic is similar for throttling and non-throttling errors (only min and max delays differ); most notably, the max number of retries is the same. Note that the retries are per-request, not shared.

AWS SDK for Go v2

In addition to max_retries the Terraform AWS Provider’s retry_mode configuration attribute determines whether standard or adaptive retry strategies are used (the default is the standard strategy).
The AWS SDK for Go v2 SDK retryer uses a token bucket for retrying throttling errors. This token bucket is shared across ALL requests for an API client.

The Problem

After migration of services from AWS SDK for Go v1 to v2 we have received multiple reports (see linked GitHub Issues) of frequent operation failures: failed to get rate limit token, retry quota exceeded, 0 available, 5 requested, which is a QuotaExceededException.
Configuring retry_mode = "adaptive" is only a partial solution as retry tokens (as opposed to request tokens) don’t cause a sleep before retry.
Increasing token_bucket_rate_limiter_capacity to a large value (it's default is 500) seems to resolve the issue but as we migrate more (and eventually all) services to AWS SDK for Go v2 we need to have a useful default configuration for this functionality.

Relations

Relates #34669.
Relates #32976.
Relates #35926.
Relates hashicorp/aws-sdk-go-base#933.
Relates hashicorp/aws-sdk-go-base#932.
Relates hashicorp/aws-sdk-go-base#918.
Relates hashicorp/aws-sdk-go-base#915.
Relates #36094.
Relates #25552.

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ewbankkit]

After discussion with the AWS Go SDK team, we have decided to follow the newly published guidance and by default configure a RateLimiter of ratelimit.None unless token_bucket_rate_limiter_capacity has been configured.
With ratelimit.None in effect the max_retries value is still respected, enabling AWS SDK for Go v1 equivalent functionality.

[github-actions]

This functionality has been released in v5.42.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.