Kinesis LimitExceededException errors are not retried

[cbroglie]

What issue did you see?

Calls to kinesis.DescribeStream can get throttled, returning LimitExceededException. However, since #1276, these calls are not retried.

The root of the problem appears to be inconsistent use of LimitExceededException by different AWS services. In the case of DynamoDB, this exception indicates that a hard limit was reached, and the request should not be retried (see #1271). But in the case of Kinesis, this is used for throttling describe calls, and it should definitely be retried (with back off).

I'd like to see a custom retryer be added to the kinesis package to retry these exceptions. If that sounds reasonable, I'm happy to submit a PR.

Steps to reproduce

See https://gist.github.com/cbroglie/f790f0df0df342de711402429f69d6ac

[jasdel]

Hi @cbroglie Thanks for reaching out to us about this issue. You're correct the ambiguity of this exception prevented the SDK from retrying this error on a global bases. Also an additional ready this exception was removed is none of the other SDKs retry this exception for the reasons mentioned.

From reading the description of the DescribeStream documentation suggests that this exception is used for both too many resources used, and too many concurrent requests for a resource. I'll reach out to the Kinesis team also to get their feedback on the retryability of this exception in the context of the DescribeStream API call.

Alternatively the following request handler could be added to the Kinesis service client that will enabling retrying of LimitExceededException.

sess := session.Must(session.NewSession())

svc := kinesis.New(sess)
svc.Handler.Retry.PushBack(func(r *request.Request) {
    if r.Operation.Name != "DescribeStream" {
        return
    }
    if aerr, ok := r.Error.(awserr.Error); !ok || aerr == nil {
        return
    }
    if aerr.Code() == kinesis.ErrCodeLimitExceededException {
        r.Retryable = aws.Bool(true)
    }
})

// Use the Kinesis service client.

[cbroglie]

Thanks @jasdel. I'm encountering this issue in the context of running Terraform, and I can add the custom retrier you suggested there if necessary. I just want to confirm whether or not it can be addressed by the SDK first, so please let me know when you hear back from the service team.

[cbroglie]

@jasdel Any updates on this?

[jasdel]

Hi @cbroglie Thanks for getting back with us. I've talked about this with other SDKs and service team and the ambiguity of this exception makes it risking for the SDK to retry directly. I think performing this logic directly by the component using the SDK will have the most insight into the context of the request, and if it should be retryable.

[cbroglie]

@jasdel Thanks for following up. I understand the technical reasons why it's hard for the SDK to solve this generically, and as you can see from the linked PRs, it was fairly trivial to address in Terraform with your sample code.

That said, this isn't a fully satisfying outcome. As a user of the SDK I would like it to be authoritative about which errors should be retried, and encapsulate this centrally, rather than having to maintain a bunch of customizations that need to be ported to each each application. I feel like users would be better served by including these customizations in the SDK.

[jasdel]

Thanks for the feedback @cbroglie. We're investigating how the SDKs can better determine in what circumstances a request can be retried. Currently this is mostly inconsistent tribal knowledge per SDK. We're working with the AWS service teams to provide a better way retries to be more definitive, explicit, and service driven.

[cbroglie]

Sounds good. Feel free to close out this issue if you'd like.

[jasdel]

Thanks, I'll close this. Let us know if you run into additional issues, have feedback, or feature requests for the SDK.

[jasdel]

Opening this issue back up due to #2530. The SDK should be able to support applying retry behavior to specific AWS Services. Its possible that the best solution will be to implement this flexible behavior in the V2 SDK, but it would be good to investigate what options we have for implementing per service retry behavior in the v1 SDK.