aws/request: Default Retryer does retry request body timeouts

[RendijsSmukulis]

Version of AWS SDK for Go?

Release v1.6.2 (2016-12-08)

Version of Go (go version)?

go1.8 windows/amd64

What issue did you see?

We've seen a number of failures when sending batches to Firehose that should have been retried, but were not.
Error in the logs:

SerializationError: failed decoding JSON RPC response
caused by: net/http: request canceled (Client.Timeout exceeded while reading body)

My understanding is that this is caused by the timeout happening during the serializer reading the response body. If I simulate a response timing out by not even sending headers, I get this error:

RequestError: send request failed
caused by: Post http://localhost:5454//: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

which is retried by the Retryer logic.

Currently, retryer.go does not include the SerializationError in the retryableCodes map (as seen here), so either the SerializationError needs to be added to the Retryer's retryableCodes map, or the underlying code should throw RequestTimeout, not SerializationError when a read of response's body takes too long.

[jasdel]

Thanks for the heads up on this @RendijsSmukulis I think this is an extension of logic added a few weeks ago to retry connection resets. I think we can add a case to the SDK's retrier logic to catch this case.

[jasdel]

Hi @RendijsSmukulis Thanks for letting us know about this issue. I just merged in #1289 which adds support for retrying temporary error such as Client.Timeout that occur when the SDK is reading the response body. Let us know if you run into any additional issues, or have feedback.