DynamoDB Create fails to return on error

[grubernaut]

Version of AWS SDK for Go?

v1.8.21

Version of Go (go version)?

go1.8.1

What issue did you see?

When calling CreateTable() with correct parameters, the function fails to return if the request would exceed the subscriber limit on the number of tables allowed to be created.
We call CreateTable() here: https://github.com/hashicorp/terraform/blob/master/builtin/providers/aws/resource_aws_dynamodb_table.go#L321, but the function never returns the 400 as expected. Is there a wait inside the request function that doesn't return within ~10s?

Expected that the 400 error would be returned so we can handle the error, but the error is never returned.

2017/05/15 10:58:21 [DEBUG] plugin: terraform: aws-provider (internal) 2017/05/15 10:58:21 [DEBUG] [aws-sdk-go] DEBUG: Request dynamodb/CreateTable Details:                                 
2017/05/15 10:58:21 [DEBUG] plugin: terraform: ---[ REQUEST POST-SIGN ]-----------------------------                                                                                         
2017/05/15 10:58:21 [DEBUG] plugin: terraform: POST / HTTP/1.1                                                                                                                               
2017/05/15 10:58:21 [DEBUG] plugin: terraform: Host: dynamodb.us-west-2.amazonaws.com                                                                                                        
2017/05/15 10:58:21 [DEBUG] plugin: terraform: User-Agent: aws-sdk-go/1.8.21 (go1.8.1; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.9.6-dev                                               
2017/05/15 10:58:21 [DEBUG] plugin: terraform: Content-Length: 222                                                                                                                           
2017/05/15 10:58:21 [DEBUG] plugin: terraform: Accept-Encoding: identity                                                                                                                     
2017/05/15 10:58:21 [DEBUG] plugin: terraform: Authorization: (Redacted)                                                             
2017/05/15 10:58:21 [DEBUG] plugin: terraform: Content-Type: application/x-amz-json-1.0                                                                                                      
2017/05/15 10:58:21 [DEBUG] plugin: terraform: X-Amz-Date: 20170515T145821Z                                                                                                                  
2017/05/15 10:58:21 [DEBUG] plugin: terraform: X-Amz-Target: DynamoDB_20120810.CreateTable                                                                                                   
2017/05/15 10:58:21 [DEBUG] plugin: terraform:                                                                                                                                               
2017/05/15 10:58:21 [DEBUG] plugin: terraform: {"AttributeDefinitions":[{"AttributeName":"id","AttributeType":"S"}],"KeySchema":[{"AttributeName":"id","KeyType":"HASH"}],"ProvisionedThrough
put":{"ReadCapacityUnits":5,"WriteCapacityUnits":5},"TableName":"over-cap-test"}                                                                                                             
2017/05/15 10:58:21 [DEBUG] plugin: terraform: -----------------------------------------------------                                                                                         
2017/05/15 10:58:22 [DEBUG] plugin: terraform: aws-provider (internal) 2017/05/15 10:58:22 [DEBUG] [aws-sdk-go] DEBUG: Response dynamodb/CreateTable Details:                                
2017/05/15 10:58:22 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------                                                                                         
2017/05/15 10:58:22 [DEBUG] plugin: terraform: HTTP/1.1 400 Bad Request                                                                                                                      
2017/05/15 10:58:22 [DEBUG] plugin: terraform: Connection: close                                                                                                                             
2017/05/15 10:58:22 [DEBUG] plugin: terraform: Content-Length: 153                                                                                                                           
2017/05/15 10:58:22 [DEBUG] plugin: terraform: Content-Type: application/x-amz-json-1.0                                                                                                      
2017/05/15 10:58:22 [DEBUG] plugin: terraform: Date: Mon, 15 May 2017 14:58:03 GMT                                                                                                           
2017/05/15 10:58:22 [DEBUG] plugin: terraform: Server: Server                                                                                                                                
2017/05/15 10:58:22 [DEBUG] plugin: terraform: X-Amz-Crc32: 2060884299                                                                                                                       
2017/05/15 10:58:22 [DEBUG] plugin: terraform: X-Amzn-Requestid: NDUFTLU30MM8AUMH3AANV73P8FVV4KQNSO5AEMVJF66Q9ASUAAJG                                                                        
2017/05/15 10:58:22 [DEBUG] plugin: terraform:
2017/05/15 10:58:22 [DEBUG] plugin: terraform: {"__type":"com.amazonaws.dynamodb.v20120810#LimitExceededException","message":"Subscriber limit exceeded: There is a limit of 256 tables per s
ubscriber"}

However, I can see that the Create request is retried, instead of returning the error via subsequent POST/RESPONSE debug statements that include the same 400 error.

Steps to reproduce

• Create 255 DynamoDB tables
• Attempt to create another table

Thanks!

[jasdel]

Thanks for creating this issue @grubernaut I'm investigating this issue. You're correct that the SDK is treating the error as a throttling which is causing the significant delay before failure. This error shouldn't be treated as a error.

[jasdel]

After a bit of investigation it looks like this exception is returned for both throttling rate limiting conditions as well as total number of tables crated reaches a limit. I think this needs more investigation into how the SDK should handle this error, because I don't think the SDK knows if it can or cannot retry this request without parsing the error message. Paring the error message would be fragile. I'll reach out to the DynamoDB service team to get a better idea how this error should be handled.

[grubernaut]

Hey @jasdel, thanks for the responses!

Just to note, we're also seeing this issue with Kinesis Firehose Delivery Streams, as noted here: hashicorp/terraform#11143.

If there's any other information or debug output needed, please let me know. Thanks for the help on this one :)

[jasdel]

Hi @grubernaut I created #1276 to address this issue by removing LimitExceededException from the list of error codes that should be throttled.

Since this throttling on +10 create/delete/update concurrent DynamoDB tasks could still fail due to the limit exceeded exception I do not think this would be a breaking change for users as they would need to have logic handling the error code anyways.

[grubernaut]

Works perfectly, thanks for the quick turnaround on this @jasdel. 🎉

[jasdel]

Thanks for the update @grubernaut Let us know if you find any additional cases, or have feedback on the SDK!