An error occurred (Throttling) when calling the DescribeStacks operation (reached max retries: 4)

[smurphnturf]

Relates to:

No response to comments on:
#1610

Description:

When running sam deploy with of two multiple stacks in parallel, Cloud Formation throttles SAM for calling describe-stacks too many times. Cloud Formation rate caps describe-stacks to 5 calls per second and this cannot be increased. In particular, this happens when two different provisioned lambdas are deployed with sam deploy as each deployment can take a few minutes

Steps to reproduce:

Deploy two provisioned lambdas in parallel at the same time

Observed result:

2021-02-15 22:49:02 - Waiting for stack create/update to complete

CloudFormation events from changeset
-------------------------------------------------------------------------------------------------
ResourceStatus           ResourceType             LogicalResourceId        ResourceStatusReason   
-------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS       AWS::Lambda::LayerVers   RenderedPromotionReade   -                      
                         ion                      rDependenciesLayerOne9                          
                                                  4c42c52c2                                       
CREATE_IN_PROGRESS       AWS::Lambda::LayerVers   PromotionSubscriptionH   -                      
                         ion                      andlerDependenciesLaye                          
                                                  rOnefb4f0ee29e                                  
CREATE_COMPLETE          AWS::Lambda::LayerVers   RenderedPromotionReade   -                      
                         ion                      rDependenciesLayerOne9                          
                                                  4c42c52c2                                       
CREATE_IN_PROGRESS       AWS::Lambda::LayerVers   RenderedPromotionReade   Resource creation      
                         ion                      rDependenciesLayerOne9   Initiated              
                                                  4c42c52c2                                       
CREATE_COMPLETE          AWS::Lambda::LayerVers   PromotionSubscriptionH   -                      
                         ion                      andlerDependenciesLaye                          
                                                  rOnefb4f0ee29e                                  
CREATE_IN_PROGRESS       AWS::Lambda::LayerVers   PromotionSubscriptionH   Resource creation      
                         ion                      andlerDependenciesLaye   Initiated              
                                                  rOnefb4f0ee29e                                  
UPDATE_IN_PROGRESS       AWS::Lambda::Function    RenderedPromotionReade   -                      
                                                  rFunction                                       
UPDATE_IN_PROGRESS       AWS::Lambda::Function    PromotionSubscriptionH   -                      
                                                  andler                                          
UPDATE_COMPLETE          AWS::Lambda::Function    RenderedPromotionReade   -                      
                                                  rFunction                                       
UPDATE_COMPLETE          AWS::Lambda::Function    PromotionSubscriptionH   -                      
                                                  andler                                          
CREATE_IN_PROGRESS       AWS::Lambda::Version     RenderedPromotionReade   -                      
                                                  rFunctionVersion36271c                          
                                                  89da                                            
CREATE_COMPLETE          AWS::Lambda::Version     RenderedPromotionReade   -                      
                                                  rFunctionVersion36271c                          
                                                  89da                                            
CREATE_IN_PROGRESS       AWS::Lambda::Version     RenderedPromotionReade   Resource creation      
                                                  rFunctionVersion36271c   Initiated              
                                                  89da                                            
UPDATE_IN_PROGRESS       AWS::Lambda::Alias       RenderedPromotionReade   -                      
                                                  rFunctionAliasLIVE                              
UPDATE_IN_PROGRESS       AWS::Lambda::Alias       RenderedPromotionReade   CodeDeploy deployment  
                                                  rFunctionAliasLIVE       started: d-P9KXYZBA8   
UPDATE_COMPLETE          AWS::Lambda::Alias       RenderedPromotionReade   -                      
                                                  rFunctionAliasLIVE                              
DELETE_IN_PROGRESS       AWS::Lambda::LayerVers   PromotionSubscriptionH   -                      
                         ion                      andlerDependenciesLaye                          
                                                  rOne6b3fdbaffc                                  
UPDATE_COMPLETE_CLEANU   AWS::CloudFormation::S   merchant-web-receipt-    -                      
P_IN_PROGRESS            tack                     private-api                                     
UPDATE_COMPLETE          AWS::CloudFormation::S   merchant-web-receipt-    -                      
                         tack                     private-api                                     
DELETE_COMPLETE          AWS::Lambda::LayerVers   RenderedPromotionReade   -                      
                         ion                      rDependenciesLayerOne5                          
                                                  65bc2990a                                       
DELETE_COMPLETE          AWS::Lambda::LayerVers   PromotionSubscriptionH   -                      
                         ion                      andlerDependenciesLaye                          
                                                  rOne6b3fdbaffc                                  
DELETE_SKIPPED           AWS::Lambda::Version     RenderedPromotionReade   -                      
                                                  rFunctionVersion185d80                          
                                                  2470                                            
-------------------------------------------------------------------------------------------------
Uploading to merchant-web-receipt-private-api/7b9bb1bb6f3347dba1c8f763ce1ba304.template  8546 / 8546.0  (100.00%)
Error: Failed to get outputs from stack: merchant-web-receipt-private-api, An error occurred (Throttling) when calling the DescribeStacks operation (reached max retries: 4): Rate exceeded

Expected result:

Successful deployment

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Both macOS Catalina 10.15.5 and Debian “Buster” 10 (python:3.8.2-slim-buster)
2. sam --version: SAM CLI, version 1.17.0
3. AWS region: ap-southeast-2

[sriram-mv]

Would you want control over the exponential back off that we use, so that can you set your own time for sleeping between retries during throttling or an option to do away with the output?

[smurphnturf]

@sriram-mv Yes exactly, an option to control the initial client_sleep (https://github.com/aws/aws-sam-cli/blob/develop/samcli/lib/deploy/deployer.py#L80) would solve the problem for us. Since there are many deployments we run that we know will take a few minutes so we don't need instant reporting on the deployment.

As well as an option to control the max_attempts (https://github.com/aws/aws-sam-cli/blob/develop/samcli/lib/deploy/deployer.py#L84) would be nice but controlling the initial client_sleep would be enough.

As for the output we would always want the output so no option regarding that needed.

Thank you

[smurphnturf]

@sriram-mv Any idea how long a change like this might take to go through reviews? I've never contributed to this project before so I have no idea. But our team is directly affected by this everyday so we are all watching closely.

[fschroder-slyp]

Hi Aws, any updates on this?
We really want to get this built to give it a try in our environment.

[sriram-mv]

@smurphnturf Thanks for your patience! I looked into into this and left some comments on the PR

[fschroder-slyp]

hey @sriram-mv, we left a few comments in #2748, could you have a look? Thx!

[dfuentes77]

I also ran into this issue when deploying two provisioned lambdas locally.

[fschroder-slyp]

Hi, any updates? We're happy to do the work, but we want to make sure it will be accepted.

[pjungermann]

I reproduced this with a bunch of parallel processes with aws cloudformation describe-stacks calls in a loop and a sam deploy for the same stack executing a change set which then failed with "Rate exceeded".

For me, adding max_retries to the aws config file helped

retry_mode = standard  # default at aws cli v2
max_attempts = 6       # whatever works for you

or similarly using the env vars

export AWS_RETRY_MODE=standard
export AWS_MAX_ATTEMPTS=6

Both as described at https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-retries.html#cli-usage-retries-configure

https://github.com/aws/aws-sam-cli/blob/develop/samcli/lib/deploy/deployer.py#L353
If I get it right sam-cli is wrapping the use of boto3/the CloudFormation client with their own retry logic with retry attempts defined at https://github.com/aws/aws-sam-cli/blob/develop/samcli/lib/deploy/deployer.py#L84

To achieve more retries overall, you could configure both, boto3 and/or sam-cli. For sam-cli there is no option yet to influence it, but influencing boto3 works already via the aws config file or env vars.

Still need to change and try this for the actual deployment pipelines for some time, but I'm confident after my tests so far.

PS: I used SAM CLI, version 1.29.0

[pjungermann]

btw: I also encountered terminal errors because of throttling at the StackCreateComplete waiter which uses describe-stacks. Not only describe-stack-events.

botocore.exceptions.WaiterError: Waiter StackCreateComplete failed: An error occurred (Throttling): Rate exceeded