Disastrous Throttling: Rate exceeded error #705
Comments
|
This can likely be solved with some well placed dependsOn conditions in the templates. We could limit the number of concurrent updates to N if we place a dependsOn statement on every stack after the N'th stack M that depends on the (M-N)th stack. If we had 15 stacks in the root and wanted to limit the number of concurrent updates to 10 then the 11th stack would depend on the 1st stack, the 12th stack would depend on the 2nd stack, and so on. I have been looking at a similar strategy within the API category but have not found a reliable way to reproduce bad deployments like this. Is there some pattern or resource type you found that always causes this to break? |
|
Thanks for the quick response. The only thing I have observed is that there is a lot of concurrency in the amplify push even if the majority of the amplify status of the larger project is 'No Change'. Is it necessary for 'No Change' to spawn push activity? |
|
@leantide Did you try pushing it again with an |
yuth
added
investigating
|
Yes, it continued to fail even after about 5 push retries, including an effort to reduce the number of services via 'amplify remove', which still tried to execute the same amount of concurrency. But I don't know the dynamics of the CloudFormation rate limiting. Maybe once I tripped the threshold, successive pushes failed because my CloudFormation was still in the exceeded state waiting for a backoff timeout. Not sure. The only way I could recover was to perform an 'amplify delete' which wiped out all AWS resources (except S3 for some reason which I removed manually). |
|
Having to rebuild my backend, I proceeded to push after every service I added. This worked fine up until 16 services. Then I adding 1 more Rest API (which adds an API, a Function, and a NoSQL Storage), putting me at 19 services (see table) and now I am having the CloudFoundation 'rate exceeded' error again. I must say that I am struggling a bit to see Amplify-CLI in its present condition being used for a backend of any non-trivial size.
|
|
@leantide We’re looking into it and would be providing a fix for this soon. |
|
Thank you. I am also observing that the console logging cycles through each resource several times, initially saying UPDATE_COMPLETE, but then later saying UPDATE_IN_PROGRESS for the same resource. Maybe this is normal. Not sure. For example: UPDATE_COMPLETE apiitems |
|
Just remembered that I am using the @multienv branch of amplify-cli. I don't know if the issue is just this branch or in the mainline also. Will your fix be applied on both lines? |
|
Yes, that is normal since we are deploying the parent Cloudformation stack with the corresponding resources as it’s children nested stacks. Cloudformation inherently looks at the diff and does nothing if there’s no diff in the corresponding resource template. The throttling is happening when the CLI is trying to read the stack events for the 19 nested child Cloudformation stacks at once. So even though your Cloudformation stack is getting deployed successfully, due to the throttling of the stack read events, the CLI is erroring out. We’re planning to gracefully handle this error in the CLI and apply an exponential backoff mechanism to avoid this throttling. |
|
Fantastic. This is most appreciated. |
|
Yes, this change would be applicable for both the branches. |
kaustavghosh06
added
Work in progress
and removed
investigating
|
We have released the fix is the following versions. Please upgrade the CLI
Please feel free to reopen the issue if this is still happening after updating the CLI |
ghost
mentioned this issue
|
This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs. Looking for a help forum? We recommend joining the Amplify Community Discord server |
Describe the bug
After using amplify-cli to add several more AWS resources to my configuration, I observe amplify push will create and execute over 20 nested stacks and then have an error as it hits the CloudFormation throttling limit, possibly because of the concurrency and the rate of the CloudFormation service calls that amplify is executing. Console log says "uncaughtException: Rate exceeded" and "stack=[Throttling: Rate exceeded". Verified by AWS support that CloudFormation logs indicate the Throttling limit occurs.
Upon seeing this 'rate exceeded' response, can amplify slow its push efforts down instead of terminating on an error?
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The backend is pushed without error
Console log:
⠙ Updating resources in the cloud. This may take a few minutes...2019-01-14T14:31:46.671Z - error: uncaughtException: Rate exceeded date=Mon Jan 14 2019 09:31:46 GMT-0500 (EST), pid=87823, uid=501, gid=20, cwd=/Users/ken/Documents/RedGazelle Project/RedGazelle, execPath=/usr/local/Cellar/node@8/8.14.0/bin/node, version=v8.14.0, argv=[/usr/local/Cellar/node@8/8.14.0/bin/node, /usr/local/bin/amplify, push], rss=204918784, heapTotal=101109760, heapUsed=70467192, external=1424975, loadavg=[3.12548828125, 3.80419921875, 3.302734375], uptime=4396173, trace=[column=29, file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/protocol/query.js, function=Request.extractError, line=50, method=extractError, native=false, column=20, file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/sequential_executor.js, function=Request.callListeners, line=106, method=callListeners, native=false, column=10, file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/sequential_executor.js, function=Request.emit, line=78, method=emit, native=false, column=14, file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/request.js, function=Request.emit, line=683, method=emit, native=false, column=10, file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/request.js, function=Request.transition, line=22, method=transition, native=false, column=12, file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/state_machine.js, function=AcceptorStateMachine.runTo, line=14, method=runTo, native=false, column=10, file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/state_machine.js, function=null, line=26, method=null, native=false, column=9, file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/request.js, function=null, line=38, method=null, native=false, column=12, file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/request.js, function=null, line=685, method=null, native=false, column=18,file=/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/sequential_executor.js, function=Request.callListeners, line=116, method=callListeners, native=false], stack=[Throttling: Rate exceeded, at Request.extractError (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/protocol/query.js:50:29), at Request.callListeners (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/sequential_executor.js:106:20), at Request.emit (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/sequential_executor.js:78:10), at Request.emit (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/request.js:683:14), at Request.transition (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/request.js:22:10), at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/state_machine.js:14:12), at /usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/state_machine.js:26:10, at Request. (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/request.js:38:9), at Request. (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/request.js:685:12), at Request.callListeners (/usr/local/lib/node_modules/@aws-amplify/cli/node_modules/amplify-provider-awscloudformation/node_modules/aws-sdk/lib/sequential_executor.js:116:18)]
The text was updated successfully, but these errors were encountered: