Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rate Exceeded errors on deployment #1647

Closed
hoegertn opened this issue Jan 31, 2019 · 7 comments · Fixed by #2053 · May be fixed by MechanicalRock/account-reaper#6
Closed

Rate Exceeded errors on deployment #1647

hoegertn opened this issue Jan 31, 2019 · 7 comments · Fixed by #2053 · May be fixed by MechanicalRock/account-reaper#6
Assignees
@rix0rrr
Labels
bug This issue is a bug. package/tools Related to AWS CDK Tools or CLI

Comments

@hoegertn
Copy link
Contributor

When deploying a CDK app with multiple stacks I can the following error if no changes are to be performed:

Deployment of stack XYZ failed: Throttling: Rate exceeded
Rate exceeded

This does not happen if there are changes. Maybe CDK should pause after each stack or retry on rate exceeded errors.
@eladb
Copy link
Contributor

eladb commented Feb 4, 2019

We need a few more details in order to be able to diagnose this. Perhaps you can try to run cdk deploy with --verbose and send us the logs?

@eladb eladb added bug This issue is a bug. package/tools Related to AWS CDK Tools or CLI needs-response labels Feb 4, 2019
@hoegertn
Copy link
Contributor Author

hoegertn commented Feb 7, 2019 

gitlab-runner-foo
Waiting for stack CDKToolkit to finish creating or updating...
gitlab-runner-foo: deploying...
s3://cdktoolkit-stagingbucket-1je31rbpo4qo7/cdk/gitlab-runner-foo/7ee729a7831685cbd57889d8848186186c25e144258212803e50525550fc6989.yml: checking if already exists
s3://cdktoolkit-stagingbucket-1je31rbpo4qo7/cdk/gitlab-runner-foo/7ee729a7831685cbd57889d8848186186c25e144258212803e50525550fc6989.yml: uploading
s3://cdktoolkit-stagingbucket-1je31rbpo4qo7/cdk/gitlab-runner-foo/7ee729a7831685cbd57889d8848186186c25e144258212803e50525550fc6989.yml: upload complete
Stored template in S3 at: https://cdktoolkit-stagingbucket-1je31rbpo4qo7.s3.amazonaws.com/cdk/gitlab-runner-foo/7ee729a7831685cbd57889d8848186186c25e144258212803e50525550fc6989.yml
Attempting to create ChangeSet CDK-221a622d-546c-4629-a97f-714e1e3e8f10 to update stack gitlab-runner-foo
gitlab-runner-foo: creating CloudFormation changeset...
Initiated creation of changeset: arn:aws:cloudformation:eu-central-1:719766288726:changeSet/CDK-221a622d-546c-4629-a97f-714e1e3e8f10/681b7950-40cd-46f3-89e6-77bc632020d3; waiting for it to finish creating...
Waiting for changeset CDK-221a622d-546c-4629-a97f-714e1e3e8f10 on stack gitlab-runner-foo to finish creating...
No changes are to be performed on gitlab-runner-foo, assuming success.

 ❌  gitlab-runner-foo failed: Throttling: Rate exceeded
Rate exceeded
Throttling: Rate exceeded
    at Request.extractError (/builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/builds/gitlab-ci/gitlab-ci-runner/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! gitlab-runner-infra@0.1.0 deploy: `cdk deploy -o build --verbose`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the gitlab-runner-infra@0.1.0 deploy script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-02-07T09_41_46_071Z-debug.log

@hoegertn
Copy link
Contributor Author

Anything new on this issue?

@hoegertn
Copy link
Contributor Author

@eladb @rix0rrr Are there any plans for this issue? This is breaking nearly all of our builds.

@rix0rrr rix0rrr self-assigned this Mar 19, 2019
@rix0rrr
Copy link
Contributor

rix0rrr commented Mar 19, 2019

Okay I might be able to take this today.

I'm curious though what other process you have running that is eating up your CloudFormation call rate?

@hoegertn
Copy link
Contributor Author

None. This is the only job running in this AWS account. Only CDK deploying about 20 stacks within one app

rix0rrr added a commit that referenced this issue Mar 19, 2019
@rix0rrr
fix(toolkit): increase retry count for operations
a97b58b 
This will help in situations where the default retry behavior is
not enough to cover throttling errors.

Fixes #1647.
@rix0rrr rix0rrr mentioned this issue Mar 19, 2019
Merged
4 tasks
rix0rrr added a commit that referenced this issue Mar 19, 2019
@rix0rrr
fix(toolkit): increase number of retries (#2053)
133dc98 
This will help in situations where the default retry behavior is
not enough to cover throttling errors.

Fixes #1647.
This was referenced Dec 12, 2019
Closed
Open
Open
@danielfariati danielfariati mentioned this issue Jan 3, 2020
Open
@rstacruz
Copy link

I came across this today with my colleague @baxang. It turns out that the Stack was creating around ~300 resources. AWS has a limit on how many resources a Stack can have.

The solution for us was to use nested stacks to separate other resources into sub-stacks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rix0rrr rix0rrr

Labels
bug This issue is a bug. package/tools Related to AWS CDK Tools or CLI
Projects
None yet
Milestone
No milestone
4 participants
@rstacruz @rix0rrr @eladb @hoegertn