Reduce number of parameters used by assets

[alexdilley]

• I'm submitting a ...

  • 🪲 bug report
  • 🚀 feature request
  • 📚 construct library gap
  • ☎️ security issue or vulnerability => Please see policy
  • ❓ support request => Please see note at the top of this template.

• What is the current behavior?

CloudFormation stacks are limited to 60 parameters; CDK produces a seemingly excessive number of parameters, thus easy to breach limitation.

• What is the expected behavior (or behavior of feature suggested)?

To perhaps instead use mappings, as suggested in the CF docs.

• What is the motivation / use case for changing the behavior or adding this feature?

To be able to define, for example, more than 20 Lambda functions in a stack: currently – for each function – CDK generates one parameter for its artifact hash, one for its S3 location, and one for its version.

[NGL321]

Hi @alexdilley,

Thank you for reaching out!
We are aware of this gap, and will address it when able. Someone will update this issue when that happens.

[niranjan94]

Hi @NGL321,

Thanks for your response.

We, Just started using CDK and ended up with this issue. We have 21 lambdas and the parameter count for it comes to more than 63. This makes it impossible for us to use CDK right now to deploy as cloudformation doesn't allow it. Anyway to make CDK inline the parameters (bucket/key/version hash) within the template directly instead of passing these as parameters ?

[niranjan94]

Current workound that I have been using is

1. synthesizing the template with cdk
2. then modifying the template to include s3 bucket and uri inline in the template instead of parameters
3. then using aws cloudformation deploy to deploy instead of cdk deploy

Code snippet incase anyone else needs to use it:
https://gist.github.com/niranjan94/92f2636a29f09bd6cc53085951e78046

[eladb]

The long-term solution should be to have only a single parameter per CDK asset. It will still require a parameter per asset since asset locations change based on their source hash but it will reduce the number of parameters by a factor of x3

[sublimemm]

The long-term solution should be to have only a single parameter per CDK asset. It will still require a parameter per asset since asset locations change based on their source hash but it will reduce the number of parameters by a factor of x3

I do not see why all parameters cannot be placed into a map parameter. They must have unique keys/ids already since parameters cannot have duplicated names, correct?

I do not think the community would consider a stack limited to 30 assets to be the long term solution. 30 assets, assuming the users needed no parameters of their own (a pretty faulty assumption imho).

[dehli]

@sublimemm I think that would also be the best way; however I'm not sure how it could be a map. CloudFormation doesn't seem to support pulling out values from a map unless it's specifically a Mapping. Is there some function I'm missing?

What I've seen is having a long string as the parameter (joining all the parameters with a special character such as |) and then using a combination of !Split and !Select. CDK would have to maintain the appropriate indices for each asset with this solution.

[sublimemm]

@eladb You said the one parameter per asset would reduce the number of parameters by a factor of 3, but I think it will actually do much more. Our stack has 5 assets and between them 63 parameters. So reducing it to 5 would be a much more palatable solution.

It's unclear to me when the CDK decides something is a new asset or not... but I thought your original suggestion was one parameter per construct/lambda/etc. One per asset seems tenable, assuming the cdk is diligent with splitting the stack into assets.

[dehli]

@sublimemm I just looked at my template and it looks like it follows the 3 parameters per asset. Maybe you have something else adding parameters?

[sublimemm]

After some digging, it's all about the lambdas. Its adding 3 per lambda, we have tons of lambdas, there are many per asset.

[sublimemm]

Here are some examples, you can see they're duplicating the bucket (obviously not needed since all CDK stacks are deployed to the same bucket (the toolkit bucket)).

They're also duplicating parameters if multiple lambdas share an asset bundle (we have tons of them that share the same asset path).

[eladb]

We are looking into improving this as part of our work on CI/CD. The current thinking is to actually reduce the number of asset parameters to zero by using a well-known convention-based physical names for the bootstrapping resources (bucket/ECR repository) and the source hash as the key (S3 object key/docker image tag). This will basically mean that we don't need any degrees of freedom during deployment. I am curious about people's thoughts on this...

[sublimemm]

We are looking into improving this as part of our work on CI/CD. The current thinking is to actually reduce the number of asset parameters to zero by using a well-known convention-based physical names for the bootstrapping resources (bucket/ECR repository) and the source hash as the key (S3 object key/docker image tag). This will basically mean that we don't need any degrees of freedom during deployment. I am curious about people's thoughts on this...

I think that is a great solution.

[dsmrt]

To add to this...
I'm hitting my max parameters (60) while using CDK assets for AppSync FunctionConfigs and Resolvers. I have about 33 resolvers and it looks like CDK creates 3 parameters per asset (as mentioned above). Asset parameters to zero would help me a ton!

[tekdroid]

I'm attempting to migrate from raw CF YAML templates to the CDK, immediately hit this issue. It generated 120 parameters first attempt. Immediately blocked by this. I just spent 3 days converting templates thinking "surely cdk handles maximum limits". Very very discouraging to see this happen before ever being able to deploy our structure. We have 10s of thousands of lines of config and I want to move the team away from that.

Would love to see some progress made toward this fix.

In the meantime, anyone have a workaround aside from the one above? Would re-structuring the hierarchy in some way help? I've tried to create extra layers but they then just pass the parameters through the layer and the problem remains.

[niranjan94]

@tekdroid yep. I tried restructuring too. But didn't work out. So I'm still using the workaround I told about above. (Inlining all the asset paths into the template) and using aws cloudformation deploy to deploy instead of cdk deploy

Updated script to do the inlining in case you are interested:

https://gist.github.com/niranjan94/92f2636a29f09bd6cc53085951e78046

[tekdroid]

@niranjan94 thanks for the updated script, I appreciate that! I'll check it out and see if I can use this method for now. Cheers!

[tekdroid]

@niranjan94 Your script is not working as expected for me. Would you mind helping me out? It removed all of the parameters, but didn't replace their usages.

Is there another medium by which we could chat for a moment? If you can't that's fine, I can reverse engineer this script.

Mainly I just wanted to see if I'm running the synthesis of the cdk output the same way you are. Looks like you're looking for Resource -> Properties -> Content, but I don't have a node at that path. I have Resource -> Properties -> TemplateURL, which is a join function for the s3 bucket/key parameters that were removed.

EDIT: No worries, after second thought I really don't want to go down this road. I'll continue our teams conversion when this is officially fixed inside the CDK.

[hoegertn]

@eladb I really like reducing the number to zero and using asset hashes. For the toolkit bucket, I would suggest an Fn::ImportValue instead of convention-based names to prevent name squatting attacks.

[eladb]

We can't use Fn::ImportValue because we need to know before deployment where to upload assets (see the cdk-assets RFC).

Copy @rix0rrr

[eladb]

@rix0rrr Another issue: When using ContainerImage.fromAsset the generated CFN template looks like this:

            "Image": {

              "Fn::Sub": {

                "Fn::Join": [

                  "",

                  [

                    "<id>.dkr.ecr.eu-central-1.",

                    {

                      "Ref": "AWS::URLSuffix"

                    },

                    "/cdk-<my-asset-ecr>:<hash>"

                  ]

                ]

              }

            },

This is invalid CloudFormation as Join is not valid inside Sub:

E1019 Sub should be a string or array of 2 items for Resources/TaskDef3BF4F22B/Properties/ContainerDefinitions/0/Image/Fn::Sub

Am I doing something wrong or is this a bug in the new asset system?

EDIT: One more: the S3 Bucket Deployment construct does not work anymore as the custom lambda does not have permissions to use the KMS key of the asset bucket.

This seems like a bug. Can you please raise a separate issue?

[eladb]

THIS IS GLORIOUS - thanks heaps! Works a charm.

Here's someone popping you all a well deserved beer:

ps. When this gets documented for general use, there should be a a big warning bit about --cloudformation-execution-policies as I initially left it off (why?!?!) and that left me to go through a tunnel of pain to return to the state CloudFormation was before I attempted it. Following your instructions worked perfectly.

Thanks so much!

Mind raising a separate issue for that?

[brennick]

@rix0rrr will the new asset system be the default system ever or will we always need to adjust cdk.json?

[damonmcminn]

THIS IS GLORIOUS - thanks heaps! Works a charm.
Here's someone popping you all a well deserved beer:

ps. When this gets documented for general use, there should be a a big warning bit about --cloudformation-execution-policies as I initially left it off (why?!?!) and that left me to go through a tunnel of pain to return to the state CloudFormation was before I attempted it. Following your instructions worked perfectly.
Thanks so much!

Mind raising a separate issue for that?

Absolutely.

[brennick]

After running the new bootstrap command, I'm getting the following error when doing cdk deploy

Could not assume role in target account (did you bootstrap the environment with the right '--trust's?): Roles may not be assumed by root accounts.

Any idea what I'm doing wrong?

EDIT: It was due to using root credentials in our CLI. Creating a IAM User resolved the issue.

[eladb]

@rix0rrr Is this documented in the CDK pipelines docs?

[mattiLeBlanc]

@rix0rrr I am having a similar issue:

Could not assume role in target account (did you bootstrap the environment with the right '--trust's?): User: arn:aws:sts::[accountID]:assumed-role/AWSReservedSSO_AdministratorAccess_ddb.../[USER]is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::[accountId]:role/cdk-hnb659fds-deploy-role-[accountId]-ap-southeast-2

So the first time I bootstrap and deploy, it works. But any sub-sequential deployment I get this error.

Only when I delete my S3 bucket and bootstrap again, can I deploy. So I can only deploy one time locally before I have to do this again.

However this wouldn't work for us in deployment because it would be madness to remove the staging bucket after each deploy.
I am kind stuck now with the option to either downgrade and create sub stacks for my more than 20 lambdas, or create a postdeploy cleanup that removes that staging bucket so we can do more than 1 deploy :-/

My CDK version is 1.49.1
We use SSO and have programmatic access for our local deployment via the aws credential file.

[mattiLeBlanc]

@NGL321 So we still have issues with deploying our stack (appsync api with lambda's, more than 20 = 60+ params) with the new CDK synthesis. The error is as mentioned in my prev post above.

Our accounts are DEV, STAGING And PROD and we have SSO users with read/write permissions and we use the ~/.aws/credentials file with our token to local deploy to our personal version of the API in the DEV account.
So when I was the first to use the new param system, I bootstrapped it with my account. This created/updated the CDKToolkit stack and added the S3 bucket to our DEV account.
All good...
But then when my colleague also started to use the new CDK he couldn't deploy, so he had to remove the CDKToolkit and bootstrap again. Now he could deploy...but surprise, I coulnd't again.
To make things even worse, when we pushed our changes to Bitbucket, the pipeline tries to deploy the DEV version of the API to the DEV account (in which we already bootstrapped), so it was complaining that the S3 bucket already exists:

StagingBucket Requested update requires the creation of a new physical resource; hence creating one.
  7/11 | 1:28:09 AM | UPDATE_FAILED        | AWS::S3::Bucket       | StagingBucket cdk-hnb659fds-assets-[account number]-ap-southeast-2 already exists
  5/11 | 1:28:09 AM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack | CDKToolkit The following resource(s) failed to update: [StagingBucket]. 
  5/11 | 1:28:12 AM | UPDATE_COMPLETE      | AWS::S3::Bucket       | StagingBucket 

So this becomes a bit hairy.
So:
A) should we move our all personal accounts to another env called USER so that DEV solely has one API and one bucket for deployment, and we have another bucket for our personal API deployment in the USER account?
B) should the bootstrap script be more robust to check if a bucket already exists so it doesn't have to recreate it?
C) why are we getting these trust issues. I am not sure if we can set a trust relationship on an SSO user. Is the issue because Bob bootstraps with DeveloperReadWrite SSO user and then when James deploys he has a different user signature so that bootstrapped bucket doesn't like him?

Our permission set for the developers includes "sts:*", so not sure what else is required to make it be trusted, and assume a role.

[ghost]

After running the new bootstrap command, I'm getting the following error when doing cdk deploy

Could not assume role in target account (did you bootstrap the environment with the right '--trust's?): Roles may not be assumed by root accounts.

Any idea what I'm doing wrong?

EDIT: It was due to using root credentials in our CLI. Creating a IAM User resolved the issue.

I'm using IAM user, but still has same error with you. Do I need new an IAM user in CLI?

[ajsivsan]

Updated the CDK to use the new bootstrap and synth still shows 63 parameters. I'm uploading a bunch of files Assets and referring it using s3_url. Does the new bootstrap work for S3 Asset class as well?

[hsks]

For anyone who is struggling, here's how we made it work in our case:
We followed @rix0rrr 's suggestion and that helped mostly (thanks for your valuable suggestion). However, we faced some minor issues even after that.
After bootstrapping the environments successfully, we still hit the parameter limit. However, a quick update of the cdk CLI to the latest version (1.67.0) in our case resolved that limit.
Moreover, we wanted to deploy using a user other than the administrator. Cloudformation throws self explanatory errors in such cases. As a fix, we need to add the deployment user under consideration as a trusted entity to some of the roles created by cdk after bootstrapping the environments (these roles are clear for the cloudformation error messages).

Doing so made it work perfectly.

[jshaw-vf]

For anyone who is struggling, here's how we made it work in our case:
We followed @rix0rrr 's suggestion and that helped mostly (thanks for your valuable suggestion). However, we faced some minor issues even after that.
After bootstrapping the environments successfully, we still hit the parameter limit. However, a quick update of the cdk CLI to the latest version (1.67.0) in our case resolved that limit.
Moreover, we wanted to deploy using a user other than the administrator. Cloudformation throws self explanatory errors in such cases. As a fix, we need to add the deployment user under consideration as a trusted entity to some of the roles created by cdk after bootstrapping the environments (these roles are clear for the cloudformation error messages).

Doing so made it work perfectly.

Nice, our team solved this the same way and ran into similar issues. It would be nice to have a list of the exact permissions a role would need to have least privilege access to make this work.

[hsks]

For anyone who is struggling, here's how we made it work in our case:
We followed @rix0rrr 's suggestion and that helped mostly (thanks for your valuable suggestion). However, we faced some minor issues even after that.
After bootstrapping the environments successfully, we still hit the parameter limit. However, a quick update of the cdk CLI to the latest version (1.67.0) in our case resolved that limit.
Moreover, we wanted to deploy using a user other than the administrator. Cloudformation throws self explanatory errors in such cases. As a fix, we need to add the deployment user under consideration as a trusted entity to some of the roles created by cdk after bootstrapping the environments (these roles are clear for the cloudformation error messages).
Doing so made it work perfectly.

Nice, our team solved this the same way and ran into similar issues. It would be nice to have a list of the exact permissions a role would need to have least privilege access to make this work.

I completely agree. We usually wait for deployments to break before modifying the trust relationships. Helps keep the permission boundary as intact as possible.
Hopefully folks at AWS will document this soon.

[MinCheTsai]

The new asset system is available properly starting at 1.45.0.

Easiest way to enable it is to put this into your cdk.json:

{
  "context": {
    "@aws-cdk/core:newStyleStackSynthesis": true
  }
}

You also have to bootstrap any environments you want to deploy into using the new version of the bootstrap stack:

$ env CDK_NEW_BOOTSTRAP=1 npx cdk bootstrap \
  --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess \
  aws://123456789012/us-east-2

This will create a new S3 Bucket and ECR Repository with a predictable name, which we will reference directly from the template without the use of asset parameters.

I have used this solution in our projects, it's work! [CDK v1.92.0]

@rix0rrr @eladb I have a maintenance question.
This solution seem hasn't public to anyone, I careful it will use for long term project, can I keep and continue upgrade CDK version in the feature? The project stacks can't create again(in production).

[henrist]

CloudFormation increased the limit from 60 parameters to 200 parameteres in October (https://aws.amazon.com/about-aws/whats-new/2020/10/aws-cloudformation-now-supports-increased-limits-on-five-service-quotas/)

[eladb]

This is solved with the new bootstrap stack which is now the default in v2 - I believe this can be closed. Feel free to add additional comments if anyone feels differently.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.