From 444a51059d382ade91363ce5c5da12901b45d55a Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Mon, 25 Apr 2022 17:40:14 -0700 Subject: [PATCH 01/53] RFC 39: Reduce aws-cdk-lib Package Size --- text/0039-reduce-module-size.md | 516 ++++++++++++++++++++++++++++++++ 1 file changed, 516 insertions(+) create mode 100644 text/0039-reduce-module-size.md diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md new file mode 100644 index 000000000..8885b0afc --- /dev/null +++ b/text/0039-reduce-module-size.md @@ -0,0 +1,516 @@ +# Reduce aws-cdk-lib package size + +* **Original Author(s):**: @madeline-k +* **Tracking Issue**: #39 +* **API Bar Raiser**: tbd + +The AWS CDK v2 framework library, `aws-cdk-lib` contains all packages from AWS +CDK v1 in one monolithic package. While this is a great value for customers in +making dependency management much easier, it has also resulted in increasing the +unpacked npm package size to 238 MB, at the time of writing. This feature +reduces the size of the package to 100 MB. + +## Working Backwards + + +* **CHANGELOG**: + +feat(aws-cdk-lib): Reduce aws-cdk-lib package size. + +‼️ If you use any of the following Constructs in a network-restricted +environment, you might encounter problems upgrading to this version. ‼️ +* lambda_layer_kubectl.KubectlLayer +* aws_eks.KubectlProvider +* aws_eks.HelmChart +* aws_eks.KubernetesManifest +* aws_eks.KubernetesObjectValue +* aws_eks.KubernetesPatch +* lambda_layer_awscli.AwsCliLayer +* aws_s3_deployment.BucketDeploymentß +* aws_stepfunctions_tasks.EmrContainersStartJobRun +* lambda_layer_node_proxy_agent.NodeProxyAgentLayer +* aws_eks.FargateProfile +* aws_eks.Cluster + +Please see https://github.com/aws/aws-cdk/issues/1234 for more details, and +comment on the issue if you run into problems. + +* **CLI Notices**: + +We can deliver a notice to customers based on what specific constructs they are +using. We will define separate notices for each different scenario. + +Title: (aws-cdk-lib): upcoming change in packaging structure might require +action to upgrade + +Body: We’ve identified that you are using at least one of the Constructs. If you are running your aws-cdk commands in an +environment that does not have access to npm, then you will need to make a +change to make version 2.x of available in your environment. See +https://github.com/aws/aws-cdk/issues/1234 for details and instructions. + +| List of Constructs | npm packages | +| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| KubectlLayer | @aws-cdk/lambda-layer-kubectl | +| KubectlProvider, HelmChart, KubernetesManifest, KubernetesObjectValue, or KubernetesPatch | @aws-cdk/lambda-layer-kubectl and @aws-cdk/lambda-layer-awscli | +| AwsCliLayer, BucketDeployment, or EmrContainersStartJobRun | @aws-cdk/lambda-layer-awscli | +| NodeProxyAgentLayer, or FargateProfile | @aws-cdk/lambda-layer-node-proxy-agent | +| aws_eks.Cluster | @aws-cdk/lambda-layer-kubectl, @aws-cdk/lambda-layer-awscli, and @aws-cdk/lambda-layer-node-proxy-agent | + +--- + +Ticking the box below indicates that the public API of this RFC has been +signed-off by the API bar raiser (the `api-approved` label was applied to the +RFC pull request): + +``` +[ ] Signed-off by API Bar Raiser @xxxxx +``` + +## Public FAQ + +### What are we launching today? + +Today, we are launching a smaller version of the AWS CDK v2 framework library, +`aws-cdk-lib`. And, a change in the AWS CDK v2 CLI to support this smaller +framework library. Before today’s release, the unpacked size of this package was +238 MB. This large size prevented many customers from being able to use it, and +caused performance problems for others. It also prevented the CDK team from +releasing certain new features. Now, the size of the package is 100 MB. This +smaller version also has a mechanism for adding new lambda-layer dependencies. +Because of this launch, the AWS CDK framework will be able to add support for +more versions of kubectl, and awscliv2. + +### Why should I use this feature? + +You should upgrade your dependencies on `aws-cdk-lib` to get the benefits of a +smaller package size. If you use `aws-cdk-lib` in a lambda function, now you can +include more code and dependencies before hitting the lambda function code size +limit of 250 MB. If you use `aws-cdk-lib` in your CI/CD pipelines and are +frequently downloading it with a fresh npm install, you will see performance +improvements in the download times. + +## Internal FAQ + +### Why is a large package a problem? Why are we doing this? + +There are many reasons this is an issue for customers. The following items need +to be addressed. + +1. The package managers that aws-cdk-lib is published on have real or + recommended filesize limits. For example: + 1. The v2 release currently gets this warning when publishing for Go. `Go: + remote: warning: File awscdk/jsii/aws-cdk-lib-2.17.0.tgz is 55.42 MB; + this is larger than GitHub's recommended maximum file size of 50.00 MB` + 2. NPM itself does not have a package size limit, but other services that + host NPM packages do. For example, [Azure Artifacts] + (https://docs.microsoft.com/en-us/azure/devops/artifacts/reference/limits?view=azure-devops) + has a limit of 500 MB. + 3. PyPi has an unofficially documented limit of 60 MB +2. A large download size might be problematic for customers with weak internet + connections or limited bandwidth. +3. Customers who download and install `aws-cdk-lib` on every CI/CD pipeline run + may be frustrated to spend resources frequently downloading a large package. +4. AWS Lambda limits the size of your function code to 250 MB. +5. The AWS Lambda limit for all uploaded assets for a Function is 75 GB. If a + customer is using Lambda Versions, and wants to maintain a Version for every + version they have published, they might end up hitting this limit because of + the large size of `aws-cdk-lib`. +6. The developer experience of using `aws-cdk-lib` in an IDE might be slower in + some cases. +7. The CDK project is currently blocked from including more ‘large + dependencies’, such as multiple version of kubectl, multiple version of the + AWS CLI, and dedicated versions of the AWS SDK for use in custom resources. + This is a decision we’ve made to avoid increasing the size of the package + further until a more scalable solution for these kinds of dependencies is + implemented. + +Most of these above issues have workarounds. For running CDK apps in Lambda, you +can use a Docker image for your Lambda code, and the limit becomes 10GB. For +package manager size limits, we can request [increases] +(https://github.com/pypa/pypi-support/issues/1642), and those are typically +approved. However, publishing a large package is a bad practice in the +open-source software ecosystem. Even if a customer has no technical limitations +for adding a ~240 MB dependency to their package, it is a red flag to them when +considering using `aws-cdk-lib`. We should be customer-obsessed, and solve the +problem of aws-cdk-lib being a huge dependency, before customers start +complaining about it. + +### Why is the package so large? + +There are some large assets contributing to the size of aws-cdk-lib, like .jsii +files, and zip files of dependencies used in custom resources. However, the +library also contains a huge volume of source code now that aws-cdk-lib combines +231 (and increasing) CDK modules. The table below breaks down the percentage +that each category of files contributes to the size of aws-cdk-lib. + +Note: The percentages do not add up to 100. Some files, e.g. various .json +files, are excluded and contribute very little to the size. + +| Category | Percentage of aws-cdk-lib size | +| ------------------------------ | ------------------------------ | +| .jsii files | 41.64% | +| Source Maps (.js.map) | 21.93% | +| Lambda Layer zipfiles | 15.48% | +| Javascript code (.js) | 9.17% | +| Type Declaration files (.d.ts) | 9.15% | +| README, etc (.md) | 0.76% | +| bundled npm dependencies | 0.69% | +| .ts-fixture | 0.37% | +| .jsiirc | 0.36% | + + + +### Why should we _not_ do this? + +The main reason to not do this is that it will inevitably break about the +monolothic `aws-cdk-lib` package in some manner. But, we can't keep everything in one +place, and reduce the size enough to have a significant positive impact. + +### What is the technical solution (design) of this feature? + +There are different categories of files contributing to the size of aws-cdk-lib, +and each category will have it’s own solution. The following sections cover the +top five categories in size contribution in order from largest to smallest. + +#### .jsii Files + +There are two .jsii files bundled in `aws-cdk-lib` for the JSII runtime to work +for any non-NodeJS languages. And, to enable code and documentation generation +from the published npm package. + +| | Size (MB) | Percentage of aws-cdk-lib | +| --------------- | --------- | ------------------------- | +| .jsii.tabl.json | 55.1 | 22.84 % | +| .jsii | 45.35 | 18.8 % | + +We will compress the .jsii.tabl.json and .jsii files. Compressing with gzip +currently creates a 4 MB and 4.7 MB file, respectively. This change will impact +the JSII runtime, and Construct Hub. NodeJS JSII runtimes don't need to use +either file, and non-NodeJS runtimes would decompress the .jsii file when +needed. Construct Hub generates documentation from the .jsii.tabl.json file, and +it will also decompress this file as needed. + +#### Source Map Files (.js.map) + +These files currently makeup 22% of the total size of `aws-cdk-lib`. Source map +files map from the transformed source to the original source. In the case of +`aws-cdk-lib`, these are not actually useful today, since we do not ship the +original .ts files. We will remove the source maps from the released package. + +#### Lambda Layer Zip Files + +There are three zip files included in the `aws-cdk-lib package`. They are listed in the table below. + +| Filename | Includes | Usage | Size (MB) | Percentage of aws-cdk-lib | +| ------------------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------- | +| lambda-layer-kubectl/lib/layer.zip | [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), [helm](https://helm.sh/) | [eks.KubectlProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/kubectl-provider.ts#L68) | 23.18 | 9.61 % | +| lambda-layer-awscli/lib/layer.zip | [awscli](https://pypi.org/project/awscli/) | [eks.KubectlProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/kubectl-provider.ts#L68), [s3-deployment.BucketDeployment](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts#L241), [stepfunctions-tasks.EmrContainersStartJobRun](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-stepfunctions-tasks/lib/emrcontainers/start-job-run.ts#L90) | 12.78 | 5.29 | +| lambda-layer-node-proxy-agent/lib/layer.zip | [node-proxy-agent](https://www.npmjs.com/package/proxy-agent) | [eks.ClusterResourceProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/cluster-resource-provider.ts#L60) | 1.37 | 0.57 | + +Each of these zips are in a module whose only purpose is to bundle a dependency +(or two) into a Lambda Layer, which is then used in custom resources that are +part of the AWS CDK framework. Together, they contribute ~16% of the size of +aws-cdk-lib. If these zips are removed, the size of aws-cdk-lib would be ~196 +MB. + +There are more dependencies like these that need to be added to the framework, +but we are currently blocked on adding them because they would increase the size +too much. The current known list is: + +1. Additional versions of + [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). This is a + pressing problem for customers using the EKS module, who are limited to a + single version of the kubectl CLI today. +2. Additional versions of [awscli](https://pypi.org/project/awscli/). See + [issue](https://github.com/aws/aws-cdk/issues/13993). +3. [Boto3](https://pypi.org/project/boto3/), and + [aws-sdk](https://www.npmjs.com/package/aws-sdk). All custom resources in the + AWS CDK framework currently rely on whatever version of the AWS SDK that is + available in the Lambda runtime by default. To make the custom resources more + robust, and allow us to choose the exact version of the AWS SDK used by them, + we should bundle these dependencies into each custom resource’s Lambda code. + +If we add all of these with the current design, that would increase the size of `aws-cdk-lib` by ~91 MB*. + +*Rough calculation: 91 MB = 9.2 MB (aws-sdk) + <1 MB (boto3) + 3x23.18 MB (three +more versions of kubectl) + 12.78 MB (one more major version of awscli) + +##### Solution + +For the Lambda Layers, there are two different approaches we will take to remove these large assets from `aws-cdk-lib`. + +###### Separate npm packages + +Use separate npm packages for each `lambda-layer-X` module, which the CDK CLI will +automatically install if not already present in the `node_modules` directory. With +this solution, customers will not have to figure out when they do or do not need +to include these dependencies, and they will not be required to access a public +endpoint besides npm during runtime of their CDK apps. Practically, this +solution breaks down into the following steps. + +1. Publish v2 compatible versions of `@aws-cdk/lambda-layer-aws-cli`, + `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-node-proxy-agent` as + their own npm packages, separate from `aws-cdk-lib`. +2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these large + dependencies themselves, but instead reference the appropriate `Layer` class + from the packages in step 1. We will need to do something clever here to make + sure customers do not have IDE or compilation errors before the CDK CLI has + an opportunity to download and install the lambda layer packages. +3. Modify the CDK CLI to verify that the correct packages are available in the + `node_modules` directory during synthesis. If they are not available, download + them from npm. +4. We need to keep the dependencies up-to-date to get security updates and new + features. Each Lambda Layer will need to be treated slightly differently. + 1. `@aws-cdk/lambda-layer-kubectl` - This one has two dependencies to keep + updated. All updates described below will result in a minor version bump + to this package. + 1. [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) + 1. Kubernetes users will need to be able to choose the minor version + of kubectl that they want to use. And the aws-eks module will + need to as well. + 2. The public API for this package will have a `KubectlV1_20Layer` + class for each currently supported minor version of kubernetes. + 3. A weekly automated task will check the [Kubernetes API](https://dl.k8s.io/release/stable.txt) for a new patch version of + each currently supported minor version, and automatically update + the existing `Layer` class to use the new patch version. This + update will be performed with an auto-approved PR. + 4. A weekly automated task will check for new supported minor + versions of kubectl. If there is one, a PR will be automatically + created which creates a new `Layer` class in the API for the new + minor version. A human will need to review this one. + 2. [helm](https://helm.sh/) is also included in each Lambda Layer in this + package + 1. Helm has a somewhat complicated version support policy documented + [here](https://helm.sh/docs/topics/version_skew/). + 2. For each `KubectlVX_YLayer` class, we will include the latest + available version of Helm which is compatible with that version + of Kubernetes. + 3. A weekly automated task will update the minor and patch versions + of helm within the supported ranges. + 4. The automated task that checks for new supported minor versions + of kubectl, will also attempt to figure out what is the latest + version of helm compatible with that version of kubectl. This + will be verified by a human in PR review to make sure it is + correct. + 2. `@aws-cdk/lambda-layer-aws-cli` - Major versions 1 and 2 will both be + available, and automatically updated to the latest minor or patch + version. + 1. This package will have two APIs: `AwsCliLayer`, and `AwsCliV2Layer`. + 2. Automatically check for new minor and patch versions from Pypi - + awscli (https://pypi.org/project/awscli/), awscliv2 + (https://pypi.org/project/awscliv2/). Each update will result in a + minor version bump of this package. + 3. `@aws-cdk/lambda-layer-node-proxy-agent` - New minor versions will + automatically be picked up and available with a minor version bump of the + library. + 1. This can be implemented with `npm-check-updates`, since + `node-proxy-agent` is an npm package. + 4. AWS CDK v2 will automatically pick up new minor versions of the above + libraries. + +_What about customers executing their CDK apps in network-restricted environments?_ + +This question raises an important drawback of this solution. It will result in a +breaking change for customers who are using these Lambda Layers in +network-restricted environments. If customers are able to use the CDK CLI in +their network-restricted environments, then it is reasonable to assume they were +able to acquire the aws-cdk package from npm somehow. This implies customers +will be able to also acquire the lambda layer packages if we publish them +separately to npm. They can either make sure the necessary packages are already +installed and available in the `node_modules` directory where their CDK +applications are executed, or include them in a private npm registry. + +Unfortunately, there is no way to remove these large files and host them +somewhere else without causing a breaking change for these customers. We will +publish a CLI notice directly to potentially-affected customers of this change +well in advance of releasing it. And, advertise the change in advance of release +on the aws-cdk GitHub repository. + +_What about customers in ADC regions?_ + +Today, there is a mechanism for publishing AWS CDK into ADC (Amazon Dedicated +Cloud) regions. This mechanism includes bundling the direct dependencies of the +CDK and publishing them into the ADC regions, e.g. `constructs`. We will add the +lambda-layer packages to this process, and when customers use the aws-cdk npm +package in ADC regions, the right dependencies will already be present in the +`node_modules` directory. + +###### Upstream custom resources that consume the lambda layers + +The second, and more long-term part of this solution will be to upstream the +custom resources where the zip files are used. Since all of these zip files are +used in custom resources, another way to remove them from `aws-cdk-lib` is to +remove the custom resources themselves. We will create Uluru resources that are +available in the CloudFormation public registry under the namespace `AWS::CDK`. +This will increase the operational burden on our team by an unknown amount. We +should start with a single custom resource, and evaluate the maintenance cost of +this as it is implemented. In addition to reducing the size of the package, this +solution also has another benefit of addressing pain points with CDK publishing +custom resources into customer accounts. The lambda-layer APIs on their own, are +also valuable to customers. If we are able to remove their dependencies within +aws-cdk-lib by removing the custom resources, then we could vend these libraries +separately for customers who use them explicitly. The details of this solution +is out of scope for this document, and will probably need its own RFC. + +#### Javascript Files (.js) + +The Javascript files make up about 10% of the overall package size. Today, they +are already minified in the released package. There is not much else that can be +done to make them smaller. + +#### Type Declaration Files (.d.ts) + +We can minify these files, which could result in a space saving of up to 20 MB. +Minifying is the process of removing new-lines and whitespace characters from +the files. The 20 MB number was collected by removing all new-line and +whitespace characters. In reality, we will need to do something slightly more +sophisticated and keep whitespace in documentation comments, so that +type-checking in IDEs still has readable type references. + +### Is this a breaking change? + +Yes, see CHANGELOG and CLI Notice sections. + +### What alternative solutions did you consider? + +Each category of files has different alternative solutions to reduce the size. + +#### .jsii Files + +1. The .jsii.tabl.json and .jsii files are both “pretty-printed.” We can reduce + their size to 42 MB (-23%) and 30 MB (-33%) by removing all the whitespace + and newlines. We will not take this solution, because the space savings is + not as significant as compression. And, then we can maintain the benefit of + using pretty-printed JSON when uncompressed. +2. Vend the Rosetta tablet file (.jsii.tabl.json) separately. On each release, + we would publish this file as a separate artifact on GitHub releases, and + reference the URI from the jsii assembly (.jsii). This solution will not be + pursued at this time. There are several issues with de-coupling the Rosetta + tablet file from the artifacts that customers install, and with requiring the + JSII runtime to run in an environment that has access to the public endpoint + where it would be hosted. + +#### Source Map Files (.js.map) + +1. Minifying the files, instead of removing them. Since the source maps are not + useful in their current form, it is best to just remove them. In the future, + we could consider adding them back in a minified form. +2. Releasing an additional 2.X.Y-debug package with every release, that will + contain the source maps and source files. In addition to reducing the size of + the aws-cdk-lib package, we want to reduce the volume of artifacts we publish + to npm and other package managers in total. This would be in direct conflict + with that goal by doubling the amount of content that is published on each + release. And, there is currently no demonstrated customer need for a debug + package like this. + +#### Lambda Layer Zip Files + +1. Host the zip files in a static website. This would be implemented as a CDK + app in a new GitHub repository. The CDK CLI will download and cache these zip + files at synth time. The existing `lambda-layer-X` modules will be modified + to reference the large dependencies from a local cache. The CDK CLI will be + modified to update that cache at synth time, if needed. Then, the rest of + their usage will remain the same. The URLs for each dependency would follow a + predictable format, including each dependency’s version. Some automated + mechanism would be required to add new artifacts to the website, as new + versions of each dependency are released. And, to add new versions of each + dependency to aws-cdk-lib. The problem with this solution is that it requires + access to a public endpoint during synth of CDK applications. Many customers + run CDK apps in network-restricted environments (Construct Hub, ADC region + customers), and this solution does not lend itself to a straightforward + workaround for these customers. +2. Download the the zip files from a static website in the framework + implementation of each `Layer` class, instead of in the CLI. We should not do + this, because the CLI already has logic for managing credentials and + making network calls. This type of work should remain in the CLI, and not + overflow into the framework. +3. Use regionalized S3 buckets to host the zip files. With this solution, the + customer does not need to download the zip file and then re-upload it to + their deployment environment. Instead, the Lambda Layer or Lambda Function + resource definition in their CloudFormation template references the zip + hosted by us in the same region as their deployment environment. The + maintenance burden of keeping up with deploying these artifacts and keeping + them up to date in every AWS region is too high. This solution is the same as + the one described in this [comment] + (https://github.com/aws/aws-cdk-rfcs/issues/39#issuecomment-593092612). +4. Separate packages that `aws-cdk-lib` peer depends on. With this solution, + customers do not need to include the large `lambda-layer-X` packages in their + dependencies unless they are actually using one of the Constructs that + requires them. This is not a reasonable solution, since it would too + confusing for customers to know when they need these peer dependencies, and + when they do not. + +#### Javascript Files (.js) + +No alternatives were considered yet. + +#### Type Declaration Files (.d.ts) + +Remove these files. We can’t remove the type declaration files, because that +would break the amazing type-checking features of CDK! + +#### Generated L1 Construct files + +The auto-generated L1 code overlaps with the Javascript, Source Map, and Type +Declaration categories already mentioned, and it composes about 25% of the +overall package. The size of this code grows linearly as the volume of +CloudFormation resources and features grows. Alternative solutions focusing on +this category: + +1. Separate out the L1 constructs to their own library. This would compromise + one of the major values of AWS CDK v2, which is that all AWS CDK constructs + are in a single library, and customers no longer have to manage a complicated + dependency tree for their CDK apps and libraries. We might consider this in + the future as a logical way of breaking up the library into multiple + components, if we are not able to keep the size under control with other + solutions described in this document. +2. Invent a new way of generating this code that has a smaller footprint. This + could be a huge undertaking, and needs a lot more investigation into whether + there are possible solutions for this. + +### What are the drawbacks of this solution? + +1. This solution does not address the fact that the size of aws-cdk-lib +increases with each weekly release, because we are adding code and new features +at a rapid pace, and AWS is always adding new services and features. It is +possible that after all of the proposed solutions are implemented, that the +package size will still continue to increase past a reasonable size. At that +point, we might need to invent more creative solutions to reduce the size, or +implement some of the alternative solutions outlined here. + +2. This solution has a breaking change for some customers. Until we get some + feedback from the community on this RFC, we don't know for sure if the + proposed migration for those customers is reasonable. + + +### What is the high-level project plan? + +This project will have two main phases: + +#### Phase 1 — Remove large files & minify as much as possible + +Each category of files can be worked on in parallel and addressed separately. +The highest priority category to start with will be the Lambda Layer zip files. +Even though the Lambda Layers are not the biggest contributor to size today, +they are the biggest potential contributor to size. And, there is customer need +to add more Lambda Layers, which we are not able to do until the large +dependencies are not required to be bundled in aws-cdk-lib. + +The next category will be the .jsii files. After both of these solutions are +implemented, the the aws-cdk-lib unpacked npm package size will be about 100 MB. + +The lowest priority category will be removing the Source Map files and minifying +the Type Declaration files. The final size of the package after this work is +still to be determined. + +Work for phase 1 will be tracked on this project board: +https://github.com/aws/aws-cdk/projects/15 + +#### Phase 2 — Upstream custom resources as Uluru resources + +For this phase, we will start by upstreaming the `KubectlProvider` custom +resource. This is the best option to start with, because it uses the largest +Lambda-Layer dependency, kubectl. And, because there is the most customer demand +for this custom resource to be removed from CDK. This work will be quite +complex, and the detailed design is out of scope of this document. \ No newline at end of file From 73df5f4c96b3e47cd61ff29e9660ab38459bef01 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Mon, 25 Apr 2022 17:50:58 -0700 Subject: [PATCH 02/53] formatting fixes --- text/0039-reduce-module-size.md | 42 +++++++++++++++++---------------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 8885b0afc..cb10e69a7 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -43,10 +43,10 @@ using. We will define separate notices for each different scenario. Title: (aws-cdk-lib): upcoming change in packaging structure might require action to upgrade -Body: We’ve identified that you are using at least one of the Constructs. If you are running your aws-cdk commands in an +Body: We’ve identified that you are using at least one of the [List of +Constructs] Constructs. If you are running your aws-cdk commands in an environment that does not have access to npm, then you will need to make a -change to make version 2.x of available in your environment. See +change to make version 2.x of [npm packages] available in your environment. See https://github.com/aws/aws-cdk/issues/1234 for details and instructions. | List of Constructs | npm packages | @@ -103,8 +103,8 @@ to be addressed. remote: warning: File awscdk/jsii/aws-cdk-lib-2.17.0.tgz is 55.42 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB` 2. NPM itself does not have a package size limit, but other services that - host NPM packages do. For example, [Azure Artifacts] - (https://docs.microsoft.com/en-us/azure/devops/artifacts/reference/limits?view=azure-devops) + host NPM packages do. For example, [Azure + Artifacts](https://docs.microsoft.com/en-us/azure/devops/artifacts/reference/limits?view=azure-devops) has a limit of 500 MB. 3. PyPi has an unofficially documented limit of 60 MB 2. A large download size might be problematic for customers with weak internet @@ -127,20 +127,20 @@ to be addressed. Most of these above issues have workarounds. For running CDK apps in Lambda, you can use a Docker image for your Lambda code, and the limit becomes 10GB. For -package manager size limits, we can request [increases] -(https://github.com/pypa/pypi-support/issues/1642), and those are typically -approved. However, publishing a large package is a bad practice in the -open-source software ecosystem. Even if a customer has no technical limitations -for adding a ~240 MB dependency to their package, it is a red flag to them when -considering using `aws-cdk-lib`. We should be customer-obsessed, and solve the -problem of aws-cdk-lib being a huge dependency, before customers start -complaining about it. +package manager size limits, we can request +[increases](https://github.com/pypa/pypi-support/issues/1642), and those have +historically been approved. However, publishing a large package is a bad +practice in the open-source software ecosystem. Even if a customer has no +technical limitations for adding a ~240 MB dependency to their package, it is a +red flag to them when considering using `aws-cdk-lib`. We should be +customer-obsessed, and solve the problem of aws-cdk-lib being a huge dependency, +before customers start complaining about it. ### Why is the package so large? There are some large assets contributing to the size of aws-cdk-lib, like .jsii files, and zip files of dependencies used in custom resources. However, the -library also contains a huge volume of source code now that aws-cdk-lib combines +library also contains a huge volume of source code now that `aws-cdk-lib` combines 231 (and increasing) CDK modules. The table below breaks down the percentage that each category of files contributes to the size of aws-cdk-lib. @@ -163,9 +163,11 @@ files, are excluded and contribute very little to the size. ### Why should we _not_ do this? -The main reason to not do this is that it will inevitably break about the -monolothic `aws-cdk-lib` package in some manner. But, we can't keep everything in one -place, and reduce the size enough to have a significant positive impact. +The main reason to not do this is that it will inevitably break up the +monolothic `aws-cdk-lib` package in some manner, and we will lose some of the +wonderful simplicity of releasing and using a single package. But, we can't keep +everything in one place, and reduce the size enough to have a significant +positive impact. ### What is the technical solution (design) of this feature? @@ -200,7 +202,7 @@ original .ts files. We will remove the source maps from the released package. #### Lambda Layer Zip Files -There are three zip files included in the `aws-cdk-lib package`. They are listed in the table below. +There are three zip files included in the `aws-cdk-lib` package. They are listed in the table below. | Filename | Includes | Usage | Size (MB) | Percentage of aws-cdk-lib | | ------------------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------- | @@ -432,8 +434,8 @@ Each category of files has different alternative solutions to reduce the size. hosted by us in the same region as their deployment environment. The maintenance burden of keeping up with deploying these artifacts and keeping them up to date in every AWS region is too high. This solution is the same as - the one described in this [comment] - (https://github.com/aws/aws-cdk-rfcs/issues/39#issuecomment-593092612). + the one described in this + [comment](https://github.com/aws/aws-cdk-rfcs/issues/39#issuecomment-593092612). 4. Separate packages that `aws-cdk-lib` peer depends on. With this solution, customers do not need to include the large `lambda-layer-X` packages in their dependencies unless they are actually using one of the Constructs that From 76425a19dbe1eb23ef9cdc48422c8934858668dc Mon Sep 17 00:00:00 2001 From: Kaizen Conroy <36202692+kaizencc@users.noreply.github.com> Date: Tue, 26 Apr 2022 12:48:18 -0400 Subject: [PATCH 03/53] modify table in 'why is the package so large' with 'Size (MB)' column and 'Total' row --- text/0039-reduce-module-size.md | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index cb10e69a7..28d90bfd4 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -147,19 +147,18 @@ that each category of files contributes to the size of aws-cdk-lib. Note: The percentages do not add up to 100. Some files, e.g. various .json files, are excluded and contribute very little to the size. -| Category | Percentage of aws-cdk-lib size | -| ------------------------------ | ------------------------------ | -| .jsii files | 41.64% | -| Source Maps (.js.map) | 21.93% | -| Lambda Layer zipfiles | 15.48% | -| Javascript code (.js) | 9.17% | -| Type Declaration files (.d.ts) | 9.15% | -| README, etc (.md) | 0.76% | -| bundled npm dependencies | 0.69% | -| .ts-fixture | 0.37% | -| .jsiirc | 0.36% | - - +| Category | Size (MB) | Percentage of aws-cdk-lib size | +| ------------------------------ | ------------------------------ | ------------------------------ | +| .jsii files | 100.35 | 41.64% | +| Source Maps (.js.map) | 52.85 | 21.93% | +| Lambda Layer zipfiles | 37.31 | 15.48% | +| Javascript code (.js) | 22.10 | 9.17% | +| Type Declaration files (.d.ts) | 22.05 | 9.15% | +| README, etc (.md) | 1.83 | 0.76% | +| bundled npm dependencies | 1.66 | 0.69% | +| .ts-fixture | 0.89 | 0.37% | +| .jsiirc | 0.87 | 0.36% | +| Total | 239.91 | 99.55% | ### Why should we _not_ do this? @@ -515,4 +514,4 @@ For this phase, we will start by upstreaming the `KubectlProvider` custom resource. This is the best option to start with, because it uses the largest Lambda-Layer dependency, kubectl. And, because there is the most customer demand for this custom resource to be removed from CDK. This work will be quite -complex, and the detailed design is out of scope of this document. \ No newline at end of file +complex, and the detailed design is out of scope of this document. From 42ae4f613f2e3410d91ba1f396130b30cb3d6c1e Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 10:24:31 -0700 Subject: [PATCH 04/53] formatting fixes --- text/0039-reduce-module-size.md | 63 ++++++++++++++++----------------- 1 file changed, 31 insertions(+), 32 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 28d90bfd4..7bbddf525 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -15,7 +15,7 @@ reduces the size of the package to 100 MB. * **CHANGELOG**: -feat(aws-cdk-lib): Reduce aws-cdk-lib package size. +feat(aws-cdk-lib): Reduce aws-cdk-lib package size. ‼️ If you use any of the following Constructs in a network-restricted environment, you might encounter problems upgrading to this version. ‼️ @@ -32,8 +32,8 @@ environment, you might encounter problems upgrading to this version. ‼️ * aws_eks.FargateProfile * aws_eks.Cluster -Please see https://github.com/aws/aws-cdk/issues/1234 for more details, and -comment on the issue if you run into problems. +Please see [#1234](https://github.com/aws/aws-cdk/issues/1234) for more details, and +comment on the issue if you run into problems. * **CLI Notices**: @@ -47,7 +47,7 @@ Body: We’ve identified that you are using at least one of the [List of Constructs] Constructs. If you are running your aws-cdk commands in an environment that does not have access to npm, then you will need to make a change to make version 2.x of [npm packages] available in your environment. See -https://github.com/aws/aws-cdk/issues/1234 for details and instructions. +[#1234](https://github.com/aws/aws-cdk/issues/1234) for details and instructions. | List of Constructs | npm packages | | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | @@ -95,7 +95,7 @@ improvements in the download times. ### Why is a large package a problem? Why are we doing this? There are many reasons this is an issue for customers. The following items need -to be addressed. +to be addressed. 1. The package managers that aws-cdk-lib is published on have real or recommended filesize limits. For example: @@ -105,19 +105,19 @@ to be addressed. 2. NPM itself does not have a package size limit, but other services that host NPM packages do. For example, [Azure Artifacts](https://docs.microsoft.com/en-us/azure/devops/artifacts/reference/limits?view=azure-devops) - has a limit of 500 MB. + has a limit of 500 MB. 3. PyPi has an unofficially documented limit of 60 MB 2. A large download size might be problematic for customers with weak internet - connections or limited bandwidth. + connections or limited bandwidth. 3. Customers who download and install `aws-cdk-lib` on every CI/CD pipeline run may be frustrated to spend resources frequently downloading a large package. -4. AWS Lambda limits the size of your function code to 250 MB. +4. AWS Lambda limits the size of your function code to 250 MB. 5. The AWS Lambda limit for all uploaded assets for a Function is 75 GB. If a customer is using Lambda Versions, and wants to maintain a Version for every version they have published, they might end up hitting this limit because of the large size of `aws-cdk-lib`. 6. The developer experience of using `aws-cdk-lib` in an IDE might be slower in - some cases. + some cases. 7. The CDK project is currently blocked from including more ‘large dependencies’, such as multiple version of kubectl, multiple version of the AWS CLI, and dedicated versions of the AWS SDK for use in custom resources. @@ -142,10 +142,10 @@ There are some large assets contributing to the size of aws-cdk-lib, like .jsii files, and zip files of dependencies used in custom resources. However, the library also contains a huge volume of source code now that `aws-cdk-lib` combines 231 (and increasing) CDK modules. The table below breaks down the percentage -that each category of files contributes to the size of aws-cdk-lib. +that each category of files contributes to the size of aws-cdk-lib. Note: The percentages do not add up to 100. Some files, e.g. various .json -files, are excluded and contribute very little to the size. +files, are excluded and contribute very little to the size. | Category | Size (MB) | Percentage of aws-cdk-lib size | | ------------------------------ | ------------------------------ | ------------------------------ | @@ -257,10 +257,10 @@ solution breaks down into the following steps. dependencies themselves, but instead reference the appropriate `Layer` class from the packages in step 1. We will need to do something clever here to make sure customers do not have IDE or compilation errors before the CDK CLI has - an opportunity to download and install the lambda layer packages. + an opportunity to download and install the lambda layer packages. 3. Modify the CDK CLI to verify that the correct packages are available in the `node_modules` directory during synthesis. If they are not available, download - them from npm. + them from npm. 4. We need to keep the dependencies up-to-date to get security updates and new features. Each Lambda Layer will need to be treated slightly differently. 1. `@aws-cdk/lambda-layer-kubectl` - This one has two dependencies to keep @@ -275,20 +275,20 @@ solution breaks down into the following steps. 3. A weekly automated task will check the [Kubernetes API](https://dl.k8s.io/release/stable.txt) for a new patch version of each currently supported minor version, and automatically update the existing `Layer` class to use the new patch version. This - update will be performed with an auto-approved PR. + update will be performed with an auto-approved PR. 4. A weekly automated task will check for new supported minor versions of kubectl. If there is one, a PR will be automatically created which creates a new `Layer` class in the API for the new - minor version. A human will need to review this one. + minor version. A human will need to review this one. 2. [helm](https://helm.sh/) is also included in each Lambda Layer in this package 1. Helm has a somewhat complicated version support policy documented - [here](https://helm.sh/docs/topics/version_skew/). + [here](https://helm.sh/docs/topics/version_skew/). 2. For each `KubectlVX_YLayer` class, we will include the latest available version of Helm which is compatible with that version - of Kubernetes. + of Kubernetes. 3. A weekly automated task will update the minor and patch versions - of helm within the supported ranges. + of helm within the supported ranges. 4. The automated task that checks for new supported minor versions of kubectl, will also attempt to figure out what is the latest version of helm compatible with that version of kubectl. This @@ -299,9 +299,9 @@ solution breaks down into the following steps. version. 1. This package will have two APIs: `AwsCliLayer`, and `AwsCliV2Layer`. 2. Automatically check for new minor and patch versions from Pypi - - awscli (https://pypi.org/project/awscli/), awscliv2 - (https://pypi.org/project/awscliv2/). Each update will result in a - minor version bump of this package. + [awscli](https://pypi.org/project/awscli/), + [awscliv2](https://pypi.org/project/awscliv2/). Each update will + result in a minor version bump of this package. 3. `@aws-cdk/lambda-layer-node-proxy-agent` - New minor versions will automatically be picked up and available with a minor version bump of the library. @@ -320,7 +320,7 @@ able to acquire the aws-cdk package from npm somehow. This implies customers will be able to also acquire the lambda layer packages if we publish them separately to npm. They can either make sure the necessary packages are already installed and available in the `node_modules` directory where their CDK -applications are executed, or include them in a private npm registry. +applications are executed, or include them in a private npm registry. Unfortunately, there is no way to remove these large files and host them somewhere else without causing a breaking change for these customers. We will @@ -352,13 +352,13 @@ custom resources into customer accounts. The lambda-layer APIs on their own, are also valuable to customers. If we are able to remove their dependencies within aws-cdk-lib by removing the custom resources, then we could vend these libraries separately for customers who use them explicitly. The details of this solution -is out of scope for this document, and will probably need its own RFC. +is out of scope for this document, and will probably need its own RFC. #### Javascript Files (.js) The Javascript files make up about 10% of the overall package size. Today, they are already minified in the released package. There is not much else that can be -done to make them smaller. +done to make them smaller. #### Type Declaration Files (.d.ts) @@ -367,7 +367,7 @@ Minifying is the process of removing new-lines and whitespace characters from the files. The 20 MB number was collected by removing all new-line and whitespace characters. In reality, we will need to do something slightly more sophisticated and keep whitespace in documentation comments, so that -type-checking in IDEs still has readable type references. +type-checking in IDEs still has readable type references. ### Is this a breaking change? @@ -434,7 +434,7 @@ Each category of files has different alternative solutions to reduce the size. maintenance burden of keeping up with deploying these artifacts and keeping them up to date in every AWS region is too high. This solution is the same as the one described in this - [comment](https://github.com/aws/aws-cdk-rfcs/issues/39#issuecomment-593092612). + [comment](https://github.com/aws/aws-cdk-rfcs/issues/39#issuecomment-593092612). 4. Separate packages that `aws-cdk-lib` peer depends on. With this solution, customers do not need to include the large `lambda-layer-X` packages in their dependencies unless they are actually using one of the Constructs that @@ -465,7 +465,7 @@ this category: dependency tree for their CDK apps and libraries. We might consider this in the future as a logical way of breaking up the library into multiple components, if we are not able to keep the size under control with other - solutions described in this document. + solutions described in this document. 2. Invent a new way of generating this code that has a smaller footprint. This could be a huge undertaking, and needs a lot more investigation into whether there are possible solutions for this. @@ -484,10 +484,9 @@ implement some of the alternative solutions outlined here. feedback from the community on this RFC, we don't know for sure if the proposed migration for those customers is reasonable. - ### What is the high-level project plan? -This project will have two main phases: +This project will have two phases: #### Phase 1 — Remove large files & minify as much as possible @@ -504,9 +503,9 @@ implemented, the the aws-cdk-lib unpacked npm package size will be about 100 MB. The lowest priority category will be removing the Source Map files and minifying the Type Declaration files. The final size of the package after this work is still to be determined. - -Work for phase 1 will be tracked on this project board: -https://github.com/aws/aws-cdk/projects/15 + +Work for phase 1 will be tracked on [this project +board](https://github.com/aws/aws-cdk/projects/15). #### Phase 2 — Upstream custom resources as Uluru resources From b65e298b83791285019137c2e58ae9ebb17aee89 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 11:55:21 -0700 Subject: [PATCH 05/53] update sizes, add appendix for size calculation, and move discussion about future size of Lambda Layers to it's own FAQ --- text/0039-reduce-module-size.md | 138 +++++++++++++++++++++----------- 1 file changed, 90 insertions(+), 48 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 7bbddf525..780ce3d41 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -7,8 +7,8 @@ The AWS CDK v2 framework library, `aws-cdk-lib` contains all packages from AWS CDK v1 in one monolithic package. While this is a great value for customers in making dependency management much easier, it has also resulted in increasing the -unpacked npm package size to 238 MB, at the time of writing. This feature -reduces the size of the package to 100 MB. +unpacked npm package size to 234 MB, at the time of writing. This feature +reduces the size of the package to under 100 MB. ## Working Backwards @@ -74,12 +74,12 @@ RFC pull request): Today, we are launching a smaller version of the AWS CDK v2 framework library, `aws-cdk-lib`. And, a change in the AWS CDK v2 CLI to support this smaller framework library. Before today’s release, the unpacked size of this package was -238 MB. This large size prevented many customers from being able to use it, and +234 MB. This large size prevented many customers from being able to use it, and caused performance problems for others. It also prevented the CDK team from -releasing certain new features. Now, the size of the package is 100 MB. This -smaller version also has a mechanism for adding new lambda-layer dependencies. -Because of this launch, the AWS CDK framework will be able to add support for -more versions of kubectl, and awscliv2. +releasing certain new features. Now, the size of the package is less than 100 +MB. This smaller version also has a mechanism for adding new lambda-layer +dependencies. Because of this launch, the AWS CDK framework will be able to add +support for more versions of kubectl, and awscliv2. ### Why should I use this feature? @@ -106,7 +106,7 @@ to be addressed. host NPM packages do. For example, [Azure Artifacts](https://docs.microsoft.com/en-us/azure/devops/artifacts/reference/limits?view=azure-devops) has a limit of 500 MB. - 3. PyPi has an unofficially documented limit of 60 MB + 3. PyPi has an unofficially documented limit of 60 MB. 2. A large download size might be problematic for customers with weak internet connections or limited bandwidth. 3. Customers who download and install `aws-cdk-lib` on every CI/CD pipeline run @@ -131,12 +131,12 @@ package manager size limits, we can request [increases](https://github.com/pypa/pypi-support/issues/1642), and those have historically been approved. However, publishing a large package is a bad practice in the open-source software ecosystem. Even if a customer has no -technical limitations for adding a ~240 MB dependency to their package, it is a +technical limitations for adding a 234 MB dependency to their package, it is a red flag to them when considering using `aws-cdk-lib`. We should be customer-obsessed, and solve the problem of aws-cdk-lib being a huge dependency, before customers start complaining about it. -### Why is the package so large? +### Why is the package so large today? There are some large assets contributing to the size of aws-cdk-lib, like .jsii files, and zip files of dependencies used in custom resources. However, the @@ -147,18 +147,60 @@ that each category of files contributes to the size of aws-cdk-lib. Note: The percentages do not add up to 100. Some files, e.g. various .json files, are excluded and contribute very little to the size. -| Category | Size (MB) | Percentage of aws-cdk-lib size | -| ------------------------------ | ------------------------------ | ------------------------------ | -| .jsii files | 100.35 | 41.64% | -| Source Maps (.js.map) | 52.85 | 21.93% | -| Lambda Layer zipfiles | 37.31 | 15.48% | -| Javascript code (.js) | 22.10 | 9.17% | -| Type Declaration files (.d.ts) | 22.05 | 9.15% | -| README, etc (.md) | 1.83 | 0.76% | -| bundled npm dependencies | 1.66 | 0.69% | -| .ts-fixture | 0.89 | 0.37% | -| .jsiirc | 0.87 | 0.36% | -| Total | 239.91 | 99.55% | +| Category | Percentage of aws-cdk-lib size | +| ------------------------------ | ------------------------------ | +| .jsii files | 41.64% | +| Source Maps (.js.map) | 21.93% | +| Lambda Layer zip files | 15.48% | +| Javascript code (.js) | 9.17% | +| Type Declaration files (.d.ts) | 9.15% | +| README, etc (.md) | 0.76% | +| bundled npm dependencies | 0.69% | +| .ts-fixture | 0.37% | +| .jsiirc | 0.36% | + + ### Why is the package going to be large in the future? + +The CDK team is currently blocked on adding more Lambda Layer zip files to the +AWS CDK framework, because adding them would dramatically increase the size of +the package. Each of these zips are in a module whose only purpose is to bundle +a dependency (or two) into a Lambda Layer, which is then used in custom +resources that are part of the AWS CDK framework. + +There are more dependencies like these that need to be added to the framework, +but we are currently blocked on adding them because they would increase the size +too much. The current known list is: + +1. Additional versions of + [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). This is a + pressing problem for customers using the EKS module, who are limited to a + single version of the kubectl CLI today. +2. Additional versions of [awscli](https://pypi.org/project/awscli/). See + [issue](https://github.com/aws/aws-cdk/issues/13993). +3. [Boto3](https://pypi.org/project/boto3/), and + [aws-sdk](https://www.npmjs.com/package/aws-sdk). All custom resources in the + AWS CDK framework currently rely on whatever version of the AWS SDK that is + available in the Lambda runtime by default. To make the custom resources more + robust, and allow us to choose the exact version of the AWS SDK used by them, + we should bundle these dependencies into each custom resource’s Lambda code. + +If we add all of these with the current design, that would increase the size of +`aws-cdk-lib` by approximately 130 MiB*, and make the Lambda Layers the biggest +contributor of size at 39%. + +*See Appendix A to see where this estimate came from. + +| Category | Percentage of aws-cdk-lib size | +| ------------------------------ | ------------------------------ | +| Lambda Layer zip files | 38.83% | +| .jsii files | 30.14% | +| Source Maps (.js.map) | 15.87% | +| Javascript code (.js) | 6.63% | +| Type Declaration files (.d.ts) | 6.62% | +| README, etc (.md) | 0.55% | +| bundled npm dependencies | 0.50% | +| .ts-fixture | 0.27% | +| .jsiirc | 0.26% | ### Why should we _not_ do this? @@ -209,33 +251,7 @@ There are three zip files included in the `aws-cdk-lib` package. They are listed | lambda-layer-awscli/lib/layer.zip | [awscli](https://pypi.org/project/awscli/) | [eks.KubectlProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/kubectl-provider.ts#L68), [s3-deployment.BucketDeployment](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts#L241), [stepfunctions-tasks.EmrContainersStartJobRun](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-stepfunctions-tasks/lib/emrcontainers/start-job-run.ts#L90) | 12.78 | 5.29 | | lambda-layer-node-proxy-agent/lib/layer.zip | [node-proxy-agent](https://www.npmjs.com/package/proxy-agent) | [eks.ClusterResourceProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/cluster-resource-provider.ts#L60) | 1.37 | 0.57 | -Each of these zips are in a module whose only purpose is to bundle a dependency -(or two) into a Lambda Layer, which is then used in custom resources that are -part of the AWS CDK framework. Together, they contribute ~16% of the size of -aws-cdk-lib. If these zips are removed, the size of aws-cdk-lib would be ~196 -MB. -There are more dependencies like these that need to be added to the framework, -but we are currently blocked on adding them because they would increase the size -too much. The current known list is: - -1. Additional versions of - [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). This is a - pressing problem for customers using the EKS module, who are limited to a - single version of the kubectl CLI today. -2. Additional versions of [awscli](https://pypi.org/project/awscli/). See - [issue](https://github.com/aws/aws-cdk/issues/13993). -3. [Boto3](https://pypi.org/project/boto3/), and - [aws-sdk](https://www.npmjs.com/package/aws-sdk). All custom resources in the - AWS CDK framework currently rely on whatever version of the AWS SDK that is - available in the Lambda runtime by default. To make the custom resources more - robust, and allow us to choose the exact version of the AWS SDK used by them, - we should bundle these dependencies into each custom resource’s Lambda code. - -If we add all of these with the current design, that would increase the size of `aws-cdk-lib` by ~91 MB*. - -*Rough calculation: 91 MB = 9.2 MB (aws-sdk) + <1 MB (boto3) + 3x23.18 MB (three -more versions of kubectl) + 12.78 MB (one more major version of awscli) ##### Solution @@ -514,3 +530,29 @@ resource. This is the best option to start with, because it uses the largest Lambda-Layer dependency, kubectl. And, because there is the most customer demand for this custom resource to be removed from CDK. This work will be quite complex, and the detailed design is out of scope of this document. + + +## Appendix A - Notes on size calculations + +1. All size calculations in this document were done based on + `aws-cdk-lib@2.19.0`. +2. `du -k` was used to get sizes in Kb (Kebibytes), for calculating the + percentages. +3. The total size of the package according to `du` that was used to calculate + percentages is 247008 Kb. +4. The total size of 234 MB that is referenced in the doc, is the unpacked size + of version 2.19.0 according to npm. +5. The estimate for the 130 MiB increase in size for future Lambda Layer zips + was calculated from these estimates: + + | | Size (KiB) | Source | + | --------------------------- | ---------- | --------------------------------------------- | + | kubectl | 23740 | actual size of zip in released package | + | awscli | 13088 | '' | + | node-proxy-agent | 1400 | '' | + | 3 minor versions of kubectl | | = 3 * 23740 | + | boto3 | 132.5 | [pypi](https://pypi.org/project/boto3/#files) | + | awscliv2 | 13088 | estimate same size as awscli | + | --------------------------- | ---------- | --------------------------------------------- | + | Total KiB | 132545 | | + | Total MiB | 129.44 | = 132545 / 1024 | From ed0d6b5528442b5834ced223a6e2570c2dd2ae79 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 11:58:51 -0700 Subject: [PATCH 06/53] fix table showing estimates of future lambda layer sizes in Appendix A --- text/0039-reduce-module-size.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 780ce3d41..352c06599 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -550,9 +550,10 @@ complex, and the detailed design is out of scope of this document. | kubectl | 23740 | actual size of zip in released package | | awscli | 13088 | '' | | node-proxy-agent | 1400 | '' | - | 3 minor versions of kubectl | | = 3 * 23740 | - | boto3 | 132.5 | [pypi](https://pypi.org/project/boto3/#files) | + | 3 minor versions of kubectl | 71220 | = 3 * 23740 | + | boto3 | 133 | [pypi](https://pypi.org/project/boto3/#files) | | awscliv2 | 13088 | estimate same size as awscli | + | aws-sdk | 9876 | estimate same size as awscli | | --------------------------- | ---------- | --------------------------------------------- | | Total KiB | 132545 | | | Total MiB | 129.44 | = 132545 / 1024 | From f9a891dacd334bb9a4e9f81cf9143a2bbdf4c6a6 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 12:07:44 -0700 Subject: [PATCH 07/53] formatting fixes --- text/0039-reduce-module-size.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 352c06599..3f8c7464b 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -12,13 +12,13 @@ reduces the size of the package to under 100 MB. ## Working Backwards - * **CHANGELOG**: feat(aws-cdk-lib): Reduce aws-cdk-lib package size. ‼️ If you use any of the following Constructs in a network-restricted environment, you might encounter problems upgrading to this version. ‼️ + * lambda_layer_kubectl.KubectlLayer * aws_eks.KubectlProvider * aws_eks.HelmChart @@ -159,7 +159,7 @@ files, are excluded and contribute very little to the size. | .ts-fixture | 0.37% | | .jsiirc | 0.36% | - ### Why is the package going to be large in the future? +### Why is the package going to be large in the future? The CDK team is currently blocked on adding more Lambda Layer zip files to the AWS CDK framework, because adding them would dramatically increase the size of @@ -251,8 +251,6 @@ There are three zip files included in the `aws-cdk-lib` package. They are listed | lambda-layer-awscli/lib/layer.zip | [awscli](https://pypi.org/project/awscli/) | [eks.KubectlProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/kubectl-provider.ts#L68), [s3-deployment.BucketDeployment](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts#L241), [stepfunctions-tasks.EmrContainersStartJobRun](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-stepfunctions-tasks/lib/emrcontainers/start-job-run.ts#L90) | 12.78 | 5.29 | | lambda-layer-node-proxy-agent/lib/layer.zip | [node-proxy-agent](https://www.npmjs.com/package/proxy-agent) | [eks.ClusterResourceProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/cluster-resource-provider.ts#L60) | 1.37 | 0.57 | - - ##### Solution For the Lambda Layers, there are two different approaches we will take to remove these large assets from `aws-cdk-lib`. @@ -531,7 +529,6 @@ Lambda-Layer dependency, kubectl. And, because there is the most customer demand for this custom resource to be removed from CDK. This work will be quite complex, and the detailed design is out of scope of this document. - ## Appendix A - Notes on size calculations 1. All size calculations in this document were done based on From a56a58724ee1ed9253fa6d2352f847f07d90bdb7 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 12:09:19 -0700 Subject: [PATCH 08/53] fix aws-sdk row in table in Appendix A --- text/0039-reduce-module-size.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 3f8c7464b..3ec6df46f 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -542,15 +542,15 @@ complex, and the detailed design is out of scope of this document. 5. The estimate for the 130 MiB increase in size for future Lambda Layer zips was calculated from these estimates: - | | Size (KiB) | Source | - | --------------------------- | ---------- | --------------------------------------------- | - | kubectl | 23740 | actual size of zip in released package | - | awscli | 13088 | '' | - | node-proxy-agent | 1400 | '' | - | 3 minor versions of kubectl | 71220 | = 3 * 23740 | - | boto3 | 133 | [pypi](https://pypi.org/project/boto3/#files) | - | awscliv2 | 13088 | estimate same size as awscli | - | aws-sdk | 9876 | estimate same size as awscli | - | --------------------------- | ---------- | --------------------------------------------- | - | Total KiB | 132545 | | - | Total MiB | 129.44 | = 132545 / 1024 | + | | Size (KiB) | Source | + | --------------------------- | ---------- | ------------------------------------------------ | + | kubectl | 23740 | actual size of zip in released package | + | awscli | 13088 | '' | + | node-proxy-agent | 1400 | '' | + | 3 minor versions of kubectl | 71220 | = 3 * 23740 | + | boto3 | 133 | [pypi](https://pypi.org/project/boto3/#files) | + | awscliv2 | 13088 | estimate same size as awscli | + | aws-sdk | 9876 | zipped size of aws-sdk after a fresh npm install | + | --------------------------- | ---------- | --------------------------------------------- | + | Total KiB | 132545 | | + | Total MiB | 129.44 | = 132545 / 1024 | From 638d27f88dd53f65b84a6f75703c43d0293edf4d Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 12:10:55 -0700 Subject: [PATCH 09/53] Remove sizes for each individual lambda layer zip It is not as important to include this anymore now that we are focusing on the potential future size of all the lambda layer zips. The size of each individual file has no bearing on the solution. --- text/0039-reduce-module-size.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 3ec6df46f..555a35a5b 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -243,16 +243,6 @@ original .ts files. We will remove the source maps from the released package. #### Lambda Layer Zip Files -There are three zip files included in the `aws-cdk-lib` package. They are listed in the table below. - -| Filename | Includes | Usage | Size (MB) | Percentage of aws-cdk-lib | -| ------------------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------- | -| lambda-layer-kubectl/lib/layer.zip | [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), [helm](https://helm.sh/) | [eks.KubectlProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/kubectl-provider.ts#L68) | 23.18 | 9.61 % | -| lambda-layer-awscli/lib/layer.zip | [awscli](https://pypi.org/project/awscli/) | [eks.KubectlProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/kubectl-provider.ts#L68), [s3-deployment.BucketDeployment](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts#L241), [stepfunctions-tasks.EmrContainersStartJobRun](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-stepfunctions-tasks/lib/emrcontainers/start-job-run.ts#L90) | 12.78 | 5.29 | -| lambda-layer-node-proxy-agent/lib/layer.zip | [node-proxy-agent](https://www.npmjs.com/package/proxy-agent) | [eks.ClusterResourceProvider](https://github.com/aws/aws-cdk/blob/5a895a308ef2b6e66a330038c7ae35ea95a0fba4/packages/%40aws-cdk/aws-eks/lib/cluster-resource-provider.ts#L60) | 1.37 | 0.57 | - -##### Solution - For the Lambda Layers, there are two different approaches we will take to remove these large assets from `aws-cdk-lib`. ###### Separate npm packages From 06201cf988cb213ca7eb1a4823f54d039cbbf034 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 12:13:28 -0700 Subject: [PATCH 10/53] wrap lines to 80 characters: --- text/0039-reduce-module-size.md | 65 ++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 30 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 555a35a5b..3b4028ad2 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -32,8 +32,8 @@ environment, you might encounter problems upgrading to this version. ‼️ * aws_eks.FargateProfile * aws_eks.Cluster -Please see [#1234](https://github.com/aws/aws-cdk/issues/1234) for more details, and -comment on the issue if you run into problems. +Please see [#1234](https://github.com/aws/aws-cdk/issues/1234) for more details, +and comment on the issue if you run into problems. * **CLI Notices**: @@ -47,7 +47,8 @@ Body: We’ve identified that you are using at least one of the [List of Constructs] Constructs. If you are running your aws-cdk commands in an environment that does not have access to npm, then you will need to make a change to make version 2.x of [npm packages] available in your environment. See -[#1234](https://github.com/aws/aws-cdk/issues/1234) for details and instructions. +[#1234](https://github.com/aws/aws-cdk/issues/1234) for details and +instructions. | List of Constructs | npm packages | | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | @@ -140,9 +141,9 @@ before customers start complaining about it. There are some large assets contributing to the size of aws-cdk-lib, like .jsii files, and zip files of dependencies used in custom resources. However, the -library also contains a huge volume of source code now that `aws-cdk-lib` combines -231 (and increasing) CDK modules. The table below breaks down the percentage -that each category of files contributes to the size of aws-cdk-lib. +library also contains a huge volume of source code now that `aws-cdk-lib` +combines 231 (and increasing) CDK modules. The table below breaks down the +percentage that each category of files contributes to the size of aws-cdk-lib. Note: The percentages do not add up to 100. Some files, e.g. various .json files, are excluded and contribute very little to the size. @@ -243,28 +244,29 @@ original .ts files. We will remove the source maps from the released package. #### Lambda Layer Zip Files -For the Lambda Layers, there are two different approaches we will take to remove these large assets from `aws-cdk-lib`. +For the Lambda Layers, there are two different approaches we will take to remove +these large assets from `aws-cdk-lib`. ###### Separate npm packages -Use separate npm packages for each `lambda-layer-X` module, which the CDK CLI will -automatically install if not already present in the `node_modules` directory. With -this solution, customers will not have to figure out when they do or do not need -to include these dependencies, and they will not be required to access a public -endpoint besides npm during runtime of their CDK apps. Practically, this -solution breaks down into the following steps. +Use separate npm packages for each `lambda-layer-X` module, which the CDK CLI +will automatically install if not already present in the `node_modules` +directory. With this solution, customers will not have to figure out when they +do or do not need to include these dependencies, and they will not be required +to access a public endpoint besides npm during runtime of their CDK apps. +Practically, this solution breaks down into the following steps. 1. Publish v2 compatible versions of `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-node-proxy-agent` as their own npm packages, separate from `aws-cdk-lib`. -2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these large - dependencies themselves, but instead reference the appropriate `Layer` class - from the packages in step 1. We will need to do something clever here to make - sure customers do not have IDE or compilation errors before the CDK CLI has - an opportunity to download and install the lambda layer packages. +2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these + large dependencies themselves, but instead reference the appropriate `Layer` + class from the packages in step 1. We will need to do something clever here + to make sure customers do not have IDE or compilation errors before the CDK + CLI has an opportunity to download and install the lambda layer packages. 3. Modify the CDK CLI to verify that the correct packages are available in the - `node_modules` directory during synthesis. If they are not available, download - them from npm. + `node_modules` directory during synthesis. If they are not available, + download them from npm. 4. We need to keep the dependencies up-to-date to get security updates and new features. Each Lambda Layer will need to be treated slightly differently. 1. `@aws-cdk/lambda-layer-kubectl` - This one has two dependencies to keep @@ -276,16 +278,18 @@ solution breaks down into the following steps. need to as well. 2. The public API for this package will have a `KubectlV1_20Layer` class for each currently supported minor version of kubernetes. - 3. A weekly automated task will check the [Kubernetes API](https://dl.k8s.io/release/stable.txt) for a new patch version of - each currently supported minor version, and automatically update - the existing `Layer` class to use the new patch version. This - update will be performed with an auto-approved PR. + 3. A weekly automated task will check the [Kubernetes + API](https://dl.k8s.io/release/stable.txt) for a new patch + version of each currently supported minor version, and + automatically update the existing `Layer` class to use the new + patch version. This update will be performed with an + auto-approved PR. 4. A weekly automated task will check for new supported minor versions of kubectl. If there is one, a PR will be automatically created which creates a new `Layer` class in the API for the new minor version. A human will need to review this one. - 2. [helm](https://helm.sh/) is also included in each Lambda Layer in this - package + 2. [helm](https://helm.sh/) is also included in each Lambda Layer in + this package 1. Helm has a somewhat complicated version support policy documented [here](https://helm.sh/docs/topics/version_skew/). 2. For each `KubectlVX_YLayer` class, we will include the latest @@ -314,7 +318,8 @@ solution breaks down into the following steps. 4. AWS CDK v2 will automatically pick up new minor versions of the above libraries. -_What about customers executing their CDK apps in network-restricted environments?_ +_What about customers executing their CDK apps in network-restricted +environments?_ This question raises an important drawback of this solution. It will result in a breaking change for customers who are using these Lambda Layers in @@ -427,9 +432,9 @@ Each category of files has different alternative solutions to reduce the size. workaround for these customers. 2. Download the the zip files from a static website in the framework implementation of each `Layer` class, instead of in the CLI. We should not do - this, because the CLI already has logic for managing credentials and - making network calls. This type of work should remain in the CLI, and not - overflow into the framework. + this, because the CLI already has logic for managing credentials and making + network calls. This type of work should remain in the CLI, and not overflow + into the framework. 3. Use regionalized S3 buckets to host the zip files. With this solution, the customer does not need to download the zip file and then re-upload it to their deployment environment. Instead, the Lambda Layer or Lambda Function From a82e105bc9d0b02b722a3144f7f1cfbea2e26ac2 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 13:43:40 -0700 Subject: [PATCH 11/53] Cosmetic edits, move Lambda Layer solution to be first, add more detail in breaking change question --- text/0039-reduce-module-size.md | 79 ++++++++++++++++++--------------- 1 file changed, 42 insertions(+), 37 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 3b4028ad2..becedd585 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -98,7 +98,7 @@ improvements in the download times. There are many reasons this is an issue for customers. The following items need to be addressed. -1. The package managers that aws-cdk-lib is published on have real or +1. The package managers that `aws-cdk-lib` is published on have real or recommended filesize limits. For example: 1. The v2 release currently gets this warning when publishing for Go. `Go: remote: warning: File awscdk/jsii/aws-cdk-lib-2.17.0.tgz is 55.42 MB; @@ -134,16 +134,16 @@ historically been approved. However, publishing a large package is a bad practice in the open-source software ecosystem. Even if a customer has no technical limitations for adding a 234 MB dependency to their package, it is a red flag to them when considering using `aws-cdk-lib`. We should be -customer-obsessed, and solve the problem of aws-cdk-lib being a huge dependency, -before customers start complaining about it. +customer-obsessed, and solve the problem of `aws-cdk-lib` being a huge +dependency, before customers start complaining about it. ### Why is the package so large today? -There are some large assets contributing to the size of aws-cdk-lib, like .jsii -files, and zip files of dependencies used in custom resources. However, the -library also contains a huge volume of source code now that `aws-cdk-lib` -combines 231 (and increasing) CDK modules. The table below breaks down the -percentage that each category of files contributes to the size of aws-cdk-lib. +There are some large assets contributing to the size of `aws-cdk-lib`, like +.jsii files, and zip files of dependencies used in custom resources. The library +also contains a huge volume of source code now that `aws-cdk-lib` combines 231 +(and increasing) CDK modules. The table below breaks down the percentage that +each category of files contributes to the size of aws-cdk-lib. Note: The percentages do not add up to 100. Some files, e.g. various .json files, are excluded and contribute very little to the size. @@ -169,8 +169,8 @@ a dependency (or two) into a Lambda Layer, which is then used in custom resources that are part of the AWS CDK framework. There are more dependencies like these that need to be added to the framework, -but we are currently blocked on adding them because they would increase the size -too much. The current known list is: +but we are currently blocked on adding them with the current design because they +would increase the size too much. The current known list is: 1. Additional versions of [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). This is a @@ -215,32 +215,8 @@ positive impact. There are different categories of files contributing to the size of aws-cdk-lib, and each category will have it’s own solution. The following sections cover the -top five categories in size contribution in order from largest to smallest. - -#### .jsii Files - -There are two .jsii files bundled in `aws-cdk-lib` for the JSII runtime to work -for any non-NodeJS languages. And, to enable code and documentation generation -from the published npm package. - -| | Size (MB) | Percentage of aws-cdk-lib | -| --------------- | --------- | ------------------------- | -| .jsii.tabl.json | 55.1 | 22.84 % | -| .jsii | 45.35 | 18.8 % | - -We will compress the .jsii.tabl.json and .jsii files. Compressing with gzip -currently creates a 4 MB and 4.7 MB file, respectively. This change will impact -the JSII runtime, and Construct Hub. NodeJS JSII runtimes don't need to use -either file, and non-NodeJS runtimes would decompress the .jsii file when -needed. Construct Hub generates documentation from the .jsii.tabl.json file, and -it will also decompress this file as needed. - -#### Source Map Files (.js.map) - -These files currently makeup 22% of the total size of `aws-cdk-lib`. Source map -files map from the transformed source to the original source. In the case of -`aws-cdk-lib`, these are not actually useful today, since we do not ship the -original .ts files. We will remove the source maps from the released package. +top five categories in size contribution in order from largest to smallest +potential size. #### Lambda Layer Zip Files @@ -363,6 +339,31 @@ aws-cdk-lib by removing the custom resources, then we could vend these libraries separately for customers who use them explicitly. The details of this solution is out of scope for this document, and will probably need its own RFC. +#### .jsii Files + +There are two .jsii files bundled in `aws-cdk-lib` for the JSII runtime to work +for any non-NodeJS languages. And, to enable code and documentation generation +from the published npm package. + +| | Size (MB) | Percentage of aws-cdk-lib | +| --------------- | --------- | ------------------------- | +| .jsii.tabl.json | 55.1 | 22.84 % | +| .jsii | 45.35 | 18.8 % | + +We will compress the .jsii.tabl.json and .jsii files. Compressing with gzip +currently creates a 4 MB and 4.7 MB file, respectively. This change will impact +the JSII runtime, and Construct Hub. NodeJS JSII runtimes don't need to use +either file, and non-NodeJS runtimes would decompress the .jsii file when +needed. Construct Hub generates documentation from the .jsii.tabl.json file, and +it will also decompress this file as needed. + +#### Source Map Files (.js.map) + +These files currently makeup 22% of the total size of `aws-cdk-lib`. Source map +files map from the transformed source to the original source. In the case of +`aws-cdk-lib`, these are not actually useful today, since we do not ship the +original .ts files. We will remove the source maps from the released package. + #### Javascript Files (.js) The Javascript files make up about 10% of the overall package size. Today, they @@ -380,7 +381,11 @@ type-checking in IDEs still has readable type references. ### Is this a breaking change? -Yes, see CHANGELOG and CLI Notice sections. +Yes, see CHANGELOG and CLI Notice sections. The solution for the Lambda Layer +zip files will have a breaking change for customers executing CDK applications +that use the Lambda Layers in some network-restricted environments. All +alternative solutions will also have a breaking change for these customers. The +proposed solution will have the easiest migration path for these customers. ### What alternative solutions did you consider? From 1479a885f2040d989d3e90b27eefdf4ed64ff322 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 14:37:51 -0700 Subject: [PATCH 12/53] remove table with .jsii file sizes --- text/0039-reduce-module-size.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index becedd585..14128c9da 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -341,14 +341,11 @@ is out of scope for this document, and will probably need its own RFC. #### .jsii Files -There are two .jsii files bundled in `aws-cdk-lib` for the JSII runtime to work -for any non-NodeJS languages. And, to enable code and documentation generation -from the published npm package. - -| | Size (MB) | Percentage of aws-cdk-lib | -| --------------- | --------- | ------------------------- | -| .jsii.tabl.json | 55.1 | 22.84 % | -| .jsii | 45.35 | 18.8 % | +There are two files, .jsii.tabl.json and .jsii, bundled in `aws-cdk-lib` for the +JSII runtime to work for any non-NodeJS languages. And, to enable code and +documentation generation from the published npm package. Today, they makeup 42 % +of the package size. And in the future, after adding more Lambda Layers would +still compose 30%. We will compress the .jsii.tabl.json and .jsii files. Compressing with gzip currently creates a 4 MB and 4.7 MB file, respectively. This change will impact @@ -362,7 +359,7 @@ it will also decompress this file as needed. These files currently makeup 22% of the total size of `aws-cdk-lib`. Source map files map from the transformed source to the original source. In the case of `aws-cdk-lib`, these are not actually useful today, since we do not ship the -original .ts files. We will remove the source maps from the released package. +original `.ts` files. We will remove the source maps from the released package. #### Javascript Files (.js) From d7d13de27d6ac0c4706c65d08ce1d229dc919955 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 14:46:21 -0700 Subject: [PATCH 13/53] Move cloudformation public registry resources to alternate solution section --- text/0039-reduce-module-size.md | 46 ++++++++++++--------------------- 1 file changed, 16 insertions(+), 30 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 14128c9da..e9b0f0869 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -322,23 +322,6 @@ lambda-layer packages to this process, and when customers use the aws-cdk npm package in ADC regions, the right dependencies will already be present in the `node_modules` directory. -###### Upstream custom resources that consume the lambda layers - -The second, and more long-term part of this solution will be to upstream the -custom resources where the zip files are used. Since all of these zip files are -used in custom resources, another way to remove them from `aws-cdk-lib` is to -remove the custom resources themselves. We will create Uluru resources that are -available in the CloudFormation public registry under the namespace `AWS::CDK`. -This will increase the operational burden on our team by an unknown amount. We -should start with a single custom resource, and evaluate the maintenance cost of -this as it is implemented. In addition to reducing the size of the package, this -solution also has another benefit of addressing pain points with CDK publishing -custom resources into customer accounts. The lambda-layer APIs on their own, are -also valuable to customers. If we are able to remove their dependencies within -aws-cdk-lib by removing the custom resources, then we could vend these libraries -separately for customers who use them explicitly. The details of this solution -is out of scope for this document, and will probably need its own RFC. - #### .jsii Files There are two files, .jsii.tabl.json and .jsii, bundled in `aws-cdk-lib` for the @@ -452,6 +435,21 @@ Each category of files has different alternative solutions to reduce the size. requires them. This is not a reasonable solution, since it would too confusing for customers to know when they need these peer dependencies, and when they do not. +5. Upstream the custom resources where the zip files are used. Since all of + these zip files are used in custom resources, another way to remove them from + `aws-cdk-lib` is to remove the custom resources themselves. We will create + Uluru resources that are available in the CloudFormation public registry + under the namespace `AWS::CDK`. In addition to reducing the size of the + package, this solution also has another benefit of addressing pain points + with CDK publishing custom resources into customer accounts. The lambda-layer + APIs on their own are also valuable to customers. If we are able to remove + their dependencies within aws-cdk-lib by removing the custom resources, then + we could vend these libraries separately for customers who use them + explicitly. We are not going to pursue this solution right now, since this + will increase the operational burden on our team by an unknown amount. The + proposed solution does not prevent us from also pursuing this in the + future. This would remove the need for customers in network-restricted to + perform a workaround to get these dependencies. #### Javascript Files (.js) @@ -497,10 +495,6 @@ implement some of the alternative solutions outlined here. ### What is the high-level project plan? -This project will have two phases: - -#### Phase 1 — Remove large files & minify as much as possible - Each category of files can be worked on in parallel and addressed separately. The highest priority category to start with will be the Lambda Layer zip files. Even though the Lambda Layers are not the biggest contributor to size today, @@ -515,17 +509,9 @@ The lowest priority category will be removing the Source Map files and minifying the Type Declaration files. The final size of the package after this work is still to be determined. -Work for phase 1 will be tracked on [this project +Work will be tracked on [this project board](https://github.com/aws/aws-cdk/projects/15). -#### Phase 2 — Upstream custom resources as Uluru resources - -For this phase, we will start by upstreaming the `KubectlProvider` custom -resource. This is the best option to start with, because it uses the largest -Lambda-Layer dependency, kubectl. And, because there is the most customer demand -for this custom resource to be removed from CDK. This work will be quite -complex, and the detailed design is out of scope of this document. - ## Appendix A - Notes on size calculations 1. All size calculations in this document were done based on From 1b0471a53b235f7e78b5862a183e6eb492cd8430 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 14:48:06 -0700 Subject: [PATCH 14/53] formatting --- text/0039-reduce-module-size.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index e9b0f0869..47421b656 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -482,12 +482,12 @@ this category: ### What are the drawbacks of this solution? 1. This solution does not address the fact that the size of aws-cdk-lib -increases with each weekly release, because we are adding code and new features -at a rapid pace, and AWS is always adding new services and features. It is -possible that after all of the proposed solutions are implemented, that the -package size will still continue to increase past a reasonable size. At that -point, we might need to invent more creative solutions to reduce the size, or -implement some of the alternative solutions outlined here. + increases with each weekly release, because we are adding code and new + features at a rapid pace, and AWS is always adding new services and features. + It is possible that after all of the proposed solutions are implemented, that + the package size will still continue to increase past a reasonable size. At + that point, we might need to invent more creative solutions to reduce the + size, or implement some of the alternative solutions outlined here. 2. This solution has a breaking change for some customers. Until we get some feedback from the community on this RFC, we don't know for sure if the From 7e1c95200038339ab14139189a6b871310e6c11d Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 14:50:16 -0700 Subject: [PATCH 15/53] Fix lambda layer section headings --- text/0039-reduce-module-size.md | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 47421b656..9c5217055 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -220,11 +220,6 @@ potential size. #### Lambda Layer Zip Files -For the Lambda Layers, there are two different approaches we will take to remove -these large assets from `aws-cdk-lib`. - -###### Separate npm packages - Use separate npm packages for each `lambda-layer-X` module, which the CDK CLI will automatically install if not already present in the `node_modules` directory. With this solution, customers will not have to figure out when they @@ -294,8 +289,8 @@ Practically, this solution breaks down into the following steps. 4. AWS CDK v2 will automatically pick up new minor versions of the above libraries. -_What about customers executing their CDK apps in network-restricted -environments?_ +##### What about customers executing their CDK apps in network-restricted +environments? This question raises an important drawback of this solution. It will result in a breaking change for customers who are using these Lambda Layers in @@ -313,7 +308,7 @@ publish a CLI notice directly to potentially-affected customers of this change well in advance of releasing it. And, advertise the change in advance of release on the aws-cdk GitHub repository. -_What about customers in ADC regions?_ +##### What about customers in ADC regions? Today, there is a mechanism for publishing AWS CDK into ADC (Amazon Dedicated Cloud) regions. This mechanism includes bundling the direct dependencies of the From a92092b47b6640bb5691485e283f41908ef7d4ab Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 15:00:10 -0700 Subject: [PATCH 16/53] formatting edits --- text/0039-reduce-module-size.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 9c5217055..a06a5c813 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -289,18 +289,17 @@ Practically, this solution breaks down into the following steps. 4. AWS CDK v2 will automatically pick up new minor versions of the above libraries. -##### What about customers executing their CDK apps in network-restricted -environments? +##### What about customers executing their CDK apps in network-restricted environments? This question raises an important drawback of this solution. It will result in a breaking change for customers who are using these Lambda Layers in network-restricted environments. If customers are able to use the CDK CLI in their network-restricted environments, then it is reasonable to assume they were -able to acquire the aws-cdk package from npm somehow. This implies customers -will be able to also acquire the lambda layer packages if we publish them -separately to npm. They can either make sure the necessary packages are already -installed and available in the `node_modules` directory where their CDK -applications are executed, or include them in a private npm registry. +able to acquire the `aws-cdk` package from npm somehow. This implies customers +will be able to also acquire the lambda layer packages from npm. They can either +make sure the necessary packages are already installed and available in the +`node_modules` directory where their CDK applications are executed, or include +them in a private npm registry. Unfortunately, there is no way to remove these large files and host them somewhere else without causing a breaking change for these customers. We will @@ -321,9 +320,9 @@ package in ADC regions, the right dependencies will already be present in the There are two files, .jsii.tabl.json and .jsii, bundled in `aws-cdk-lib` for the JSII runtime to work for any non-NodeJS languages. And, to enable code and -documentation generation from the published npm package. Today, they makeup 42 % +documentation generation from the published npm package. Today, they makeup 42% of the package size. And in the future, after adding more Lambda Layers would -still compose 30%. +compose 30%. We will compress the .jsii.tabl.json and .jsii files. Compressing with gzip currently creates a 4 MB and 4.7 MB file, respectively. This change will impact @@ -358,7 +357,7 @@ type-checking in IDEs still has readable type references. Yes, see CHANGELOG and CLI Notice sections. The solution for the Lambda Layer zip files will have a breaking change for customers executing CDK applications -that use the Lambda Layers in some network-restricted environments. All +that use the Lambda Layers in some network-restricted environments. Most alternative solutions will also have a breaking change for these customers. The proposed solution will have the easiest migration path for these customers. @@ -438,7 +437,7 @@ Each category of files has different alternative solutions to reduce the size. package, this solution also has another benefit of addressing pain points with CDK publishing custom resources into customer accounts. The lambda-layer APIs on their own are also valuable to customers. If we are able to remove - their dependencies within aws-cdk-lib by removing the custom resources, then + their dependencies within `aws-cdk-lib` by removing the custom resources, then we could vend these libraries separately for customers who use them explicitly. We are not going to pursue this solution right now, since this will increase the operational burden on our team by an unknown amount. The From 2ba049cb429327049692bb2cedda11def56ce341 Mon Sep 17 00:00:00 2001 From: Kaizen Conroy <36202692+kaizencc@users.noreply.github.com> Date: Tue, 26 Apr 2022 18:29:19 -0400 Subject: [PATCH 17/53] Update text/0039-reduce-module-size.md --- text/0039-reduce-module-size.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index a06a5c813..dc3c15d2c 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -320,7 +320,7 @@ package in ADC regions, the right dependencies will already be present in the There are two files, .jsii.tabl.json and .jsii, bundled in `aws-cdk-lib` for the JSII runtime to work for any non-NodeJS languages. And, to enable code and -documentation generation from the published npm package. Today, they makeup 42% +documentation generation from the published npm package. Today, they make up 42% of the package size. And in the future, after adding more Lambda Layers would compose 30%. From 7687faa92504c8b25fa4f87d805108b77e2e26da Mon Sep 17 00:00:00 2001 From: Kaizen Conroy <36202692+kaizencc@users.noreply.github.com> Date: Tue, 26 Apr 2022 19:37:15 -0400 Subject: [PATCH 18/53] Update text/0039-reduce-module-size.md --- text/0039-reduce-module-size.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index dc3c15d2c..1eb1429db 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -321,8 +321,8 @@ package in ADC regions, the right dependencies will already be present in the There are two files, .jsii.tabl.json and .jsii, bundled in `aws-cdk-lib` for the JSII runtime to work for any non-NodeJS languages. And, to enable code and documentation generation from the published npm package. Today, they make up 42% -of the package size. And in the future, after adding more Lambda Layers would -compose 30%. +of the package size. And in the future, after adding more Lambda Layers, the .jsii files would +compose 30% of the package size. We will compress the .jsii.tabl.json and .jsii files. Compressing with gzip currently creates a 4 MB and 4.7 MB file, respectively. This change will impact From 3cfab40aa6d53e3a0473c1e6b697f1aa45907c6a Mon Sep 17 00:00:00 2001 From: Kaizen Conroy <36202692+kaizencc@users.noreply.github.com> Date: Tue, 26 Apr 2022 19:37:20 -0400 Subject: [PATCH 19/53] Update text/0039-reduce-module-size.md --- text/0039-reduce-module-size.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 1eb1429db..1c70864e7 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -160,7 +160,7 @@ files, are excluded and contribute very little to the size. | .ts-fixture | 0.37% | | .jsiirc | 0.36% | -### Why is the package going to be large in the future? +### Why is the package going to be larger in the future? The CDK team is currently blocked on adding more Lambda Layer zip files to the AWS CDK framework, because adding them would dramatically increase the size of From 1bd16bc02a7913217e23b8de7203c39a9a0e60a9 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 18:16:12 -0700 Subject: [PATCH 20/53] fix --- text/0039-reduce-module-size.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 1c70864e7..d6b5372f9 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -432,7 +432,7 @@ Each category of files has different alternative solutions to reduce the size. 5. Upstream the custom resources where the zip files are used. Since all of these zip files are used in custom resources, another way to remove them from `aws-cdk-lib` is to remove the custom resources themselves. We will create - Uluru resources that are available in the CloudFormation public registry + AWS resources that are available in the CloudFormation public registry under the namespace `AWS::CDK`. In addition to reducing the size of the package, this solution also has another benefit of addressing pain points with CDK publishing custom resources into customer accounts. The lambda-layer From 9d5cf7edc5b1f877519e0ea053c3ca39fa9a12ff Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Tue, 26 Apr 2022 18:20:43 -0700 Subject: [PATCH 21/53] make 'lambda' references consistent --- text/0039-reduce-module-size.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index d6b5372f9..5de48546f 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -78,15 +78,15 @@ framework library. Before today’s release, the unpacked size of this package w 234 MB. This large size prevented many customers from being able to use it, and caused performance problems for others. It also prevented the CDK team from releasing certain new features. Now, the size of the package is less than 100 -MB. This smaller version also has a mechanism for adding new lambda-layer +MB. This smaller version also has a mechanism for adding new Lambda Layer dependencies. Because of this launch, the AWS CDK framework will be able to add support for more versions of kubectl, and awscliv2. ### Why should I use this feature? You should upgrade your dependencies on `aws-cdk-lib` to get the benefits of a -smaller package size. If you use `aws-cdk-lib` in a lambda function, now you can -include more code and dependencies before hitting the lambda function code size +smaller package size. If you use `aws-cdk-lib` in a Lambda function, now you can +include more code and dependencies before hitting the Lambda function code size limit of 250 MB. If you use `aws-cdk-lib` in your CI/CD pipelines and are frequently downloading it with a fresh npm install, you will see performance improvements in the download times. @@ -234,7 +234,7 @@ Practically, this solution breaks down into the following steps. large dependencies themselves, but instead reference the appropriate `Layer` class from the packages in step 1. We will need to do something clever here to make sure customers do not have IDE or compilation errors before the CDK - CLI has an opportunity to download and install the lambda layer packages. + CLI has an opportunity to download and install the packages. 3. Modify the CDK CLI to verify that the correct packages are available in the `node_modules` directory during synthesis. If they are not available, download them from npm. @@ -296,7 +296,7 @@ breaking change for customers who are using these Lambda Layers in network-restricted environments. If customers are able to use the CDK CLI in their network-restricted environments, then it is reasonable to assume they were able to acquire the `aws-cdk` package from npm somehow. This implies customers -will be able to also acquire the lambda layer packages from npm. They can either +will be able to also acquire the `@aws-cdk/lambda-layer-X` packages from npm. They can either make sure the necessary packages are already installed and available in the `node_modules` directory where their CDK applications are executed, or include them in a private npm registry. @@ -312,7 +312,7 @@ on the aws-cdk GitHub repository. Today, there is a mechanism for publishing AWS CDK into ADC (Amazon Dedicated Cloud) regions. This mechanism includes bundling the direct dependencies of the CDK and publishing them into the ADC regions, e.g. `constructs`. We will add the -lambda-layer packages to this process, and when customers use the aws-cdk npm +`@aws-cdk/lambda-layer-X` packages to this process, and when customers use the aws-cdk npm package in ADC regions, the right dependencies will already be present in the `node_modules` directory. @@ -435,7 +435,7 @@ Each category of files has different alternative solutions to reduce the size. AWS resources that are available in the CloudFormation public registry under the namespace `AWS::CDK`. In addition to reducing the size of the package, this solution also has another benefit of addressing pain points - with CDK publishing custom resources into customer accounts. The lambda-layer + with CDK publishing custom resources into customer accounts. The `XXLayer` APIs on their own are also valuable to customers. If we are able to remove their dependencies within `aws-cdk-lib` by removing the custom resources, then we could vend these libraries separately for customers who use them From 39ba3e9491fdeb7841592c6bd65b53af54833fbf Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Thu, 28 Apr 2022 23:22:32 -0700 Subject: [PATCH 22/53] grammar, spelling --- text/0039-reduce-module-size.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 5de48546f..ab1814f8f 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -120,7 +120,7 @@ to be addressed. 6. The developer experience of using `aws-cdk-lib` in an IDE might be slower in some cases. 7. The CDK project is currently blocked from including more ‘large - dependencies’, such as multiple version of kubectl, multiple version of the + dependencies’, such as multiple versions of kubectl, multiple versions of the AWS CLI, and dedicated versions of the AWS SDK for use in custom resources. This is a decision we’ve made to avoid increasing the size of the package further until a more scalable solution for these kinds of dependencies is @@ -426,7 +426,7 @@ Each category of files has different alternative solutions to reduce the size. 4. Separate packages that `aws-cdk-lib` peer depends on. With this solution, customers do not need to include the large `lambda-layer-X` packages in their dependencies unless they are actually using one of the Constructs that - requires them. This is not a reasonable solution, since it would too + requires them. This is not a reasonable solution, since it would be too confusing for customers to know when they need these peer dependencies, and when they do not. 5. Upstream the custom resources where the zip files are used. Since all of From ffa008e89426c57d2aee22f81b8d2b1a29da715d Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Thu, 28 Apr 2022 23:52:41 -0700 Subject: [PATCH 23/53] Add detail on adding new lambda layers --- text/0039-reduce-module-size.md | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index ab1814f8f..41e2dec7d 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -229,12 +229,16 @@ Practically, this solution breaks down into the following steps. 1. Publish v2 compatible versions of `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-node-proxy-agent` as - their own npm packages, separate from `aws-cdk-lib`. + their own npm packages, separate from `aws-cdk-lib`. This publishing process + will be an extension of the existing AWS CDK V2 release machinery. Any + submodules with the prefix `lambda-layer` will be published the same way. 2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these large dependencies themselves, but instead reference the appropriate `Layer` - class from the packages in step 1. We will need to do something clever here - to make sure customers do not have IDE or compilation errors before the CDK - CLI has an opportunity to download and install the packages. + class from the packages in step 1. Since `aws-cdk-lib`'s source code is + auto-generated from the v1 submodules, this could be an autogenerated + wrapper. We will need to do something clever here to make sure customers do + not have IDE or compilation errors before the CDK CLI has an opportunity to + download and install the packages. 3. Modify the CDK CLI to verify that the correct packages are available in the `node_modules` directory during synthesis. If they are not available, download them from npm. @@ -289,6 +293,19 @@ Practically, this solution breaks down into the following steps. 4. AWS CDK v2 will automatically pick up new minor versions of the above libraries. +After the above is implemented, this mechanism can be extended to include +additional Lambda Layers with different dependencies. The solution should be +implemented in such a way that new dependencies can be added with the below +steps. +1. Add a new directory in the `aws-cdk` monorepo: `@aws-cdk/lambda-layer-MyNewDependency`. +2. Implement the MyNewDependencyLayer API. +3. If the dependency is from npm, then updates will happen automatically. If it + is from some other source, then extend the automation + from the above steps to also update the new dependency. + +Once this solution is implemented, it will be possible to write more concrete +steps. And those will be included in the Contributing Guide. + ##### What about customers executing their CDK apps in network-restricted environments? This question raises an important drawback of this solution. It will result in a From 2b207ec3c1bddacd3fc9937cba66ab206e907ee2 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Thu, 28 Apr 2022 23:55:57 -0700 Subject: [PATCH 24/53] add Otavio as the bar raiser --- text/0039-reduce-module-size.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 41e2dec7d..3103d5fff 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -1,8 +1,8 @@ # Reduce aws-cdk-lib package size -* **Original Author(s):**: @madeline-k +* **Original Author(s):**: @{madeline-k} * **Tracking Issue**: #39 -* **API Bar Raiser**: tbd +* **API Bar Raiser**: @{otaviomacedo} The AWS CDK v2 framework library, `aws-cdk-lib` contains all packages from AWS CDK v1 in one monolithic package. While this is a great value for customers in From 70e32ceba62cd2e52d6d7dc46adcab6d85796b73 Mon Sep 17 00:00:00 2001 From: Madeline Kusters <80541297+madeline-k@users.noreply.github.com> Date: Fri, 29 Apr 2022 19:25:05 -0700 Subject: [PATCH 25/53] Update 0039-reduce-module-size.md --- text/0039-reduce-module-size.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 3103d5fff..4e3ea5ce3 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -1,8 +1,8 @@ # Reduce aws-cdk-lib package size -* **Original Author(s):**: @{madeline-k} +* **Original Author(s):**: [@madeline-k](https://github.com/madeline-k) * **Tracking Issue**: #39 -* **API Bar Raiser**: @{otaviomacedo} +* **API Bar Raiser**: [@otaviomacedo](https://github.com/otaviomacedo) The AWS CDK v2 framework library, `aws-cdk-lib` contains all packages from AWS CDK v1 in one monolithic package. While this is a great value for customers in From f90055c5f9ccfb8e985b4a4182c2d1b54c939f47 Mon Sep 17 00:00:00 2001 From: Madeline Kusters <80541297+madeline-k@users.noreply.github.com> Date: Wed, 4 May 2022 16:28:08 -0700 Subject: [PATCH 26/53] Update text/0039-reduce-module-size.md Spelling Co-authored-by: Rico Huijbers --- text/0039-reduce-module-size.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 4e3ea5ce3..7111b5c69 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -527,7 +527,7 @@ board](https://github.com/aws/aws-cdk/projects/15). 1. All size calculations in this document were done based on `aws-cdk-lib@2.19.0`. -2. `du -k` was used to get sizes in Kb (Kebibytes), for calculating the +2. `du -k` was used to get sizes in Kb (Kibibytes), for calculating the percentages. 3. The total size of the package according to `du` that was used to calculate percentages is 247008 Kb. From aee3d7dfdbabd4e421ab7b5aff4bad7b466c2974 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 16:35:14 -0700 Subject: [PATCH 27/53] Remove mention of potential compilation errors for users. This should not be an issue --- text/0039-reduce-module-size.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 7111b5c69..09f0c4a71 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -234,11 +234,9 @@ Practically, this solution breaks down into the following steps. submodules with the prefix `lambda-layer` will be published the same way. 2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these large dependencies themselves, but instead reference the appropriate `Layer` - class from the packages in step 1. Since `aws-cdk-lib`'s source code is - auto-generated from the v1 submodules, this could be an autogenerated - wrapper. We will need to do something clever here to make sure customers do - not have IDE or compilation errors before the CDK CLI has an opportunity to - download and install the packages. + cosntruct from the packages in step 1. Since `aws-cdk-lib`'s source code is + auto-generated from the v1 submodules, this should be an autogenerated + wrapper. 3. Modify the CDK CLI to verify that the correct packages are available in the `node_modules` directory during synthesis. If they are not available, download them from npm. From b4586f5d05b67306cb3cbc58f44ff35ee68e531f Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 16:37:39 -0700 Subject: [PATCH 28/53] Remove mention of 'under 100 MB', it's not helpful --- text/0039-reduce-module-size.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 09f0c4a71..7b0883db9 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -4,11 +4,11 @@ * **Tracking Issue**: #39 * **API Bar Raiser**: [@otaviomacedo](https://github.com/otaviomacedo) -The AWS CDK v2 framework library, `aws-cdk-lib` contains all packages from AWS -CDK v1 in one monolithic package. While this is a great value for customers in -making dependency management much easier, it has also resulted in increasing the -unpacked npm package size to 234 MB, at the time of writing. This feature -reduces the size of the package to under 100 MB. +The AWS CDK v2 framework library, `aws-cdk-lib` contains all framework packages +from AWS CDK v1 in one monolithic package. While this is a great value for +customers in making dependency management much easier, it has also resulted in +increasing the unpacked npm package size to 234 MB, at the time of writing. The +goal of this project is to significantly reduce this size. ## Working Backwards From 92b0125e29539dd34a8ec86572b18bc735261ea0 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 16:46:31 -0700 Subject: [PATCH 29/53] rephrase alternate lambda layer solution 1, and update description for releasing Lamba Layer packages separately --- text/0039-reduce-module-size.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 7b0883db9..49b16c9e6 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -230,10 +230,12 @@ Practically, this solution breaks down into the following steps. 1. Publish v2 compatible versions of `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-node-proxy-agent` as their own npm packages, separate from `aws-cdk-lib`. This publishing process - will be an extension of the existing AWS CDK V2 release machinery. Any - submodules with the prefix `lambda-layer` will be published the same way. + will be an extension of the existing AWS CDK V2 release machinery, which + automatically publishes some packages as part of `aws-cdk-lib`, and some + packages separately. Any submodules with the prefix `lambda-layer` will be + published this way. 2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these - large dependencies themselves, but instead reference the appropriate `Layer` + large dependencies themselves, and instead reference the appropriate `Layer` cosntruct from the packages in step 1. Since `aws-cdk-lib`'s source code is auto-generated from the v1 submodules, this should be an autogenerated wrapper. @@ -417,13 +419,12 @@ Each category of files has different alternative solutions to reduce the size. modified to update that cache at synth time, if needed. Then, the rest of their usage will remain the same. The URLs for each dependency would follow a predictable format, including each dependency’s version. Some automated - mechanism would be required to add new artifacts to the website, as new + mechanism would be required to add new artifacts to the website as new versions of each dependency are released. And, to add new versions of each dependency to aws-cdk-lib. The problem with this solution is that it requires - access to a public endpoint during synth of CDK applications. Many customers - run CDK apps in network-restricted environments (Construct Hub, ADC region - customers), and this solution does not lend itself to a straightforward - workaround for these customers. + access to a public endpoint during synthesis of CDK applications and does not + lend itself to a straightforward workaround for customers who run CDK apps in + network-restricted environments (Construct Hub, ADC region customers). 2. Download the the zip files from a static website in the framework implementation of each `Layer` class, instead of in the CLI. We should not do this, because the CLI already has logic for managing credentials and making From 534e3ddabc334293f78394f846d60c63397cdc80 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 17:13:19 -0700 Subject: [PATCH 30/53] describe dev ex with the lambda-layer packages. --- text/0039-reduce-module-size.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 49b16c9e6..b6c5693ab 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -233,12 +233,18 @@ Practically, this solution breaks down into the following steps. will be an extension of the existing AWS CDK V2 release machinery, which automatically publishes some packages as part of `aws-cdk-lib`, and some packages separately. Any submodules with the prefix `lambda-layer` will be - published this way. + published this way. Even though these packages will be released separately, + their source code will remain in the aws-cdk repo. 2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these large dependencies themselves, and instead reference the appropriate `Layer` - cosntruct from the packages in step 1. Since `aws-cdk-lib`'s source code is + construct from the packages in step 1. Since `aws-cdk-lib`'s source code is auto-generated from the v1 submodules, this should be an autogenerated - wrapper. + wrapper. The first two steps should not impact the current developer workflow + of working with these `Layer` constructs. Developers will work with each + @aws-cdk module in its corresponding source directory. The publishing + pipeline will publish the `@aws-cdk/lambda-layer-X` packages, and the + `aws-cdk-lib` code generation process will create the wrappers that reference + them. 3. Modify the CDK CLI to verify that the correct packages are available in the `node_modules` directory during synthesis. If they are not available, download them from npm. From c43213218bec5f42cad5cb68bad5a6c89eb82821 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 17:14:05 -0700 Subject: [PATCH 31/53] fix placeholder issue number --- text/0039-reduce-module-size.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index b6c5693ab..db754e51c 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -32,8 +32,8 @@ environment, you might encounter problems upgrading to this version. ‼️ * aws_eks.FargateProfile * aws_eks.Cluster -Please see [#1234](https://github.com/aws/aws-cdk/issues/1234) for more details, -and comment on the issue if you run into problems. +Please see [#1234567](https://github.com/aws/aws-cdk/issues/1234567) for more +details, and comment on the issue if you run into problems. * **CLI Notices**: @@ -47,7 +47,7 @@ Body: We’ve identified that you are using at least one of the [List of Constructs] Constructs. If you are running your aws-cdk commands in an environment that does not have access to npm, then you will need to make a change to make version 2.x of [npm packages] available in your environment. See -[#1234](https://github.com/aws/aws-cdk/issues/1234) for details and +[#1234567](https://github.com/aws/aws-cdk/issues/1234567) for details and instructions. | List of Constructs | npm packages | From faf8df264adf581bf3a839d0a14c291275c4f384 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 17:32:43 -0700 Subject: [PATCH 32/53] add another workaround for Node users in npm-restricted environments, fix markdown table --- text/0039-reduce-module-size.md | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index db754e51c..34dc8a188 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -50,13 +50,13 @@ change to make version 2.x of [npm packages] available in your environment. See [#1234567](https://github.com/aws/aws-cdk/issues/1234567) for details and instructions. -| List of Constructs | npm packages | -| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | -| KubectlLayer | @aws-cdk/lambda-layer-kubectl | -| KubectlProvider, HelmChart, KubernetesManifest, KubernetesObjectValue, or KubernetesPatch | @aws-cdk/lambda-layer-kubectl and @aws-cdk/lambda-layer-awscli | -| AwsCliLayer, BucketDeployment, or EmrContainersStartJobRun | @aws-cdk/lambda-layer-awscli | -| NodeProxyAgentLayer, or FargateProfile | @aws-cdk/lambda-layer-node-proxy-agent | -| aws_eks.Cluster | @aws-cdk/lambda-layer-kubectl, @aws-cdk/lambda-layer-awscli, and @aws-cdk/lambda-layer-node-proxy-agent | +| List of Constructs | npm packages | +| -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| lambda-layer-kubectl.KubectlLayer | @aws-cdk/lambda-layer-kubectl | +| aws-eks.KubectlProvider
aws-eks.HelmChart
aws-eks.KubernetesManifest
aws-eks.KubernetesObjectValue
aws-eks.KubernetesPatch | @aws-cdk/lambda-layer-kubectl and @aws-cdk/lambda-layer-awscli | +| lambda-layer-awscli.AwsCliLayer
aws-s3-deployment.BucketDeployment
aws-step-functions-tasks.EmrContainersStartJobRun | @aws-cdk/lambda-layer-awscli | +| lamdba-layer-node-proxy-agent.NodeProxyAgentLayer
aws-eks.FargateProfile | @aws-cdk/lambda-layer-node-proxy-agent | +| aws_eks.Cluster | @aws-cdk/lambda-layer-kubectl, @aws-cdk/lambda-layer-awscli, and @aws-cdk/lambda-layer-node-proxy-agent | --- @@ -319,10 +319,13 @@ breaking change for customers who are using these Lambda Layers in network-restricted environments. If customers are able to use the CDK CLI in their network-restricted environments, then it is reasonable to assume they were able to acquire the `aws-cdk` package from npm somehow. This implies customers -will be able to also acquire the `@aws-cdk/lambda-layer-X` packages from npm. They can either -make sure the necessary packages are already installed and available in the -`node_modules` directory where their CDK applications are executed, or include -them in a private npm registry. +will be able to acquire the `@aws-cdk/lambda-layer-X` packages from npm. There +are a few options for these customers: +1. If using TypeScript or JavaScript, add each necessary package the + dependencies of their CDK app. +2. Make sure the necessary packages are already installed and available in the + `node_modules` directory where their CDK CLI is installed. +3. Include them in a private npm registry. The Unfortunately, there is no way to remove these large files and host them somewhere else without causing a breaking change for these customers. We will From a48c8fa18dd4b04156729daff2202ae2e4dd6b18 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 17:35:56 -0700 Subject: [PATCH 33/53] use underscores because that's what we do for aws-cdk-lib submodules --- text/0039-reduce-module-size.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 34dc8a188..f4f333ca8 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -52,10 +52,10 @@ instructions. | List of Constructs | npm packages | | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | -| lambda-layer-kubectl.KubectlLayer | @aws-cdk/lambda-layer-kubectl | -| aws-eks.KubectlProvider
aws-eks.HelmChart
aws-eks.KubernetesManifest
aws-eks.KubernetesObjectValue
aws-eks.KubernetesPatch | @aws-cdk/lambda-layer-kubectl and @aws-cdk/lambda-layer-awscli | -| lambda-layer-awscli.AwsCliLayer
aws-s3-deployment.BucketDeployment
aws-step-functions-tasks.EmrContainersStartJobRun | @aws-cdk/lambda-layer-awscli | -| lamdba-layer-node-proxy-agent.NodeProxyAgentLayer
aws-eks.FargateProfile | @aws-cdk/lambda-layer-node-proxy-agent | +| lambda_layer_kubectl.KubectlLayer | @aws-cdk/lambda_layer_kubectl | +| aws_eks.KubectlProvider
aws_eks.HelmChart
aws_eks.KubernetesManifest
aws_eks.KubernetesObjectValue
aws_eks.KubernetesPatch | @aws-cdk/lambda-layer-kubectl and @aws-cdk/lambda-layer-awscli | +| lambda_layer_awscli.AwsCliLayer
aws_s3_deployment.BucketDeployment
aws_step_functions_tasks.EmrContainersStartJobRun | @aws-cdk/lambda-layer-awscli | +| lamdba_layer_node_proxy_agent.NodeProxyAgentLayer
aws-eks.FargateProfile | @aws-cdk/lambda-layer-node-proxy-agent | | aws_eks.Cluster | @aws-cdk/lambda-layer-kubectl, @aws-cdk/lambda-layer-awscli, and @aws-cdk/lambda-layer-node-proxy-agent | --- From e1b36c4359a2e467a4686548434690f5c8eb9974 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 17:38:12 -0700 Subject: [PATCH 34/53] add Otavio approval --- text/0039-reduce-module-size.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index f4f333ca8..bd0e94bc6 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -64,9 +64,7 @@ Ticking the box below indicates that the public API of this RFC has been signed-off by the API bar raiser (the `api-approved` label was applied to the RFC pull request): -``` -[ ] Signed-off by API Bar Raiser @xxxxx -``` +- [x] Signed-off by API Bar Raiser [@otaviomacedo](https://github.com/otaviomacedo) ## Public FAQ From 8212ced7ff8b192e3fc74d3b99239a8dee800ab1 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 17:40:11 -0700 Subject: [PATCH 35/53] remove IDE slowness as a potential customer pain point --- text/0039-reduce-module-size.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index bd0e94bc6..4ea55c3f7 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -115,9 +115,7 @@ to be addressed. customer is using Lambda Versions, and wants to maintain a Version for every version they have published, they might end up hitting this limit because of the large size of `aws-cdk-lib`. -6. The developer experience of using `aws-cdk-lib` in an IDE might be slower in - some cases. -7. The CDK project is currently blocked from including more ‘large +6. The CDK project is currently blocked from including more ‘large dependencies’, such as multiple versions of kubectl, multiple versions of the AWS CLI, and dedicated versions of the AWS SDK for use in custom resources. This is a decision we’ve made to avoid increasing the size of the package From dd3eb7baf185719e1a2337957c378bd89089b5b6 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 17:44:47 -0700 Subject: [PATCH 36/53] update title of future sizes table --- text/0039-reduce-module-size.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 4ea55c3f7..674b52bbe 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -187,17 +187,17 @@ contributor of size at 39%. *See Appendix A to see where this estimate came from. -| Category | Percentage of aws-cdk-lib size | -| ------------------------------ | ------------------------------ | -| Lambda Layer zip files | 38.83% | -| .jsii files | 30.14% | -| Source Maps (.js.map) | 15.87% | -| Javascript code (.js) | 6.63% | -| Type Declaration files (.d.ts) | 6.62% | -| README, etc (.md) | 0.55% | -| bundled npm dependencies | 0.50% | -| .ts-fixture | 0.27% | -| .jsiirc | 0.26% | +| Category | Estimated future percentage of aws-cdk-lib size | +| ------------------------------ | ----------------------------------------------- | +| Lambda Layer zip files | 38.83% | +| .jsii files | 30.14% | +| Source Maps (.js.map) | 15.87% | +| Javascript code (.js) | 6.63% | +| Type Declaration files (.d.ts) | 6.62% | +| README, etc (.md) | 0.55% | +| bundled npm dependencies | 0.50% | +| .ts-fixture | 0.27% | +| .jsiirc | 0.26% | ### Why should we _not_ do this? From d1f8a36a71f764d27a1ac4454e588c7b410f97ac Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 17:46:53 -0700 Subject: [PATCH 37/53] typo --- text/0039-reduce-module-size.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 674b52bbe..6301d5c6f 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -210,7 +210,7 @@ positive impact. ### What is the technical solution (design) of this feature? There are different categories of files contributing to the size of aws-cdk-lib, -and each category will have it’s own solution. The following sections cover the +and each category will have its own solution. The following sections cover the top five categories in size contribution in order from largest to smallest potential size. From e7950301b7a18eb0294379ba2feb9a4861c08dec Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 17:48:07 -0700 Subject: [PATCH 38/53] add globally installed to Lambda Layer soln top-level description --- text/0039-reduce-module-size.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 6301d5c6f..72e180344 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -217,11 +217,12 @@ potential size. #### Lambda Layer Zip Files Use separate npm packages for each `lambda-layer-X` module, which the CDK CLI -will automatically install if not already present in the `node_modules` -directory. With this solution, customers will not have to figure out when they -do or do not need to include these dependencies, and they will not be required -to access a public endpoint besides npm during runtime of their CDK apps. -Practically, this solution breaks down into the following steps. +will automatically install if not already present in the application's +`node_modules` directory, or the global `node_modules`. With this solution, +customers will not have to figure out when they do or do not need to include +these dependencies, and they will not be required to access a public endpoint +besides npm during runtime of their CDK apps. Practically, this solution breaks +down into the following steps. 1. Publish v2 compatible versions of `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-node-proxy-agent` as From 62c3a96a16a52103575465e33b3127056ab475ac Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 4 May 2022 18:12:15 -0700 Subject: [PATCH 39/53] update .jsii files solution --- text/0039-reduce-module-size.md | 36 ++++++++++++++++----------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 72e180344..5483919fe 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -347,12 +347,16 @@ documentation generation from the published npm package. Today, they make up 42% of the package size. And in the future, after adding more Lambda Layers, the .jsii files would compose 30% of the package size. -We will compress the .jsii.tabl.json and .jsii files. Compressing with gzip -currently creates a 4 MB and 4.7 MB file, respectively. This change will impact -the JSII runtime, and Construct Hub. NodeJS JSII runtimes don't need to use -either file, and non-NodeJS runtimes would decompress the .jsii file when -needed. Construct Hub generates documentation from the .jsii.tabl.json file, and -it will also decompress this file as needed. +We will take two steps to make them smaller: +1. The .jsii.tabl.json and .jsii files are both “pretty-printed.” We can reduce + their size to 42 MB (-23%) and 30 MB (-33%) by removing all the whitespace + and newlines. +2. Xompress the .jsii.tabl.json and .jsii files. Compressing with gzip + currently creates a 4 MB and 4.7 MB file, respectively. This change will impact + the JSII runtime, and Construct Hub. NodeJS JSII runtimes don't need to use + either file, and non-NodeJS runtimes would decompress the .jsii file when + needed. Construct Hub generates documentation from the .jsii.tabl.json file, and + it will also decompress this file as needed. #### Source Map Files (.js.map) @@ -390,18 +394,13 @@ Each category of files has different alternative solutions to reduce the size. #### .jsii Files -1. The .jsii.tabl.json and .jsii files are both “pretty-printed.” We can reduce - their size to 42 MB (-23%) and 30 MB (-33%) by removing all the whitespace - and newlines. We will not take this solution, because the space savings is - not as significant as compression. And, then we can maintain the benefit of - using pretty-printed JSON when uncompressed. -2. Vend the Rosetta tablet file (.jsii.tabl.json) separately. On each release, - we would publish this file as a separate artifact on GitHub releases, and - reference the URI from the jsii assembly (.jsii). This solution will not be - pursued at this time. There are several issues with de-coupling the Rosetta - tablet file from the artifacts that customers install, and with requiring the - JSII runtime to run in an environment that has access to the public endpoint - where it would be hosted. +Vend the Rosetta tablet file (.jsii.tabl.json) separately. On each release, we +would publish this file as a separate artifact on GitHub releases, and reference +the URI from the jsii assembly (.jsii). This solution will not be pursued at +this time. There are several issues with de-coupling the Rosetta tablet file +from the artifacts that customers install, and with requiring the JSII runtime +to run in an environment that has access to the public endpoint where it would +be hosted. #### Source Map Files (.js.map) @@ -512,6 +511,7 @@ this category: ### What is the high-level project plan? Each category of files can be worked on in parallel and addressed separately. + The highest priority category to start with will be the Lambda Layer zip files. Even though the Lambda Layers are not the biggest contributor to size today, they are the biggest potential contributor to size. And, there is customer need From ed8e6af0d734f300f7382edbb67f1e714720f267 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Wed, 11 May 2022 08:53:17 -0700 Subject: [PATCH 40/53] update lambda layer solution --- text/0039-reduce-module-size.md | 61 ++++++++++++++++++--------------- 1 file changed, 34 insertions(+), 27 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 5483919fe..7efc69cb6 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -71,8 +71,7 @@ RFC pull request): ### What are we launching today? Today, we are launching a smaller version of the AWS CDK v2 framework library, -`aws-cdk-lib`. And, a change in the AWS CDK v2 CLI to support this smaller -framework library. Before today’s release, the unpacked size of this package was +`aws-cdk-lib`. Before today’s release, the unpacked size of this package was 234 MB. This large size prevented many customers from being able to use it, and caused performance problems for others. It also prevented the CDK team from releasing certain new features. Now, the size of the package is less than 100 @@ -216,13 +215,12 @@ potential size. #### Lambda Layer Zip Files -Use separate npm packages for each `lambda-layer-X` module, which the CDK CLI -will automatically install if not already present in the application's -`node_modules` directory, or the global `node_modules`. With this solution, -customers will not have to figure out when they do or do not need to include -these dependencies, and they will not be required to access a public endpoint -besides npm during runtime of their CDK apps. Practically, this solution breaks -down into the following steps. +Use separate npm packages for each `lambda-layer-X` module. `aws-cdk-lib` will +dynamically load in these packages using `require()`, and handle installing them +if they are not available. With this solution, customers will not have to figure out when they do or do not need +to include these dependencies, and they will not be required to access a public +endpoint besides npm during runtime of their CDK apps. Practically, this +solution breaks down into the following steps. 1. Publish v2 compatible versions of `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-node-proxy-agent` as @@ -233,19 +231,26 @@ down into the following steps. published this way. Even though these packages will be released separately, their source code will remain in the aws-cdk repo. 2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these - large dependencies themselves, and instead reference the appropriate `Layer` - construct from the packages in step 1. Since `aws-cdk-lib`'s source code is - auto-generated from the v1 submodules, this should be an autogenerated - wrapper. The first two steps should not impact the current developer workflow - of working with these `Layer` constructs. Developers will work with each - @aws-cdk module in its corresponding source directory. The publishing - pipeline will publish the `@aws-cdk/lambda-layer-X` packages, and the - `aws-cdk-lib` code generation process will create the wrappers that reference - them. -3. Modify the CDK CLI to verify that the correct packages are available in the - `node_modules` directory during synthesis. If they are not available, - download them from npm. -4. We need to keep the dependencies up-to-date to get security updates and new + large dependencies themselves, and instead dynamically load the appropriate + `@aws-cdk/lambda-layer-X` package from step 1. Since `aws-cdk-lib`'s source + code is auto-generated from the v1 submodules, this logic should be + implemented once, and then autogenerated or reused in each `aws-cdk-lib` + submodule. wrapper. The first two steps should not impact the current + developer workflow of working with these `Layer` constructs. Developers will + work with each `@aws-cdk` module in its corresponding source directory. The + publishing pipeline will publish the `@aws-cdk/lambda-layer-X` packages, and + the `aws-cdk-lib` code generation process will create the wrappers that + reference them. + 1. I have implemented a POC that shows the following loading mechanism will + work with all JSII target languages: + ``` + try { + this.lambdaLayerXLibrary = require(packageName); + } catch(err) { + childproc.execSync(`npm install ${packageName} --no-save`); + } + ``` +3. We need to keep the dependencies up-to-date to get security updates and new features. Each Lambda Layer will need to be treated slightly differently. 1. `@aws-cdk/lambda-layer-kubectl` - This one has two dependencies to keep updated. All updates described below will result in a minor version bump @@ -313,16 +318,18 @@ steps. And those will be included in the Contributing Guide. This question raises an important drawback of this solution. It will result in a breaking change for customers who are using these Lambda Layers in -network-restricted environments. If customers are able to use the CDK CLI in +network-restricted environments. If customers are able to use the CDK in their network-restricted environments, then it is reasonable to assume they were able to acquire the `aws-cdk` package from npm somehow. This implies customers will be able to acquire the `@aws-cdk/lambda-layer-X` packages from npm. There are a few options for these customers: -1. If using TypeScript or JavaScript, add each necessary package the +1. If using TypeScript or JavaScript, add each necessary package to the dependencies of their CDK app. -2. Make sure the necessary packages are already installed and available in the - `node_modules` directory where their CDK CLI is installed. -3. Include them in a private npm registry. The +2. If using another target JSII language, add each necessary package as a + dependency as the normally would for other construct libraries. Each + `@aws-cdk/lambda-layer-X` package will be a JSII construct library, that is + also vended to the package managers for each target JSII language. +3. Include them in a private npm registry. Unfortunately, there is no way to remove these large files and host them somewhere else without causing a breaking change for these customers. We will From 0147ffd0d13762d0ac6348a851580c266a2216ab Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Sun, 10 Jul 2022 17:32:21 -0700 Subject: [PATCH 41/53] Minor edits, format, etc --- text/0039-reduce-module-size.md | 113 ++++++++++++++++---------------- 1 file changed, 56 insertions(+), 57 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 7efc69cb6..f0f00d738 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -114,8 +114,8 @@ to be addressed. customer is using Lambda Versions, and wants to maintain a Version for every version they have published, they might end up hitting this limit because of the large size of `aws-cdk-lib`. -6. The CDK project is currently blocked from including more ‘large - dependencies’, such as multiple versions of kubectl, multiple versions of the +6. The CDK project is currently blocked from including more 'large + dependencies', such as multiple versions of kubectl, multiple versions of the AWS CLI, and dedicated versions of the AWS SDK for use in custom resources. This is a decision we’ve made to avoid increasing the size of the package further until a more scalable solution for these kinds of dependencies is @@ -208,7 +208,7 @@ positive impact. ### What is the technical solution (design) of this feature? -There are different categories of files contributing to the size of aws-cdk-lib, +There are different categories of files contributing to the size of `aws-cdk-lib`, and each category will have its own solution. The following sections cover the top five categories in size contribution in order from largest to smallest potential size. @@ -217,10 +217,10 @@ potential size. Use separate npm packages for each `lambda-layer-X` module. `aws-cdk-lib` will dynamically load in these packages using `require()`, and handle installing them -if they are not available. With this solution, customers will not have to figure out when they do or do not need -to include these dependencies, and they will not be required to access a public -endpoint besides npm during runtime of their CDK apps. Practically, this -solution breaks down into the following steps. +if they are not available. With this solution, customers will not have to figure +out when they do or do not need to include these dependencies, and they will not +be required to access a public endpoint besides npm during runtime of their CDK +apps. Practically, this solution breaks down into the following steps. 1. Publish v2 compatible versions of `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-node-proxy-agent` as @@ -234,22 +234,15 @@ solution breaks down into the following steps. large dependencies themselves, and instead dynamically load the appropriate `@aws-cdk/lambda-layer-X` package from step 1. Since `aws-cdk-lib`'s source code is auto-generated from the v1 submodules, this logic should be - implemented once, and then autogenerated or reused in each `aws-cdk-lib` - submodule. wrapper. The first two steps should not impact the current + implemented once, and then autogenerated or reused in each `aws-cdk-lib/lambda-layer-X` + submodule wrapper. The first two steps should not impact the current developer workflow of working with these `Layer` constructs. Developers will work with each `@aws-cdk` module in its corresponding source directory. The publishing pipeline will publish the `@aws-cdk/lambda-layer-X` packages, and the `aws-cdk-lib` code generation process will create the wrappers that reference them. - 1. I have implemented a POC that shows the following loading mechanism will - work with all JSII target languages: - ``` - try { - this.lambdaLayerXLibrary = require(packageName); - } catch(err) { - childproc.execSync(`npm install ${packageName} --no-save`); - } - ``` + 1. I have implemented a POC that shows the proposed dyanmic loading mechanism will + work with JSII target languages. See [this repo]() for more details. 3. We need to keep the dependencies up-to-date to get security updates and new features. Each Lambda Layer will need to be treated slightly differently. 1. `@aws-cdk/lambda-layer-kubectl` - This one has two dependencies to keep @@ -269,8 +262,9 @@ solution breaks down into the following steps. auto-approved PR. 4. A weekly automated task will check for new supported minor versions of kubectl. If there is one, a PR will be automatically - created which creates a new `Layer` class in the API for the new - minor version. A human will need to review this one. + be created which creates a new `Layer` class in the API for the + new minor version. A human, the on-call engineer, will need to + review this one. 2. [helm](https://helm.sh/) is also included in each Lambda Layer in this package 1. Helm has a somewhat complicated version support policy documented @@ -289,7 +283,7 @@ solution breaks down into the following steps. available, and automatically updated to the latest minor or patch version. 1. This package will have two APIs: `AwsCliLayer`, and `AwsCliV2Layer`. - 2. Automatically check for new minor and patch versions from Pypi - + 2. Automatically check for new minor and patch versions from PyPi - [awscli](https://pypi.org/project/awscli/), [awscliv2](https://pypi.org/project/awscliv2/). Each update will result in a minor version bump of this package. @@ -312,7 +306,7 @@ steps. from the above steps to also update the new dependency. Once this solution is implemented, it will be possible to write more concrete -steps. And those will be included in the Contributing Guide. +steps. And those will be included in the aws-cdk Contributing Guide. ##### What about customers executing their CDK apps in network-restricted environments? @@ -325,10 +319,11 @@ will be able to acquire the `@aws-cdk/lambda-layer-X` packages from npm. There are a few options for these customers: 1. If using TypeScript or JavaScript, add each necessary package to the dependencies of their CDK app. -2. If using another target JSII language, add each necessary package as a - dependency as the normally would for other construct libraries. Each +2. If using a target JSII language, add each necessary package as a dependency + as thye normally would for other construct libraries. Each `@aws-cdk/lambda-layer-X` package will be a JSII construct library, that is - also vended to the package managers for each target JSII language. + also vended to the package managers for each target JSII language. For + example, Python users can install aws-cdk.lambda-layer-awscli 2.x from pip. 3. Include them in a private npm registry. Unfortunately, there is no way to remove these large files and host them @@ -351,20 +346,18 @@ package in ADC regions, the right dependencies will already be present in the There are two files, .jsii.tabl.json and .jsii, bundled in `aws-cdk-lib` for the JSII runtime to work for any non-NodeJS languages. And, to enable code and documentation generation from the published npm package. Today, they make up 42% -of the package size. And in the future, after adding more Lambda Layers, the .jsii files would -compose 30% of the package size. +of the package size. And in the future, after adding more Lambda Layers, the +.jsii files would compose 30% of the package size. -We will take two steps to make them smaller: -1. The .jsii.tabl.json and .jsii files are both “pretty-printed.” We can reduce - their size to 42 MB (-23%) and 30 MB (-33%) by removing all the whitespace - and newlines. -2. Xompress the .jsii.tabl.json and .jsii files. Compressing with gzip +We have two options to make them smaller: +1. Compress the .jsii.tabl.json and .jsii files. Compressing with gzip currently creates a 4 MB and 4.7 MB file, respectively. This change will impact the JSII runtime, and Construct Hub. NodeJS JSII runtimes don't need to use either file, and non-NodeJS runtimes would decompress the .jsii file when needed. Construct Hub generates documentation from the .jsii.tabl.json file, and it will also decompress this file as needed. + #### Source Map Files (.js.map) These files currently makeup 22% of the total size of `aws-cdk-lib`. Source map @@ -401,13 +394,17 @@ Each category of files has different alternative solutions to reduce the size. #### .jsii Files -Vend the Rosetta tablet file (.jsii.tabl.json) separately. On each release, we -would publish this file as a separate artifact on GitHub releases, and reference -the URI from the jsii assembly (.jsii). This solution will not be pursued at -this time. There are several issues with de-coupling the Rosetta tablet file -from the artifacts that customers install, and with requiring the JSII runtime -to run in an environment that has access to the public endpoint where it would -be hosted. +1. Vend the Rosetta tablet file (.jsii.tabl.json) separately. On each release, + we would publish this file as a separate artifact on GitHub releases, and + reference the URI from the jsii assembly (.jsii). This solution will not be + pursued at this time. There are several issues with de-coupling the Rosetta + tablet file from the artifacts that customers install, including adding a + requirement that the JSII runtime to run in an environment that has access to + the public endpoint where it would be hosted. +2. The .jsii.tabl.json and .jsii files are both “pretty-printed.” We can reduce + their size to 42 MB (-23%) and 30 MB (-33%) by removing all the whitespace + and newlines. We will not implement this, since it does not have a + significant enough impact on size. #### Source Map Files (.js.map) @@ -436,7 +433,7 @@ be hosted. dependency to aws-cdk-lib. The problem with this solution is that it requires access to a public endpoint during synthesis of CDK applications and does not lend itself to a straightforward workaround for customers who run CDK apps in - network-restricted environments (Construct Hub, ADC region customers). + network-restricted environments (Construct Hub, ADC region customers, etc). 2. Download the the zip files from a static website in the framework implementation of each `Layer` class, instead of in the CLI. We should not do this, because the CLI already has logic for managing credentials and making @@ -448,30 +445,32 @@ be hosted. resource definition in their CloudFormation template references the zip hosted by us in the same region as their deployment environment. The maintenance burden of keeping up with deploying these artifacts and keeping - them up to date in every AWS region is too high. This solution is the same as - the one described in this + them up to date in every AWS region is too high. This would effectively make + our team double as a hosted artifact repository for artifacts that we don't + own. This solution is the same as the one described in this [comment](https://github.com/aws/aws-cdk-rfcs/issues/39#issuecomment-593092612). 4. Separate packages that `aws-cdk-lib` peer depends on. With this solution, customers do not need to include the large `lambda-layer-X` packages in their dependencies unless they are actually using one of the Constructs that - requires them. This is not a reasonable solution, since it would be too - confusing for customers to know when they need these peer dependencies, and - when they do not. + requires them. This would be confusing for customers to know when they need + these peer dependencies, and when they do not. It would also be difficult for + customers to manage versions and to know which versions of each package are + compatible with each other. 5. Upstream the custom resources where the zip files are used. Since all of these zip files are used in custom resources, another way to remove them from `aws-cdk-lib` is to remove the custom resources themselves. We will create - AWS resources that are available in the CloudFormation public registry - under the namespace `AWS::CDK`. In addition to reducing the size of the - package, this solution also has another benefit of addressing pain points - with CDK publishing custom resources into customer accounts. The `XXLayer` - APIs on their own are also valuable to customers. If we are able to remove - their dependencies within `aws-cdk-lib` by removing the custom resources, then - we could vend these libraries separately for customers who use them - explicitly. We are not going to pursue this solution right now, since this - will increase the operational burden on our team by an unknown amount. The - proposed solution does not prevent us from also pursuing this in the - future. This would remove the need for customers in network-restricted to - perform a workaround to get these dependencies. + AWS resources that are available in the CloudFormation public registry under + the namespace `AWS::CDK`. This option would not require customers in + network-restricted to perform a workaround to get these dependencies. In + addition to reducing the size of the package, this solution also has another + benefit of addressing pain points with CDK publishing custom resources into + customer accounts. The `XXLayer` APIs on their own are also valuable to + customers. If we are able to remove their dependencies within `aws-cdk-lib` + by removing the custom resources, then we could vend these libraries + separately for customers who use them explicitly. We are not going to pursue + this solution right now, since this will increase the operational burden on + our team by an unknown amount. The proposed solution does not prevent us from + also pursuing this in the future. #### Javascript Files (.js) From 38994d25968934325827df04f920dfc5ce43e9c0 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Sun, 10 Jul 2022 20:50:46 -0700 Subject: [PATCH 42/53] formatting, update ADC region section, incorporate feedback for lambda-layer section --- text/0039-reduce-module-size.md | 109 +++++++++++++++++--------------- 1 file changed, 57 insertions(+), 52 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index f0f00d738..63a25f917 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -64,15 +64,16 @@ Ticking the box below indicates that the public API of this RFC has been signed-off by the API bar raiser (the `api-approved` label was applied to the RFC pull request): -- [x] Signed-off by API Bar Raiser [@otaviomacedo](https://github.com/otaviomacedo) +- [x] Signed-off by API Bar Raiser + [@otaviomacedo](https://github.com/otaviomacedo) ## Public FAQ ### What are we launching today? Today, we are launching a smaller version of the AWS CDK v2 framework library, -`aws-cdk-lib`. Before today’s release, the unpacked size of this package was -234 MB. This large size prevented many customers from being able to use it, and +`aws-cdk-lib`. Before today’s release, the unpacked size of this package was 234 +MB. This large size prevented many customers from being able to use it, and caused performance problems for others. It also prevented the CDK team from releasing certain new features. Now, the size of the package is less than 100 MB. This smaller version also has a mechanism for adding new Lambda Layer @@ -208,10 +209,10 @@ positive impact. ### What is the technical solution (design) of this feature? -There are different categories of files contributing to the size of `aws-cdk-lib`, -and each category will have its own solution. The following sections cover the -top five categories in size contribution in order from largest to smallest -potential size. +There are different categories of files contributing to the size of +`aws-cdk-lib`, and each category will have its own solution. The following +sections cover the top five categories in size contribution in order from +largest to smallest potential size. #### Lambda Layer Zip Files @@ -234,15 +235,18 @@ apps. Practically, this solution breaks down into the following steps. large dependencies themselves, and instead dynamically load the appropriate `@aws-cdk/lambda-layer-X` package from step 1. Since `aws-cdk-lib`'s source code is auto-generated from the v1 submodules, this logic should be - implemented once, and then autogenerated or reused in each `aws-cdk-lib/lambda-layer-X` - submodule wrapper. The first two steps should not impact the current - developer workflow of working with these `Layer` constructs. Developers will - work with each `@aws-cdk` module in its corresponding source directory. The - publishing pipeline will publish the `@aws-cdk/lambda-layer-X` packages, and - the `aws-cdk-lib` code generation process will create the wrappers that - reference them. - 1. I have implemented a POC that shows the proposed dyanmic loading mechanism will - work with JSII target languages. See [this repo]() for more details. + implemented once, and then autogenerated or reused in each + `aws-cdk-lib/lambda-layer-X` submodule wrapper. The first two steps should + not impact the current developer workflow of working with these `Layer` + constructs. Developers will work with each `@aws-cdk` module in its + corresponding source directory. The publishing pipeline will publish the + `@aws-cdk/lambda-layer-X` packages, and the `aws-cdk-lib` code generation + process will create the wrappers that reference them. + 1. I have implemented a POC that shows the proposed dynamic loading + mechanism will work with node languages and jsii target languages. See + [this + repo](https://github.com/madeline-k/dynamically-load-npm-dependencies) + for more details. 3. We need to keep the dependencies up-to-date to get security updates and new features. Each Lambda Layer will need to be treated slightly differently. 1. `@aws-cdk/lambda-layer-kubectl` - This one has two dependencies to keep @@ -252,26 +256,27 @@ apps. Practically, this solution breaks down into the following steps. 1. Kubernetes users will need to be able to choose the minor version of kubectl that they want to use. And the aws-eks module will need to as well. - 2. The public API for this package will have a `KubectlV1_20Layer` - class for each currently supported minor version of kubernetes. + 2. We will release multiple packages, one for each currently + supported minor version of kubernetes, e.g. + `@aws-cdk/lambda-layer-kubectl-120`. 3. A weekly automated task will check the [Kubernetes API](https://dl.k8s.io/release/stable.txt) for a new patch version of each currently supported minor version, and - automatically update the existing `Layer` class to use the new - patch version. This update will be performed with an - auto-approved PR. + automatically update the version of kubectl used in each package. + This update will be performed with an auto-approved PR. 4. A weekly automated task will check for new supported minor versions of kubectl. If there is one, a PR will be automatically - be created which creates a new `Layer` class in the API for the - new minor version. A human, the on-call engineer, will need to - review this one. + be created which creates a new + `@aws-cdk/lambda-layer-kubectl-XYZ` package for the new minor + version. A human, the on-call engineer, will need to review this + one. 2. [helm](https://helm.sh/) is also included in each Lambda Layer in - this package + these packages 1. Helm has a somewhat complicated version support policy documented [here](https://helm.sh/docs/topics/version_skew/). - 2. For each `KubectlVX_YLayer` class, we will include the latest - available version of Helm which is compatible with that version - of Kubernetes. + 2. For each `@aws-cdk/lambda-layer-kubectl-XYZ` package, we will + include the latest available version of Helm which is compatible + with that version of Kubernetes. 3. A weekly automated task will update the minor and patch versions of helm within the supported ranges. 4. The automated task that checks for new supported minor versions @@ -279,14 +284,15 @@ apps. Practically, this solution breaks down into the following steps. version of helm compatible with that version of kubectl. This will be verified by a human in PR review to make sure it is correct. - 2. `@aws-cdk/lambda-layer-aws-cli` - Major versions 1 and 2 will both be - available, and automatically updated to the latest minor or patch - version. - 1. This package will have two APIs: `AwsCliLayer`, and `AwsCliV2Layer`. + 2. `@aws-cdk/lambda-layer-aws-cli` - We will release two separate packages + to make major versions 1 and 2 both available, and automatically update + to the latest minor or patch version. + 1. Two packages: `@aws-cdk/lambda-layer-aws-cli-v1`, and + `@aws-cdk/lambda-layer-aws-cli-v2`. 2. Automatically check for new minor and patch versions from PyPi - [awscli](https://pypi.org/project/awscli/), [awscliv2](https://pypi.org/project/awscliv2/). Each update will - result in a minor version bump of this package. + result in a minor version bump of the corresponding package. 3. `@aws-cdk/lambda-layer-node-proxy-agent` - New minor versions will automatically be picked up and available with a minor version bump of the library. @@ -299,11 +305,12 @@ After the above is implemented, this mechanism can be extended to include additional Lambda Layers with different dependencies. The solution should be implemented in such a way that new dependencies can be added with the below steps. -1. Add a new directory in the `aws-cdk` monorepo: `@aws-cdk/lambda-layer-MyNewDependency`. +1. Add a new directory in the `aws-cdk` monorepo: + `@aws-cdk/lambda-layer-MyNewDependency`. 2. Implement the MyNewDependencyLayer API. 3. If the dependency is from npm, then updates will happen automatically. If it - is from some other source, then extend the automation - from the above steps to also update the new dependency. + is from some other source, then extend the automation from the above steps to + also update the new dependency. Once this solution is implemented, it will be possible to write more concrete steps. And those will be included in the aws-cdk Contributing Guide. @@ -312,11 +319,11 @@ steps. And those will be included in the aws-cdk Contributing Guide. This question raises an important drawback of this solution. It will result in a breaking change for customers who are using these Lambda Layers in -network-restricted environments. If customers are able to use the CDK in -their network-restricted environments, then it is reasonable to assume they were -able to acquire the `aws-cdk` package from npm somehow. This implies customers -will be able to acquire the `@aws-cdk/lambda-layer-X` packages from npm. There -are a few options for these customers: +network-restricted environments. If customers are able to use the CDK in their +network-restricted environments, then it is reasonable to assume they were able +to acquire the `aws-cdk` package from npm somehow. This implies customers will +be able to acquire the `@aws-cdk/lambda-layer-X` packages from npm. There are a +few options for these customers: 1. If using TypeScript or JavaScript, add each necessary package to the dependencies of their CDK app. 2. If using a target JSII language, add each necessary package as a dependency @@ -337,9 +344,9 @@ on the aws-cdk GitHub repository. Today, there is a mechanism for publishing AWS CDK into ADC (Amazon Dedicated Cloud) regions. This mechanism includes bundling the direct dependencies of the CDK and publishing them into the ADC regions, e.g. `constructs`. We will add the -`@aws-cdk/lambda-layer-X` packages to this process, and when customers use the aws-cdk npm -package in ADC regions, the right dependencies will already be present in the -`node_modules` directory. +`@aws-cdk/lambda-layer-X` npm packages, and their corresponding non-Node +language versions to this process. When customers use the lambda-layers in ADC +regions, the right dependencies will already be available. #### .jsii Files @@ -349,14 +356,12 @@ documentation generation from the published npm package. Today, they make up 42% of the package size. And in the future, after adding more Lambda Layers, the .jsii files would compose 30% of the package size. -We have two options to make them smaller: -1. Compress the .jsii.tabl.json and .jsii files. Compressing with gzip - currently creates a 4 MB and 4.7 MB file, respectively. This change will impact - the JSII runtime, and Construct Hub. NodeJS JSII runtimes don't need to use - either file, and non-NodeJS runtimes would decompress the .jsii file when - needed. Construct Hub generates documentation from the .jsii.tabl.json file, and - it will also decompress this file as needed. - +We will compress the .jsii.tabl.json and .jsii files. Compressing with gzip +currently creates a 4 MB and 4.7 MB file, respectively. This change will impact +the JSII runtime, and Construct Hub. NodeJS JSII runtimes don't need to use +either file, and non-NodeJS runtimes would decompress the .jsii file when +needed. Construct Hub generates documentation from the .jsii.tabl.json file, and +it will also decompress this file as needed. #### Source Map Files (.js.map) From 163d07f056c07511857e7e97865ed3cb1edeee9c Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Sun, 10 Jul 2022 20:53:59 -0700 Subject: [PATCH 43/53] rename JSII to jsii --- text/0039-reduce-module-size.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 63a25f917..3dad6e511 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -326,12 +326,12 @@ be able to acquire the `@aws-cdk/lambda-layer-X` packages from npm. There are a few options for these customers: 1. If using TypeScript or JavaScript, add each necessary package to the dependencies of their CDK app. -2. If using a target JSII language, add each necessary package as a dependency - as thye normally would for other construct libraries. Each - `@aws-cdk/lambda-layer-X` package will be a JSII construct library, that is - also vended to the package managers for each target JSII language. For - example, Python users can install aws-cdk.lambda-layer-awscli 2.x from pip. -3. Include them in a private npm registry. +2. If using a target jsii language, add each necessary package as a dependency + as they normally would for other construct libraries. Each + `@aws-cdk/lambda-layer-X` package will be a jsii construct library, that is + also vended to the package managers for each target jsii language. For + example, Python users can install aws-cdk.lambda-layer-awscli-v2 from pip. +3. Include them in a private artifact registry. Unfortunately, there is no way to remove these large files and host them somewhere else without causing a breaking change for these customers. We will @@ -351,14 +351,14 @@ regions, the right dependencies will already be available. #### .jsii Files There are two files, .jsii.tabl.json and .jsii, bundled in `aws-cdk-lib` for the -JSII runtime to work for any non-NodeJS languages. And, to enable code and +jsii runtime to work for any non-NodeJS languages. And, to enable code and documentation generation from the published npm package. Today, they make up 42% of the package size. And in the future, after adding more Lambda Layers, the .jsii files would compose 30% of the package size. We will compress the .jsii.tabl.json and .jsii files. Compressing with gzip currently creates a 4 MB and 4.7 MB file, respectively. This change will impact -the JSII runtime, and Construct Hub. NodeJS JSII runtimes don't need to use +the jsii runtime, and Construct Hub. NodeJS jsii runtimes don't need to use either file, and non-NodeJS runtimes would decompress the .jsii file when needed. Construct Hub generates documentation from the .jsii.tabl.json file, and it will also decompress this file as needed. @@ -404,7 +404,7 @@ Each category of files has different alternative solutions to reduce the size. reference the URI from the jsii assembly (.jsii). This solution will not be pursued at this time. There are several issues with de-coupling the Rosetta tablet file from the artifacts that customers install, including adding a - requirement that the JSII runtime to run in an environment that has access to + requirement that the jsii runtime to run in an environment that has access to the public endpoint where it would be hosted. 2. The .jsii.tabl.json and .jsii files are both “pretty-printed.” We can reduce their size to 42 MB (-23%) and 30 MB (-33%) by removing all the whitespace From 52d17356f5adc5a68c55a2e03074371e96ebedf6 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Sun, 10 Jul 2022 23:49:26 -0700 Subject: [PATCH 44/53] add more detail to type declaration files section --- text/0039-reduce-module-size.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 3dad6e511..a86d8cf0a 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -381,9 +381,19 @@ done to make them smaller. We can minify these files, which could result in a space saving of up to 20 MB. Minifying is the process of removing new-lines and whitespace characters from the files. The 20 MB number was collected by removing all new-line and -whitespace characters. In reality, we will need to do something slightly more -sophisticated and keep whitespace in documentation comments, so that -type-checking in IDEs still has readable type references. +whitespace characters. In reality, we can not actually remove all whitespace. We +will need to maintain whitespace in documentation comments, so that +type-checking in IDEs still has readable type references. As well as the minimum +whitespace necessary for the files to still be readable when navigating to them +in an IDE from usage of `aws-cdk-lib`. + +For the type declaration files, we will also pursue combining them into one file +per submodule. Currently, there is one `.d.ts` file per `.ts` file in the +`aws-cdk-lib` source code. This could lead to better compression of the library +overall. And, would improve performance for `tsc` invocations, since the +compiler would not need to read as many files. These improvements have not been +measured yet. They will be measured during the implementation to make sure the +changes are worth it. ### Is this a breaking change? From 5d6fda31d60330ed3724cb60513a2ddf93bce768 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Sun, 10 Jul 2022 23:50:46 -0700 Subject: [PATCH 45/53] formatting --- text/0039-reduce-module-size.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index a86d8cf0a..f8f2491e0 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -305,6 +305,7 @@ After the above is implemented, this mechanism can be extended to include additional Lambda Layers with different dependencies. The solution should be implemented in such a way that new dependencies can be added with the below steps. + 1. Add a new directory in the `aws-cdk` monorepo: `@aws-cdk/lambda-layer-MyNewDependency`. 2. Implement the MyNewDependencyLayer API. @@ -324,6 +325,7 @@ network-restricted environments, then it is reasonable to assume they were able to acquire the `aws-cdk` package from npm somehow. This implies customers will be able to acquire the `@aws-cdk/lambda-layer-X` packages from npm. There are a few options for these customers: + 1. If using TypeScript or JavaScript, add each necessary package to the dependencies of their CDK app. 2. If using a target jsii language, add each necessary package as a dependency From 41a96dc233d00a46b19c5f6e89a8b921914aa407 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Sun, 10 Jul 2022 23:59:26 -0700 Subject: [PATCH 46/53] add more clarity around the fact that we need to add more Lambda Layer dependencies in the future --- text/0039-reduce-module-size.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index f8f2491e0..d3af608c4 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -162,7 +162,10 @@ The CDK team is currently blocked on adding more Lambda Layer zip files to the AWS CDK framework, because adding them would dramatically increase the size of the package. Each of these zips are in a module whose only purpose is to bundle a dependency (or two) into a Lambda Layer, which is then used in custom -resources that are part of the AWS CDK framework. +resources that are part of the AWS CDK framework. We need a solution that is +scalable, so that more Lambda Layer dependencies can be added without bundling +them into `aws-cdk-lib` and significantly impacting its size. See section +[Lambda Layer Zip Files](#lambda-layer-zip-files) for the proposed solution. There are more dependencies like these that need to be added to the framework, but we are currently blocked on adding them with the current design because they From 3f6cb606fab3e864dbd2596b954a821f6ad371a9 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Mon, 11 Jul 2022 11:30:15 -0700 Subject: [PATCH 47/53] update .jsii files alternative solutions --- text/0039-reduce-module-size.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index d3af608c4..23e0958df 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -423,8 +423,10 @@ Each category of files has different alternative solutions to reduce the size. the public endpoint where it would be hosted. 2. The .jsii.tabl.json and .jsii files are both “pretty-printed.” We can reduce their size to 42 MB (-23%) and 30 MB (-33%) by removing all the whitespace - and newlines. We will not implement this, since it does not have a - significant enough impact on size. + and newlines. Since we are going to be compressing these files, removing + whitespace doesn't have much benefit. It will not impact either the unpacked + npm package size, or the compressed artifact size for all package manager. + Since the files will already be compressed in both versions. #### Source Map Files (.js.map) From 0ac3ff00b2592a45ff1da443169c7dbf66367eda Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Mon, 29 Aug 2022 00:43:45 -0700 Subject: [PATCH 48/53] add public Lambda Layers as a potential solution --- text/0039-reduce-module-size.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 23e0958df..190f4a1d2 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -471,14 +471,20 @@ Each category of files has different alternative solutions to reduce the size. our team double as a hosted artifact repository for artifacts that we don't own. This solution is the same as the one described in this [comment](https://github.com/aws/aws-cdk-rfcs/issues/39#issuecomment-593092612). -4. Separate packages that `aws-cdk-lib` peer depends on. With this solution, +4. Use [public Lambda + Layers](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html). + This would essentially be the same as alternative #3, because the code for + each public Lambda Layer would need to reside in S3. This solution adds + additional complexity for the naming and versioning convention of the + artifacts, so we would likely choose S3 over this. +5. Separate packages that `aws-cdk-lib` peer depends on. With this solution, customers do not need to include the large `lambda-layer-X` packages in their dependencies unless they are actually using one of the Constructs that requires them. This would be confusing for customers to know when they need these peer dependencies, and when they do not. It would also be difficult for customers to manage versions and to know which versions of each package are compatible with each other. -5. Upstream the custom resources where the zip files are used. Since all of +6. Upstream the custom resources where the zip files are used. Since all of these zip files are used in custom resources, another way to remove them from `aws-cdk-lib` is to remove the custom resources themselves. We will create AWS resources that are available in the CloudFormation public registry under From 62b388a573e925b8fa9ccbbdd08d9b07b607f192 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Mon, 29 Aug 2022 00:50:53 -0700 Subject: [PATCH 49/53] update solution to include using separate repositories for each lambda-layer package --- text/0039-reduce-module-size.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 190f4a1d2..e6de47fb8 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -227,13 +227,12 @@ be required to access a public endpoint besides npm during runtime of their CDK apps. Practically, this solution breaks down into the following steps. 1. Publish v2 compatible versions of `@aws-cdk/lambda-layer-aws-cli`, - `@aws-cdk/lambda-layer-aws-cli`, `@aws-cdk/lambda-layer-node-proxy-agent` as + `@aws-cdk/lambda-layer-kubectl`, `@aws-cdk/lambda-layer-node-proxy-agent` as their own npm packages, separate from `aws-cdk-lib`. This publishing process - will be an extension of the existing AWS CDK V2 release machinery, which - automatically publishes some packages as part of `aws-cdk-lib`, and some - packages separately. Any submodules with the prefix `lambda-layer` will be - published this way. Even though these packages will be released separately, - their source code will remain in the aws-cdk repo. + will be done in [cdklabs](https://github.com/cdklabs) repositories. Each + package will get its own repository. See [this + issue](https://github.com/aws/aws-cdk/issues/21605) for a detailed discussion + of the options, and why we have decided to use separate repositories. 2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these large dependencies themselves, and instead dynamically load the appropriate `@aws-cdk/lambda-layer-X` package from step 1. Since `aws-cdk-lib`'s source From 6fb3cc3bdf792b9b157d3f613e6cad92c9fbc123 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Mon, 29 Aug 2022 00:57:44 -0700 Subject: [PATCH 50/53] fix trailing spaces --- text/0039-reduce-module-size.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index e6de47fb8..e3ec964d1 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -232,7 +232,7 @@ apps. Practically, this solution breaks down into the following steps. will be done in [cdklabs](https://github.com/cdklabs) repositories. Each package will get its own repository. See [this issue](https://github.com/aws/aws-cdk/issues/21605) for a detailed discussion - of the options, and why we have decided to use separate repositories. + of the options, and why we have decided to use separate repositories. 2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these large dependencies themselves, and instead dynamically load the appropriate `@aws-cdk/lambda-layer-X` package from step 1. Since `aws-cdk-lib`'s source From f4ac7bd28a637d5d603b7f230a886bc304eefc6d Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Fri, 14 Oct 2022 12:42:16 -0700 Subject: [PATCH 51/53] Updates based on the latest progress of the implementation --- text/0039-reduce-module-size.md | 177 +++++++++++++++++--------------- 1 file changed, 95 insertions(+), 82 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index e3ec964d1..5985c7efa 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -12,51 +12,63 @@ goal of this project is to significantly reduce this size. ## Working Backwards -* **CHANGELOG**: - -feat(aws-cdk-lib): Reduce aws-cdk-lib package size. - -‼️ If you use any of the following Constructs in a network-restricted -environment, you might encounter problems upgrading to this version. ‼️ - -* lambda_layer_kubectl.KubectlLayer -* aws_eks.KubectlProvider -* aws_eks.HelmChart -* aws_eks.KubernetesManifest -* aws_eks.KubernetesObjectValue -* aws_eks.KubernetesPatch -* lambda_layer_awscli.AwsCliLayer -* aws_s3_deployment.BucketDeploymentß -* aws_stepfunctions_tasks.EmrContainersStartJobRun -* lambda_layer_node_proxy_agent.NodeProxyAgentLayer -* aws_eks.FargateProfile -* aws_eks.Cluster - -Please see [#1234567](https://github.com/aws/aws-cdk/issues/1234567) for more -details, and comment on the issue if you run into problems. - -* **CLI Notices**: - -We can deliver a notice to customers based on what specific constructs they are -using. We will define separate notices for each different scenario. - -Title: (aws-cdk-lib): upcoming change in packaging structure might require -action to upgrade - -Body: We’ve identified that you are using at least one of the [List of -Constructs] Constructs. If you are running your aws-cdk commands in an -environment that does not have access to npm, then you will need to make a -change to make version 2.x of [npm packages] available in your environment. See -[#1234567](https://github.com/aws/aws-cdk/issues/1234567) for details and -instructions. - -| List of Constructs | npm packages | -| -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | -| lambda_layer_kubectl.KubectlLayer | @aws-cdk/lambda_layer_kubectl | -| aws_eks.KubectlProvider
aws_eks.HelmChart
aws_eks.KubernetesManifest
aws_eks.KubernetesObjectValue
aws_eks.KubernetesPatch | @aws-cdk/lambda-layer-kubectl and @aws-cdk/lambda-layer-awscli | -| lambda_layer_awscli.AwsCliLayer
aws_s3_deployment.BucketDeployment
aws_step_functions_tasks.EmrContainersStartJobRun | @aws-cdk/lambda-layer-awscli | -| lamdba_layer_node_proxy_agent.NodeProxyAgentLayer
aws-eks.FargateProfile | @aws-cdk/lambda-layer-node-proxy-agent | -| aws_eks.Cluster | @aws-cdk/lambda-layer-kubectl, @aws-cdk/lambda-layer-awscli, and @aws-cdk/lambda-layer-node-proxy-agent | +Aside from the change in package size, the only customer-facing changes +resulting from this RFC will be the changes made to dynamically load dependencies +for the `lambda-layer` submodules. Since these changes may cause a breaking +change for customers depending on their environment, we will release in two +steps. blurg + +### CHANGELOG + +#### v2.Y.0 + +##### ⚠ BREAKING CHANGES TO EXPERIMENTAL FEATURES +- **lambda-layer-awscli, lambda-layer-kubectl, lambda-layer-node-proxy-agent:** + synthesis might fail in certain environments, see + [#22470](https://github.com/aws/aws-cdk/issues/22470) for more information + +##### Features + +- feat(lambda-layer-awscli, lambda-layer-kubectl, + lambda-layer-node-proxy-agent): reduce aws-cdk-lib package size, by removing + layer.zip files + +#### v2.X.0 (3 months earlier) + +##### Features +- feat(lambda-layer-awscli): Dynamically load asset for AwsCliLayer, with + bundled fallback +- feat(lambda-layer-kubectl): Dynamically load asset for KubectLayer, with + bundled fallback +- feat(lambda-layer-node-proxy-agent): Dynamically load asset for AwsCliLayer, + with bundled fallback + +### CLI Notices + +Starting with 2.X.0, we will deliver a notice to customers based on if their +application is using the layer.zip that was bundled into `aws-cdk-lib` as a +fallback. This will happen if their application is using `KubectlLayer`, +`AwsCliLayer`, or `NodeProxyAgentLayer` and their environment is not able to +download packages from npm. This could happen if the environment is network +restricted, does not have npm installed, or has npm configured to use a private +registry that does not include `@aws-cdk/asset-` packages. Customers in this +situation will see: + +``` +NOTICES + +22470 (lambda-layer-awscli): [ACTION REQUIRED] Your CDK application is using AwsCliLayer + + Overview: Your app is using the AwsCliLayer construct in an environment + where npm is inaccessible. Add a dependency on @aws-cdk/asset-awscli-v1, + or the equivalent in your language, to remove this notice. + + Affected versions: aws-cdk-lib.lambda_layer_awscli.Notice: ^2.X.0 + + More information at: https://github.com/aws/aws-cdk/issues/22470 +``` + +Repeat the same for `lambda-layer-kubectl`, and `lambda-layer-node-proxy-agent`. --- @@ -226,32 +238,29 @@ out when they do or do not need to include these dependencies, and they will not be required to access a public endpoint besides npm during runtime of their CDK apps. Practically, this solution breaks down into the following steps. -1. Publish v2 compatible versions of `@aws-cdk/lambda-layer-aws-cli`, - `@aws-cdk/lambda-layer-kubectl`, `@aws-cdk/lambda-layer-node-proxy-agent` as - their own npm packages, separate from `aws-cdk-lib`. This publishing process - will be done in [cdklabs](https://github.com/cdklabs) repositories. Each - package will get its own repository. See [this - issue](https://github.com/aws/aws-cdk/issues/21605) for a detailed discussion - of the options, and why we have decided to use separate repositories. -2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to not bundle these - large dependencies themselves, and instead dynamically load the appropriate - `@aws-cdk/lambda-layer-X` package from step 1. Since `aws-cdk-lib`'s source - code is auto-generated from the v1 submodules, this logic should be - implemented once, and then autogenerated or reused in each - `aws-cdk-lib/lambda-layer-X` submodule wrapper. The first two steps should - not impact the current developer workflow of working with these `Layer` - constructs. Developers will work with each `@aws-cdk` module in its - corresponding source directory. The publishing pipeline will publish the - `@aws-cdk/lambda-layer-X` packages, and the `aws-cdk-lib` code generation - process will create the wrappers that reference them. +1. Publish `@aws-cdk/asset-aws-cli-v1`, `@aws-cdk/asset-kubectl-v20`, + `@aws-cdk/lambda-layer-node-proxy-agent` as separate v2-compatible Construct + libraries. Each package will expose a Construct such as `AwsCliAsset` which + is an `s3_assets.Asset` and bundles the relevant dependency as an Asset + construct. This publishing process will be done in + [cdklabs](https://github.com/cdklabs) repositories. Each package will get its + own repository. See [this issue](https://github.com/aws/aws-cdk/issues/21605) + for a detailed discussion of the options, and why we have decided to use + separate repositories. +2. Modify the `lambda-layer-X` submodules in `aws-cdk-lib` to dynamically load + the appropriate `@aws-cdk/asset-X` package from step 1. This implementation + should use `npm` commands to download and cache the package's archive in the + CDK home directory, and then install the package into the current process's + `node_modules`. This logic should be implemented once, and then reused in + each `lambda-layer-X` submodule. 1. I have implemented a POC that shows the proposed dynamic loading mechanism will work with node languages and jsii target languages. See [this repo](https://github.com/madeline-k/dynamically-load-npm-dependencies) for more details. 3. We need to keep the dependencies up-to-date to get security updates and new - features. Each Lambda Layer will need to be treated slightly differently. - 1. `@aws-cdk/lambda-layer-kubectl` - This one has two dependencies to keep + features. Each asset package will need to be treated slightly differently. + 1. Kubectl - This one has two dependencies to keep updated. All updates described below will result in a minor version bump to this package. 1. [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) @@ -260,7 +269,8 @@ apps. Practically, this solution breaks down into the following steps. need to as well. 2. We will release multiple packages, one for each currently supported minor version of kubernetes, e.g. - `@aws-cdk/lambda-layer-kubectl-120`. + `@aws-cdk/lambda-layer-kubectl-v20`. Each one will expose a + Construct called `KubectlLayer`. 3. A weekly automated task will check the [Kubernetes API](https://dl.k8s.io/release/stable.txt) for a new patch version of each currently supported minor version, and @@ -272,7 +282,7 @@ apps. Practically, this solution breaks down into the following steps. `@aws-cdk/lambda-layer-kubectl-XYZ` package for the new minor version. A human, the on-call engineer, will need to review this one. - 2. [helm](https://helm.sh/) is also included in each Lambda Layer in + 2. [helm](https://helm.sh/) is also included in each asset in these packages 1. Helm has a somewhat complicated version support policy documented [here](https://helm.sh/docs/topics/version_skew/). @@ -286,32 +296,35 @@ apps. Practically, this solution breaks down into the following steps. version of helm compatible with that version of kubectl. This will be verified by a human in PR review to make sure it is correct. - 2. `@aws-cdk/lambda-layer-aws-cli` - We will release two separate packages - to make major versions 1 and 2 both available, and automatically update - to the latest minor or patch version. + 2. AWS CLI - We will release two separate packages to make major versions 1 + and 2 both available, and automatically update to the latest minor or + patch version. 1. Two packages: `@aws-cdk/lambda-layer-aws-cli-v1`, and `@aws-cdk/lambda-layer-aws-cli-v2`. 2. Automatically check for new minor and patch versions from PyPi - [awscli](https://pypi.org/project/awscli/), [awscliv2](https://pypi.org/project/awscliv2/). Each update will result in a minor version bump of the corresponding package. - 3. `@aws-cdk/lambda-layer-node-proxy-agent` - New minor versions will + 3. Node proxy-agent - New minor versions will automatically be picked up and available with a minor version bump of the library. 1. This can be implemented with `npm-check-updates`, since - `node-proxy-agent` is an npm package. - 4. AWS CDK v2 will automatically pick up new minor versions of the above - libraries. + `proxy-agent` is an npm package. + 4. AWS CDK v2 will automatically pick up new minor versions of the + `@aws-cdk/asset-XYZ` packages with our current dependency upgrade + automation. After the above is implemented, this mechanism can be extended to include additional Lambda Layers with different dependencies. The solution should be implemented in such a way that new dependencies can be added with the below steps. -1. Add a new directory in the `aws-cdk` monorepo: +1. Create a cdklabs repo and publish a Construct library called + `@aws-cdk/asset-MyNewDependency`. +2. Add a new directory in the `aws-cdk` monorepo: `@aws-cdk/lambda-layer-MyNewDependency`. -2. Implement the MyNewDependencyLayer API. -3. If the dependency is from npm, then updates will happen automatically. If it +3. Implement the MyNewDependencyLayer API. +4. If the dependency is from npm, then updates will happen automatically. If it is from some other source, then extend the automation from the above steps to also update the new dependency. @@ -325,21 +338,21 @@ breaking change for customers who are using these Lambda Layers in network-restricted environments. If customers are able to use the CDK in their network-restricted environments, then it is reasonable to assume they were able to acquire the `aws-cdk` package from npm somehow. This implies customers will -be able to acquire the `@aws-cdk/lambda-layer-X` packages from npm. There are a +be able to acquire the `@aws-cdk/asset-X` packages from npm. There are a few options for these customers: 1. If using TypeScript or JavaScript, add each necessary package to the dependencies of their CDK app. 2. If using a target jsii language, add each necessary package as a dependency as they normally would for other construct libraries. Each - `@aws-cdk/lambda-layer-X` package will be a jsii construct library, that is + `@aws-cdk/asset-X` package will be a jsii construct library, that is also vended to the package managers for each target jsii language. For - example, Python users can install aws-cdk.lambda-layer-awscli-v2 from pip. + example, Python users can install aws-cdk.asset-awscli-v2 from pip. 3. Include them in a private artifact registry. Unfortunately, there is no way to remove these large files and host them somewhere else without causing a breaking change for these customers. We will -publish a CLI notice directly to potentially-affected customers of this change +publish a CLI notice directly to affected customers of this change well in advance of releasing it. And, advertise the change in advance of release on the aws-cdk GitHub repository. @@ -348,7 +361,7 @@ on the aws-cdk GitHub repository. Today, there is a mechanism for publishing AWS CDK into ADC (Amazon Dedicated Cloud) regions. This mechanism includes bundling the direct dependencies of the CDK and publishing them into the ADC regions, e.g. `constructs`. We will add the -`@aws-cdk/lambda-layer-X` npm packages, and their corresponding non-Node +`@aws-cdk/asset-X` npm packages, and their corresponding non-Node language versions to this process. When customers use the lambda-layers in ADC regions, the right dependencies will already be available. From 96b71da681de1141b5c447cedc0387a229d172e8 Mon Sep 17 00:00:00 2001 From: Madeline Kusters Date: Fri, 14 Oct 2022 13:38:29 -0700 Subject: [PATCH 52/53] fix --- text/0039-reduce-module-size.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 5985c7efa..312782a45 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -16,7 +16,7 @@ Aside from the change in package size, the only customer-facing changes resulting from this RFC will be the changes made to dynamically load dependencies for the `lambda-layer` submodules. Since these changes may cause a breaking change for customers depending on their environment, we will release in two -steps. blurg +steps. ### CHANGELOG From a75943c82a48c0b08725e931e8c911fadbb42c0b Mon Sep 17 00:00:00 2001 From: Kaizen Conroy <36202692+kaizencc@users.noreply.github.com> Date: Fri, 14 Oct 2022 17:13:56 -0400 Subject: [PATCH 53/53] update awscli v2 automatic check for version updates --- text/0039-reduce-module-size.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/text/0039-reduce-module-size.md b/text/0039-reduce-module-size.md index 312782a45..dae9f028b 100644 --- a/text/0039-reduce-module-size.md +++ b/text/0039-reduce-module-size.md @@ -301,10 +301,14 @@ apps. Practically, this solution breaks down into the following steps. patch version. 1. Two packages: `@aws-cdk/lambda-layer-aws-cli-v1`, and `@aws-cdk/lambda-layer-aws-cli-v2`. - 2. Automatically check for new minor and patch versions from PyPi - - [awscli](https://pypi.org/project/awscli/), - [awscliv2](https://pypi.org/project/awscliv2/). Each update will + 2. AWS CLI v1: automatically check for new minor and patch versions from PyPi - + [awscli](https://pypi.org/project/awscli/). Each update will result in a minor version bump of the corresponding package. + 3. AWS CLI v2: automatically check for new minor and patch versions from offical + awscliv2 [Dockerfile](https://hub.docker.com/r/amazon/aws-cli). AWS CLI v2 does + not expose an official PyPi package, but Dependabot can automatically check new + Dockerfile versions. Each update will result in a minor version bump of the + corresponding package. 3. Node proxy-agent - New minor versions will automatically be picked up and available with a minor version bump of the library.