ref(build): Use rollup to build AWS lambda layer #5146
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
size-limit report 📦
|
Lms24
approved these changes
May 23, 2022
| "build": "run-p build:rollup build:types && yarn build:extras", | ||
| "build:awslambda-layer": "node scripts/build-awslambda-layer.js", | ||
| "build": "run-p build:rollup build:types build:bundle && yarn build:extras", | ||
| "build:awslambda-layer": "echo 'WARNING: AWS lambda layer build emporarily moved to \\`build:bundle\\`.'", |
There was a problem hiding this comment.
This will be refactored in an upcoming PR, right?
There was a problem hiding this comment.
Depends on your definition of "upcoming." 😛
But yes, eventually, once I've applied the "wait only as long as you need to" to all of the "extras" steps, then the "extras" build can happen in parallel with the others, and this can move back under that umbrella.
| console.log('Creating symlink for `awslambda-auto.js` in legacy `dist` directory.'); | ||
| fsForceMkdirSync('build/aws/dist-serverless/nodejs/node_modules/@sentry/serverless/dist'); | ||
| fs.symlinkSync( | ||
| '../build/cjs/awslambda-auto.js', |
There was a problem hiding this comment.
I might be mistaken but shouldn't this file be in build/npm?
Suggested change
| '../build/cjs/awslambda-auto.js', | |
| '../build/npm/cjs/awslambda-auto.js', |
There was a problem hiding this comment.
Wow. Not only are you right, you're making me realize that there's a number of other places I should have fixed this, too. Yikes. Thank you for catching that!
| @@ -0,0 +1,66 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
I think we might want to convert this script into a node script at some point in the future because we recently moved from bash scripts to node scripts for platform independence (see #4720 for context). But this can easily be done some time in the future in a follow-up PR since it's a new script and our contributors won't need it
There was a problem hiding this comment.
I added a note to the top. I think it makes sense to keep it in bash, because it's supposed to be a mirror of the GHA, which is also written in bash.
lobsterkatie
force-pushed
the
kmclb-use-rollup-for-lambda-layer
branch
5 times, most recently
from
7565aa0 to
5b31c75
Compare
lobsterkatie
force-pushed
the
kmclb-use-rollup-for-lambda-layer
branch
2 times, most recently
from
5e56e20 to
cce54c0
Compare
AbhiPrasad
pushed a commit
that referenced
this pull request
May 30, 2022
Our current system for building the AWS lambda layer involves a script which manually traces the serverless package's dependencies, including other monorepo packages, and symlinks them into a `node_modules` folder in a directory set up to mimic an installation from npm. There are a few disadvantages to this system: - The signals we rely on to figure out what is and isn't a dependency aren't exhaustive, and we're not experts at this, both of which have made getting the right files and only the right files into the layer a brittle process, which has broken down more than once and caused issue for our users. - The script is complicated, which makes it harder to figure out exactly what's causing it when something does go wrong. - We symlink in entire packages at a time, regardless of whether or not we're using most of the code. This is true even of the serverless package itself, where we include all of the GCP code along with the AWS code. This refactors our lambda layer build process to use the new bundling config functions we use for CDN bundles, to create a bundle which can take the place of the the index file, but which contains everything it needs internally. This has the following advantages: - This puts Rollup in charge of figuring out dependencies instead of us. - The config is in line with existing configs and is therefore much easier to reason about and maintain. - It lets us easily exclude anything GCP-related in the SDK. Between that, Rollup's treeshaking, and terser's minifying, the layer will now take up much less of the finite size allotted to each lambda function. - Removing extraneous files means less to cache and retrieve from the cache in each GHA job. Key changes: - The layer now builds in `packages/serverless/build/aws` rather than at the top level of the repo. (It is now moved to the top level only in our GHA workflow creating the lambda zip.) - In that workflow, the process to determine the SDK version has been simplified. - The bundle builds based not on the main index file but on a new bundle-specific index file, which only includes AWS code. - There is new rollup config just for the layer, which uses the bundle-building functions to make the main bundle and the npm-package-building functions to create a separate module for the optional script which automatically starts up the SDK in AWS. - The old build script has been replaced by one which runs rollup with that config, and then symlinks the built auto-startup script into its legacy location, so as to be backwards compatible with folks who run it using an environment variable pointing to the old path. - The building of the layer has temporarily been shifted from `build:awslambda-layer` to `build:bundle`, so that it will run during the first phase of the repo-level `yarn build`. (The goal is to eventually do everything in one phase, for greater parallelization and shorter overall build time.) h/t to @antonpirker for all his help testing and thinking through this with me.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our current system for building the AWS lambda layer involves a script which manually traces the serverless package's dependencies, including other monorepo packages, and symlinks them into a
node_modulesfolder in a directory set up to mimic an installation from npm. There are a few disadvantages to this system:This refactors our lambda layer build process to use the new bundling config functions we use for CDN bundles, to create a bundle which can take the place of the the index file, but which contains everything it needs internally. This has the following advantages:
Key changes:
packages/serverless/build/awsrather than at the top level of the repo. (It is now moved to the top level only in our GHA workflow creating the lambda zip.)build:awslambda-layertobuild:bundle, so that it will run during the first phase of the repo-levelyarn build. (The goal is to eventually do everything in one phase, for greater parallelization and shorter overall build time.)h/t to @antonpirker for all his help testing and thinking through this with me.