feat(lambda-python): support setting environment vars for bundling #18635

setu4993 · 2022-01-25T05:19:02Z

While using the Python Lambda with Code Artifact, discovered that Code Artifact was still inaccessible because bundling occurs at run time, which can only access env vars, not build args.

This is not a security issue because bundled output doesn't contain any of the secret values.

Note: Without this, using Code Artifact (or any other private packaging for Python Lambdas) is currently broken.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

gitpod-io · 2022-01-25T05:19:05Z

setu4993 · 2022-01-25T05:20:26Z

packages/@aws-cdk/aws-lambda-python/lib/Dockerfile

+# Upgrade pip (required by cryptography v3.4 and above, which is a dependency of poetry)
+RUN pip install --upgrade pip
+RUN pip install pipenv poetry


Moved this up since if using Code Artifact, this install uses the private repo, which forces rebuilds of the Docker image every time (and skipping Docker cache).

setu4993 · 2022-01-25T05:50:21Z

packages/@aws-cdk/aws-lambda-python/test/bundling.test.ts

+test('Bundling with custom build arg for `PIP_EXTRA_INDEX_URL`', () => {
+  const entry = path.join(__dirname, 'lambda-handler');
+  const testPypi = 'https://test.pypi.org/simple/';
+  Bundling.bundle({
+    entry: entry,
+    runtime: Runtime.PYTHON_3_7,
+    buildArgs: { PIP_EXTRA_INDEX_URL: testPypi },
+  });
+
+  expect(DockerImage.fromBuild).toHaveBeenCalledWith(expect.stringMatching(path.join(__dirname, '../lib')), expect.objectContaining({
+    buildArgs: expect.objectContaining({
+      PIP_EXTRA_INDEX_URL: testPypi,
+    }),
+  }));
+});


Added this to pacify the validate-pr CI. Not sure if there's a way to validate env vars in Docker images (the change we are making here).

setu4993 · 2022-01-25T18:19:03Z

@corymhall : Any feedback on this? Tiny fix, so would be great to get into the next release.

corymhall · 2022-01-25T19:20:48Z

@setu4993

Alternatively, this could be passed as environment variables to the BundlingOptions, but that'd require quite a bit more of refactoring to modify the current DockerImage.fromBuild(...) call, which doesn't support the environment prop.

I think it should be pretty easy to add environment variable support. You just need to add environment to BundlingOptions and then assign it to a public environment property on Bundling. Those env vars then get added to the Docker environment.

aws-cdk/packages/@aws-cdk/aws-lambda-nodejs/lib/bundling.ts

Line 82 in 2465b51

public readonly environment?: { [key: string]: string };

aws-cdk/packages/@aws-cdk/aws-lambda-nodejs/lib/bundling.ts

Line 146 in 2465b51

this.environment = props.environment;

setu4993 · 2022-01-26T07:27:41Z

I think it should be pretty easy to add environment variable support. You just need to add environment to BundlingOptions and then assign it to a public environment property on Bundling. Those env vars then get added to the Docker environment.

Neat! I thought I was missing something obvious. Thanks for the pointer, updating the PR.

setu4993

@corymhall : Thanks for the tip! Updated the PR and docs.

setu4993 · 2022-01-26T08:33:40Z

packages/@aws-cdk/aws-lambda-python/README.md

+});
+```
+
+The index URL or the token are only used during bundling and thus not included in the final asset.


Added note about tokens not being persisted in the final asset.

setu4993 · 2022-01-26T08:33:56Z

packages/@aws-cdk/aws-lambda-python/README.md

@@ -172,4 +199,33 @@ new lambda.PythonFunction(this, 'function', {
 });
 ```

-This type of an example should work for `pip` and `poetry` based dependencies, but will not work for `pipenv`.
+**Note:** Setting custom build args for bundling will force the base bundling image to be rebuilt every time (i.e. skip the Docker cache).


And about rebuilding Docker images for bundling if using build args.

setu4993 · 2022-01-26T08:34:44Z

packages/@aws-cdk/aws-lambda-python/README.md

+Setting only environment variable should work for `pip` and `poetry` based dependencies, whereas `pipenv` based dependencies will require **both** build args and environment variables to be set.
+
+
+Example for using Code Artifact with `pipenv`-based dependencies:


Running into this also made clear how we could use pipenv with Code Artifact.

Why does pipenv require both env vars and build args?

Thanks for asking, @corymhall. That was an incorrect note on my part stemming from how bundling was occurring earlier (in the Docker build step), and I didn't completely understand it since I don't use pipenv.

Over the last couple days, I created a test project with pipenv and validated installing a package into it that is only on Test PyPI. It works with the PIP_INDEX_URL environment variable set (same as for pip and poetry), which it'll be when the Docker run step occurs. I updated the documentation now to reflect that.

packages/@aws-cdk/aws-lambda-python/test/bundling.test.ts

corymhall · 2022-01-26T15:12:11Z

packages/@aws-cdk/aws-lambda-python/README.md

+Setting only environment variable should work for `pip` and `poetry` based dependencies, whereas `pipenv` based dependencies will require **both** build args and environment variables to be set.
+
+
+Example for using Code Artifact with `pipenv`-based dependencies:


Why does pipenv require both env vars and build args?

Pull request has been modified.

corymhall

Looks good!

mergify · 2022-01-28T13:03:32Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2022-01-28T13:53:52Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
Commit ID: 90fe3ad
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2022-01-28T13:54:20Z

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

…ws#18635) While using the Python Lambda with Code Artifact, discovered that Code Artifact was still inaccessible because bundling occurs at _run_ time, which can only access env vars, not build args. This is not a security issue because bundled output doesn't contain any of the secret values. **Note:** Without this, using Code Artifact (or any other private packaging for Python Lambdas) is currently broken. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions bot added the @aws-cdk/aws-lambda-python label Jan 25, 2022

github-actions bot assigned corymhall Jan 25, 2022

setu4993 commented Jan 25, 2022

View reviewed changes

setu4993 force-pushed the fix/lambda-python/bundling-arg-env branch from 1847f89 to 2e8ba56 Compare January 25, 2022 05:49

setu4993 commented Jan 25, 2022

View reviewed changes

setu4993 changed the title ~~fix(lambda-python): set env vars for bundling~~ fix(lambda-python): set env vars from build args for bundling Jan 25, 2022

setu4993 force-pushed the fix/lambda-python/bundling-arg-env branch from 2e8ba56 to 220d089 Compare January 26, 2022 07:39

setu4993 changed the title ~~fix(lambda-python): set env vars from build args for bundling~~ fix(lambda-python): support setting environment vars for bundling Jan 26, 2022

setu4993 force-pushed the fix/lambda-python/bundling-arg-env branch from 00005df to a8dbafe Compare January 26, 2022 08:32

setu4993 changed the title ~~fix(lambda-python): support setting environment vars for bundling~~ feat(lambda-python): support setting environment vars for bundling Jan 26, 2022

setu4993 commented Jan 26, 2022

View reviewed changes

corymhall previously requested changes Jan 26, 2022

View reviewed changes

setu4993 added 3 commits January 28, 2022 00:33

feat: support setting environment variables during bundling

ceb1c9a

test: add test for validating env vars

4a65206

docs: update to explain usage of Code Artifact

1a7fc45

setu4993 force-pushed the fix/lambda-python/bundling-arg-env branch from 3a75397 to 1a7fc45 Compare January 28, 2022 08:33

corymhall approved these changes Jan 28, 2022

View reviewed changes

Merge branch 'master' into fix/lambda-python/bundling-arg-env

90fe3ad

mergify bot merged commit 30e2233 into aws:master Jan 28, 2022

setu4993 deleted the fix/lambda-python/bundling-arg-env branch January 28, 2022 21:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(lambda-python): support setting environment vars for bundling #18635

feat(lambda-python): support setting environment vars for bundling #18635

setu4993 commented Jan 25, 2022 •

edited

Loading

gitpod-io bot commented Jan 25, 2022

setu4993 Jan 25, 2022 •

edited

Loading

setu4993 Jan 25, 2022

setu4993 commented Jan 25, 2022

corymhall commented Jan 25, 2022

setu4993 commented Jan 26, 2022

setu4993 left a comment

setu4993 Jan 26, 2022

setu4993 Jan 26, 2022 •

edited

Loading

setu4993 Jan 26, 2022

corymhall Jan 26, 2022

setu4993 Jan 28, 2022 •

edited

Loading

corymhall Jan 26, 2022

corymhall left a comment

mergify bot commented Jan 28, 2022

aws-cdk-automation commented Jan 28, 2022

mergify bot commented Jan 28, 2022

		Setting only environment variable should work for `pip` and `poetry` based dependencies, whereas `pipenv` based dependencies will require both build args and environment variables to be set.


		Example for using Code Artifact with `pipenv`-based dependencies:

feat(lambda-python): support setting environment vars for bundling #18635

feat(lambda-python): support setting environment vars for bundling #18635

Conversation

setu4993 commented Jan 25, 2022 • edited Loading

gitpod-io bot commented Jan 25, 2022

setu4993 Jan 25, 2022 • edited Loading

Choose a reason for hiding this comment

setu4993 Jan 25, 2022

Choose a reason for hiding this comment

setu4993 commented Jan 25, 2022

corymhall commented Jan 25, 2022

setu4993 commented Jan 26, 2022

setu4993 left a comment

Choose a reason for hiding this comment

setu4993 Jan 26, 2022

Choose a reason for hiding this comment

setu4993 Jan 26, 2022 • edited Loading

Choose a reason for hiding this comment

setu4993 Jan 26, 2022

Choose a reason for hiding this comment

corymhall Jan 26, 2022

Choose a reason for hiding this comment

setu4993 Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

corymhall Jan 26, 2022

Choose a reason for hiding this comment

corymhall left a comment

Choose a reason for hiding this comment

mergify bot commented Jan 28, 2022

aws-cdk-automation commented Jan 28, 2022

AWS CodeBuild CI Report

mergify bot commented Jan 28, 2022

setu4993 commented Jan 25, 2022 •

edited

Loading

setu4993 Jan 25, 2022 •

edited

Loading

setu4993 Jan 26, 2022 •

edited

Loading

setu4993 Jan 28, 2022 •

edited

Loading