(aws-lambda-python): cache Docker layer with dependencies

[m-radzikowski]

Describe the feature

Bundling Python Lambdas that contain requirements.txt, Pipfile, or poetry.lock file happens in a Docker container. Firstly, the requirements.txt file is generated (for pipenv and Poetry), and then dependencies are installed.

However, this happens each time any code change is made. Each time you change Lambda code (not its dependencies), the Docker build performs the above steps, downloading libraries from the internet.

This is the responsible code:

aws-cdk/packages/@aws-cdk/aws-lambda-python/lib/bundling.ts

Lines 109 to 122 in 4613886

	private createBundlingCommand(options: BundlingCommandOptions): string[] {
	const packaging = Packaging.fromEntry(options.entry, options.poetryIncludeHashes);
	let bundlingCommands: string[] = [];
	bundlingCommands.push(...options.commandHooks?.beforeBundling(options.inputDir, options.outputDir) ?? []);
	bundlingCommands.push(`cp -rTL ${options.inputDir}/ ${options.outputDir}`);
	bundlingCommands.push(`cd ${options.outputDir}`);
	bundlingCommands.push(packaging.exportCommand ?? '');
	if (packaging.dependenciesFile) {
	bundlingCommands.push(`python -m pip install -r ${DependenciesFile.PIP} -t ${options.outputDir}`);
	}
	bundlingCommands.push(...options.commandHooks?.afterBundling(options.inputDir, options.outputDir) ?? []);
	return bundlingCommands;
	}
	}

Dependencies change far less often than the code. Best practices for building in Docker are to firstly download dependencies and only then copy the code. This allows the dependencies layer to be cached, and on consecutive runs, only the code is updated while the dependencies layer is cached.

Use Case

This will greatly reduce consecutive Lambda bundling times, as dependencies will be fetched from the internet only when they change. When only the code changes, a cached Docker layer with dependencies will be used.

Proposed Solution

In short, bundling Python Lambda should be changed from:

1. Copy all source files (line 113)
2. Generate requirements.txt (line 115)
3. Install dependencies (line 117)

to:

1. Copy dependencies files (i.e. poetry.lock)
2. Generate requirements.txt
3. Install dependencies
4. Copy the rest of the files

Other Information

I am willing to implement it after greenlighting by the CDK team.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.61.1

Environment details (OS name and version, etc.)

any

[pahud]

Thanks for your idea. I am making it p2 feature request and will raise awareness to the team.

[m-radzikowski]

I looked more at this issue. Changing the command order will not help here because those commands do not create next Docker layers. They are all executed with docker run here (or, to be precise, here), something like:

docker run --rm build_python_image bash -c "cp source/ /asset-input && cd /asset-input && pip install -r requirements.txt && ..."

I see several potential solutions.

Option 1 - build second Docker image

To utilize caching from Docker layers, we would need to build an image from a Dockerfile, where those commands are executed one by one. We could dynamically create a Dockerfile, put only (possibly generated) requirements.txt file in it, and install dependencies. Then the bundling commands would only copy rest of the files.

So instead of:

aws-cdk/packages/@aws-cdk/aws-lambda-python/lib/bundling.ts

Lines 100 to 106 in 29bdd6c

	this.image = image ?? DockerImage.fromBuild(path.join(__dirname, '../lib'), {
	buildArgs: {
	...props.buildArgs,
	IMAGE: runtime.bundlingImage.image,
	},
	platform: architecture.dockerPlatform,
	});

something like this:

    const baseImage = image ?? DockerImage.fromBuild(path.join(__dirname, '../lib'), {
      buildArgs: {
        ...props.buildArgs,
        IMAGE: runtime.bundlingImage.image,
      },
      platform: architecture.dockerPlatform,
    });

    const tempDockerfile = makeTempFile();
    fs.writeFileSync(tempDockerfile, `
      FORM ${baseImage.image}
      
      RUN mkdir -p ${outputPath} && cd ${outputPath}
      COPY requirements.txt .
      RUN python -m pip install -r ${DependenciesFile.PIP} -t ${outputPath}
    `);

    this.image = DockerImage.fromBuild(tempDockerfile);

Then remove commands for installing dependencies from createBundlingCommand().

A variation of this would be using docker commit to create new layers.

Option 2 - build Lambda Layer for dependencies

Build a Lambda Layer with dependencies only and attach it to the Lambda. Similar to the above, to not rebuild them every time (fetching dependencies from the internet), we need caching. We can either generate and build a Dockerfile to utilize Docker layers (like in option 1), or maybe cache the lockfile in cdk.out and check if it changed instead of rebuilding the Layer every time? This will solve the problem for Poetry and Pipenv at least.

Option 3 - cache dependencies

If we would install dependencies to a separate directory in the bundle, like /asset-output/cdk-python-libs, maybe on the next build we could copy them from the previous asset bundle instead of re-installing them from the internet, if the lockfile did not change?

However, I'm not sure if accessing the previous bundle is possible (we would need to know the bundle hash). Alternatively, maybe we could copy libs dir to our cache after bundling? So next time we take it from our cache dir.

Additional paths for Python libraries can be added with LD_LIBRARY_PATH Lambda env variable.

Option 4 - copy files from Python virtual env

The whole pip install for Python Lambda is, as I understand, a consequence of unstandardized dependency installation location in Python and virtual environments. For comparison, Nodejs Lambda does not run npm install, since the dependencies are accessible in node_modules no matter the dependency management tool.

For pipenv and Poetry, we could get the virtual environment location (pipenv --where and poetry env info --path) where dependencies are installed. For pip, we could add a parameter to specify the location.

Then we could copy dependencies from the virtual environment location instead of downloading them from the internet.

Consequently, you would need to install dependencies manually before running cdk synth/deploy. But it's already like this for Nodejs (and Go? I'm not sure). Would it work if you provided install command (like poetry install) in a before bundle command hook?

This could be a flag, so the current mode is the default (as a more reliable one), but you could opt-in to copy already installed dependencies from disk instead of fetching them from the internet each time.

---

I'm happy to work on this, but there is a need for input from the CDK team first to find the best solution.

[avandekleut]

These all sound like good ideas. I actually expected the lambda layer method to be how the python layers worked. I’d love to see this implemented.

[jvcl]

I ran into this this week, developing a CDK app using the Python Lambda Construct. Currently using SAM to run it the project locally it is really slow as every time there is code change the bundling install all the packages again even when the dependencies didn't change.

[lsmarsden]

I'm seeing a similar problem, including when the lambda code is not changed at all. For example, running multiple tests against a stack that contains a PythonFunction causes repeated bundling (identical to this issue for aws-lambda-nodejs).

Yes, the first Docker layers are cached, but as mentioned by others, the dependency download isn't, and takes a long time. For the time being, I've set every Python lambda function that doesn't have dependencies to use a standard Function implementation over PythonFunction.

However, using a PythonFunction for the lambda functions with dependencies has extended a currently small test suite from approx 1 min to over 10 min due to the repeated bundling and dependency installations on each test. A crude benchmark is showing the following differences for test runs bundling the same python function (with cached Docker layers):

• PythonFunction with requirements.txt ~ 40s
• PythonFunction with no requirements.txt ~ 1s
• Function with no bundling ~ 10ms

It would be fantastic to have an implementation to fix this. Those 40 seconds stack up when you have multiple tests.

[lsmarsden]

I've been looking for a workaround in the meantime, with the following Dockerfile (only covering requirements.txt):

FROM python:3.10.12

ARG FUNCTION_SRC

ENV ASSET_DIR=/assets

COPY $FUNCTION_SRC $ASSET_DIR

RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir -r $ASSET_DIR/requirements.txt -t $ASSET_DIR && \
    chmod -R 777 $ASSET_DIR

CMD ["python"]

Using this Dockerfile as my image in the CDK code (in Java):

        String lambdaResourcesFolder = "src/main/resources/lambda/";

        PythonFunction pythonFunction = PythonFunction.Builder.create(this, "pythonFunction")
                .entry(lambdaResourcesFolder + "testFunction")
                .runtime(Runtime.PYTHON_3_10)
                .bundling(BundlingOptions.builder()
                        .image(DockerImage.fromBuild(lambdaResourcesFolder, DockerBuildOptions.builder()
                                .buildArgs(Map.of("FUNCTION_SRC", "testFunction"))
                                .build()))
                        .command(List.of("bash", "-c", "mv /assets/* /asset-output"))
                        .build())
                .build();

This allows the use of the cache layers for the install, but note the specified command in the BundlingOptions. From what I've observed, during cdk synth the Docker image run command is supplied with two additional volumes, one is to mount the local resource (supplied in .entry(...)) to /asset-input, and the other mounts the generated cdk.out/asset.{assetHash} directory to /asset-output. I can't find where this happens in the source, but supplying command(List.of("")) in the BundlingOptions gets the run command printed in the stack:

docker run --rm -u "501:20" -v "{source}:/asset-input:delegated" -v "/private/var/.../T/cdk.outjKtJiD/asset.{assetHash}:/asset-output:delegated" -w "/asset-input" cdk-db6fa0ee081c77999e6c779e847a608742d5cb8c92ebc17fdc1bcc7592808dd2

The problem is that if /asset-output is used in the Dockerfile, then the subsequent volume mounting will remove/shadow all the content in the container. This means we can't directly put assets into the /asset-output folder prior to synth. I don't know if this becomes possible if we somehow knew what the asset hash was going to be.

Hence, the Dockerfile needs to bundle into a separate directory and then mv the bundle into /asset-output during the docker run command. In my benchmark case, this has dropped the time taken from ~40s to ~10s, which is a decent improvement, but still a good chunk of time is taken to move the dependencies to the correct folder. My function only has botocore and PyYAML dependencies defined as well, so it's not a large dependency tree.

I'm wondering if there will be a similar issue with the solutions described by @m-radzikowski, notably with option 1 and option 2. I'm hopeful there's something I've overlooked here, as I'd like to be able to bundle the asset directly into /asset-output somehow to speed things up even more.

Curious if anyone has other findings.

[jbschooley]

A solution I have found is to use the SAM build image. It also only seems to work with requirements.txt, but a docker build is not triggered unless I make changes to the source.

Edit to add examples:

new python.PythonLayerVersion(this, 'MyLayer', {
    compatibleRuntimes: [
        lambda.Runtime.PYTHON_3_9,
    ],
    bundling: {
        image: lambda.Runtime.PYTHON_3_9.bundlingImage,
    }
})

Works well for x86. If you're running a lambda on ARM though, this image will still try to build for x86 so you have to specify the ARM image.

new python.PythonFunction(this, 'MyFunction', {
    architecture: lambda.Architecture.ARM_64,
    runtime: lambda.Runtime.PYTHON_3_11,
    bundling: {
        image: cdk.DockerImage.fromRegistry('public.ecr.aws/sam/build-python3.11:latest-arm64')
    }
})

Images can be found here https://gallery.ecr.aws/sam