aws-s3-deployment timeout on larger dataset

[helzich]

🐛 Bug Report

What is the problem?

Deployment to an s3 bucket with the s3-deployment module times out on a moderately sized dataset containing 6 zip files with a combined size of 271MB. Error messages differ, I got these:
Failed to create resource. Command '['python3', '/var/task/aws', 's3', 'cp', 's3://cdktoolkit-stagingbucket-mhglm1j9x6gh/assets/[...]f5.zip', '/tmp/tmp8p62yjtl/archive.zip']' died with <Signals.SIGKILL: 9>
and:
Custom Resource failed to stabilize in expected time

Synching the same contents with the AWS CLI works in about 4min.

Running with the same code as quoted below, but small sized test files, all is good, too and the files appear in the destination bucket.

I did not see any timeout/data size warnings in the docs, other than a lambda execution limit of 15min, but I am not sure this is applicable here.

Reproduction Steps

I used the Python code below with AWS CDK 1.8.0 (build 5244f97).

		# Get the existing bukcet
		layer_bucket = s3.Bucket.from_bucket_name(
			self,
			id = layer_bucket_name,
			bucket_name = layer_bucket_name
		)

		# Upload from a local directory
		s3deploy.BucketDeployment(
			self,
			id = layer_bucket_name + 'Deployment',
			source = s3deploy.Source.asset('./builds/dev/lambda_layers/'),
			destination_bucket = layer_bucket,
		)

Verbose Log

See error messages above.

Environment

• CDK CLI Version: 1.8.0 (build 5244f97)
• Module Version: 1.8.0
• OS: Amazon Linux in a Docker container
• Language: Python

Other information

n/a

[engineal]

I just ran into this issue as well, in typescript. The s3 sync command tries to get the asset from the staging bucket, but if the asset is larger than the deployment lambda's memorySize of 128 MB, then the lambda hits the memory limit and the s3 sync command pauses.

I used the following typescript code to increase the memory size:
(staticBucketContents.node.findChild('CustomResourceHandler').lambdaFunction.node.defaultChild as CfnFunction).memorySize = 512;
where staticBucketContents is my BucketDeployment. Even though typescript throws an error while building this, it still generates the template with the overridden memory size. The increased memory size fixed the issue.

It would be nice to at least be able to properly set the memorySize for the BucketDeployment's CustomResourceHandler, or better yet, set the memorySize based off the size of the asset.

[nmussy]

I think this could be more simply remedied by buffering this process:

aws-cdk/packages/aws-cdk/lib/assets.ts

Lines 123 to 131 in e1a5739

	const data = await fs.readFile(filePath);
	const s3KeyPrefix = `assets/${asset.id}/`;
	const { filename, key, changed, hash } = await toolkitInfo.uploadIfChanged(data, {
	s3KeyPrefix,
	s3KeySuffix: path.extname(filePath),
	contentType
	});

Thoughts?

[eladb]

Yes, at the minimum we should allow you to specify the lambda memory limit.

@nmussy not sure I understand the buffering idea.

[nmussy]

The contents of the files are wholly loaded in memory and uploaded to S3. Instead of increasing the RAM to whatever the asset size will be (and then some for overhead), we could do a multi-part S3 upload, loading the file pieces in memory as needed.

[eladb]

Implementation of memory limit configuration: #4204

[mark-r-m]

We've also had this - and the zip file of our static react site to go to S3 is only 138 mb.