Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cancelling bundling of Lambda function causes next deployment to fail #33201

Open
1 task
rtpascual opened this issue Jan 27, 2025 · 3 comments
Open
1 task
Labels
@aws-cdk/core Related to core CDK functionality bug This issue is a bug. effort/medium Medium work item – several days of effort p2

Comments

@rtpascual
Copy link

Describe the bug

Cancelling cdk synth when a Lambda function is bundling can cause an empty bundling-temp-<hash> folder in cdk.out. This results in the next deployment of the Lambda to fail with InvalidParameterValueException: Uploaded file must be a non-empty zip.

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

CDK does not use existing bundling-temp folder if it is empty.

Current Behavior

CDK uses existing empty bundling-temp folder for deployment.

Reproduction Steps

  1. Setup Amplify (Gen 2) https://docs.amplify.aws/react/start/manual-installation/
  2. Setup a function https://docs.amplify.aws/react/build-a-backend/functions/set-up-function/
  3. Run npx ampx sandbox --debug
  4. Ctrl+C right when Bundling asset <Lambda>/Code/Stage... appears
  5. Notice empty bundling-temp folder in cdk.outImage
  6. Run npx ampx sandbox again

Possible Solution

No response

Additional Information/Context

We are seeing this for Amplify customers but I believe this can happen using only CDK for example in #18459 (comment).

CDK CLI Version

2.176.0

Framework Version

No response

Node.js Version

22.13.1

OS

Linux/Mac/Windows (More frequently on Windows from what we've seen)

Language

TypeScript

Language Version

No response

Other information

No response

@rtpascual rtpascual added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 27, 2025
@github-actions github-actions bot added the @aws-cdk/aws-lambda Related to AWS Lambda label Jan 27, 2025
@pahud pahud self-assigned this Jan 28, 2025
@pahud
Copy link
Contributor

pahud commented Jan 28, 2025

Hi

I was not able to create the bundling-tmp folder in amplify but I get your point here.

Also, #18459 (comment) is still relevant in this case.
I'll bring it up to the team and request their input here.

@pahud pahud added p1 and removed needs-triage This issue or PR still needs to be triaged. labels Jan 28, 2025
@pahud pahud removed their assignment Jan 28, 2025
@pahud pahud added the effort/medium Medium work item – several days of effort label Jan 28, 2025
@rix0rrr
Copy link
Contributor

rix0rrr commented Feb 18, 2025

This is done to avoid duplicate work, but apparently we can't distinguish between unfinished and finished previous work.

The directory was probably called *-tmp-* to signal that it was unfinished, and it should have been rename()d to something not containing that string, and only the existence of the non-tmp directory should count as "work already done previously".

Should be a matter of unconditionally deleting the temp folder before starting bundling.

@samson-keung samson-keung self-assigned this Feb 18, 2025
@samson-keung
Copy link
Contributor

samson-keung commented Feb 19, 2025

Some findings:

The following code creates the asset folders (e.g. cdk.out/asset.xxxxx:

private determineBundleDir(outdir: string, sourceHash?: string) {
if (sourceHash) {
return path.resolve(outdir, renderAssetFilename(sourceHash));
}
// When the asset hash isn't known in advance, bundler outputs to an
// intermediate directory named after the asset's cache key
return path.resolve(outdir, `bundling-temp-${this.cacheKey}`);
}

and:

const bundleDir = this.determineBundleDir(this.assetOutdir, assetHash);
this.bundle(bundling, bundleDir);

If the bundling is abruptly terminated (i.e. with Ctrl+C, the folder will not be populated with actual bundling output.

Then on the next bundling run, if the CDK bundling code sees the asset folder exists, it will skip re-bundling:

private bundle(options: BundlingOptions, bundleDir: string) {
if (fs.existsSync(bundleDir)) { return; }


What I did to reproduce the issue:

  1. I created a stack with a single resource:
new lambda.Function(this, 'LambdaFunction', {
      runtime: lambda.Runtime.NODEJS_LATEST,
      handler: 'index.handler',
      code: lambda.Code.fromAsset('src'), // put some random lambda code into `src/index.js`
    });
  1. run cdk synth
  2. delete the index.js in cdk.out/asset.xxxxxx to mimic the bundling logic being terminated and no bundled output got stored in the folder.
  3. Run cdk synth again and observe that the cdk.out/asset.xxxxxx folder is still empty.

A workaround for now would be to remove the empty asset folder and re-ran cdk synth so the bundling logic can be executed again and fully completes.

@samson-keung samson-keung removed their assignment Feb 19, 2025
@samson-keung samson-keung added @aws-cdk/core Related to core CDK functionality p2 and removed @aws-cdk/aws-lambda Related to AWS Lambda p1 labels Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/core Related to core CDK functionality bug This issue is a bug. effort/medium Medium work item – several days of effort p2
Projects
None yet
Development

No branches or pull requests

4 participants