[pipelines] 1.67.0 breaks custom asset bundling when using multiple pipelines

[danielfrankcom]

After updating to CDK 1.67.0, my custom asset bundling logic no longer works while using multiple pipelines. This bundling logic uses the BundlingOptions construct to move the lambda function handler to a nested folder structure. When executing cdk synth on the new CDK version, only the first pipeline defined in app.py uses the bundling logic, and the other copies the code from the path without modifying it.

If I revert back to 1.66.0 I no longer see this behavior, with no other code changes. Both pipelines produce the correct artifacts using the older CDK version.

Reproduction Steps

I have stripped down my code to provide a minimal example, and have posted it here.

Running cdk synth against this code produces 2 assemblies in cdk.out, where each assembly has an asset representing the lambda function. The cdk synth output can also be seen by running the pipeline on AWS, although it is easier to test locally.

What did you expect to happen?

I expected both assets in the cdk.out directory to contain the bundled directory structures.

Both assets should contain a directory structure that looks like the following:

asset.1234
  - entry.py
  - lib
    - function
      - function.py

What actually happened?

Only the first pipeline declared in app.py has the bundled directory structure.

The second asset is bundled using the default settings, which ignore the custom bundling logic.

asset.1234
  - function.py

Environment

• CLI Version : 1.67.0 (build 2b4dd71)
• Framework Version: 1.67.0
• Node.js Version: v12.18.4
• OS : WSL2 in Windows 10 (same behavior occurs in CodeBuild)
• Language (Version): Python 3.8

---

This is 🐛 Bug Report

[jogold]

@danielfrankcom can you share the exact folder structure/content of your cdk.out after running synth here?

[danielfrankcom]

@jogold absolutely!

To generate this, I deleted the cdk.out folder entirely, and ran cdk synth on the project that I linked above. This is the file tree for version 1.66.0, which is the way that I expect it to look.

cdk.out
├── Develop1.assets.json
├── Develop1.template.json
├── Develop2.assets.json
├── Develop2.template.json
├── assembly-Develop1-DevelopLayer
│   ├── Develop1DevelopLayerLayer1283E4BED.assets.json
│   ├── Develop1DevelopLayerLayer1283E4BED.template.json
│   ├── asset.88d670728dd9800639080b11a0a9600df1d1f0c270a30a21d0c684b503fe1b26
│   │   ├── entry.py
│   │   └── lib
│   │       └── function
│   │           ├── __pycache__
│   │           │   └── function.cpython-38.pyc
│   │           └── function.py
│   ├── cdk.out
│   └── manifest.json
├── assembly-Develop2-DevelopLayer
│   ├── Develop2DevelopLayerLayer17829D95B.assets.json
│   ├── Develop2DevelopLayerLayer17829D95B.template.json
│   ├── asset.88d670728dd9800639080b11a0a9600df1d1f0c270a30a21d0c684b503fe1b26
│   │   ├── entry.py
│   │   └── lib
│   │       └── function
│   │           ├── __pycache__
│   │           │   └── function.cpython-38.pyc
│   │           └── function.py
│   ├── cdk.out
│   └── manifest.json
├── cdk.out
├── manifest.json
└── tree.json

After generating this tree I changed the CDK dependencies in requirements.txt to 1.67.0, and executed pip install -r requirements.txt to update the installed dependencies.

I deleted the cdk.out folder again, and ran cdk synth to generate this tree for 1.67.0:

cdk.out
├── Develop1.assets.json
├── Develop1.template.json
├── Develop2.assets.json
├── Develop2.template.json
├── assembly-Develop1-DevelopLayer
│   ├── Develop1DevelopLayerLayer1283E4BED.assets.json
│   ├── Develop1DevelopLayerLayer1283E4BED.template.json
│   ├── asset.88d670728dd9800639080b11a0a9600df1d1f0c270a30a21d0c684b503fe1b26
│   │   ├── entry.py
│   │   └── lib
│   │       └── function
│   │           ├── __pycache__
│   │           │   └── function.cpython-38.pyc
│   │           └── function.py
│   ├── cdk.out
│   └── manifest.json
├── assembly-Develop2-DevelopLayer
│   ├── Develop2DevelopLayerLayer17829D95B.assets.json
│   ├── Develop2DevelopLayerLayer17829D95B.template.json
│   ├── asset.88d670728dd9800639080b11a0a9600df1d1f0c270a30a21d0c684b503fe1b26
│   │   ├── __pycache__
│   │   │   └── function.cpython-38.pyc
│   │   └── function.py
│   ├── cdk.out
│   └── manifest.json
├── cdk.out
├── manifest.json
└── tree.json

You can see in the second tree that the second asset only contains the function.py file, and is not modified by the bundling logic that is supposed to be creating the lib/function directory structure and adding the entry.py file.

[jogold]

@rix0rrr this is due to #10478: the cache logic fails here because Stages do not share the same outdir (but works with multiple Stacks in the same app). But this means that identical assets used in multiple Stages are copied/bundled as many times as there are Stages, what do you suggest here?

[hoegertn]

@rix0rrr this is also what I was talking about on Wednesday with building assets multiple times

[rix0rrr]

Yep, I figured as much.

Feels like we can do two things:

• Add more deduplication logic in the asset bundling so that it will know to only do bundling (or whatever the new operation is, I didn't follow this too closely) once. We already have the cacheKey which depends only on sourcePath, so that should match for multiple assets, feels like.
• Introducing the concept of an asset outdir that will be shared between all (nested) assemblies in the root assembly, then all old logic applies again.

I feel like the 2nd one is better.

Thoughts?

[hoegertn]

Seconds options sounds great.

[jogold]

I feel like the 2nd one is better.

Agree

[jogold]

Not sure we need an extra directory for this, leaving all the assets in the top level outdir is fine and is the behavior when there are no Stages in an App. @rix0rrr is there a simple way to get to the top level outdir in AssetStaging? Then it's fixed.

A note on the origin of this issue: caching works but the second time we come across the asset there's no bundleDir here

aws-cdk/packages/@aws-cdk/core/lib/asset-staging.ts

Lines 190 to 191 in ddff37c

	// Staging the bundling asset.
	if (this.bundleDir) {

because bundling was skipped. But since the targetPath is different (inside a nested outdir) the asset is not considered as already staged:

aws-cdk/packages/@aws-cdk/core/lib/asset-staging.ts

Lines 205 to 208 in ddff37c

	// Already staged
	if (fs.existsSync(targetPath)) {
	return;
	}

so it falls back to a normal asset copy:

aws-cdk/packages/@aws-cdk/core/lib/asset-staging.ts

Lines 210 to 219 in ddff37c

	// Copy file/directory to staging directory
	const stat = fs.statSync(this.sourcePath);
	if (stat.isFile()) {
	fs.copyFileSync(this.sourcePath, targetPath);
	} else if (stat.isDirectory()) {
	fs.mkdirSync(targetPath);
	FileSystem.copyDirectory(this.sourcePath, targetPath, this.fingerprintOptions);
	} else {
	throw new Error(`Unknown file type: ${this.sourcePath}`);
	}

[rix0rrr]

because bundling was skipped. But since the targetPath is different (inside a nested outdir) the asset is not considered as already staged:

Is it fair to consider this case as a logic bug? Seems that the fallback behavior here is an accident and not actually intended, right?

[jogold]

Seems that the fallback behavior here is an accident and not actually intended, right?

correct

[jogold]

Seems that the fallback behavior here is an accident and not actually intended, right?

correct

In fact, the code should properly throw if bundling is used and we have a cache hit and we cannot find the already staged asset.

[rix0rrr]

I was thinking that too. Kk, good that we agree :).

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.