‼️ NOTICE: aws:asset:original-path is non deterministic

[issei-m]

What is the problem?

Since 1.134.0, a Function generates the metadata entry aws:asset:original-path which includes the absolute path of the asset to be deployed. This information is system-dependent and therefore results in non-deterministic outputs.

Please add your +1 👍 to let us know you have encountered this

Workaround

Use --no-asset-metadata or set assetMetadata: false in your cdk.json.

---

Status: IN PROGRESS

Original Issue

Since 1.134.0, a Function is starting to generate the metadata entry aws:asset:original-path which relies on the full path of the asset to be deployed, but the path is definitely depending on the executor environment.

I guess the relating change would be this.

Let's say we defined a Function construct like following:

new lambda.Function(this, 'NiceFunction', {
  functionName: 'nice-function',
  runtime: lambda.Runtime.RUBY_2_7,
  code: lambda.Code.fromAsset(path.join(path.resolve(__dirname, '..'), 'lambda', 'nice_function')),
  handler: 'app.LambdaFunctions::NiceFunction.process',
  timeout: cdk.Duration.seconds(60),
  role: someRole
});

Then, if the source script path would be /path/to/my-cdk/lib/my-stack.ts then the aws:asset:original-path would be /path/to/my-cdk/lambda/nice_function.

But when another operator tries to deploy this stack, and if the source script path located on /foo/bar/my-cdk/lib/my-stack.ts then the aws:asset:original-path would claim it as/foo/bar/my-cdk/lambda/nice_function...
So we always face the differences when different operator hits cdk diff and even make a deployment when hitting cdk deploy.

It's very annoying...

Reproduction Steps

Create a stack containing a construct I described in "What is the problem?" section and hit cdk diff.

What did you expect to happen?

No differences appear unless we made changes in the stack.

What actually happened?

So we almost always face the "differences" when different operators hit cdk diff and even make a deployment when hitting cdk deploy.

CDK CLI Version

1.134.0

Framework Version

No response

Node.js Version

v16.6.1

OS

macOS 11.6.1 (20G224)

Language

Typescript

Language Version

No response

Other information

No response

[peterwoodworth]

Thanks for the thorough report @issei-m

Before this gets fixed up, you can override your function's metadata with the use of escape hatches

@rix0rrr pinging for visibility

[polothy]

Another workaround might be cdk synth --asset-metadata false then cdk deploy -a cdk.out {OTHER_ARGS_HERE}

[polothy]

Just FYI, if you deploy via CodeBuild with CodePipeline, this causes re-deployment because the paths change on every CodeBuild.

[eladb]

For workaround - --no-asset-metadata can be used directly with cdk deploy as well or can also be added to cdk.json

[eladb]

@moelasmar we need to revise #17591 so that original-path will be relative and not absolute.

[eladb]

@peterwoodworth I am tagging this as P0 since it's breaking customers.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[Dreamescaper]

I have the same issue even for lambdas, created by CDK itself (e.g. for S3 deployment).
Here's the diff:

[~] AWS::Lambda::LayerVersion doc-deployment/AwsCliLayer docdeploymentAwsCliLayerD5B93928 
 └─ [~] Metadata
     └─ [~] .aws:asset:original-path:
         ├─ [-] /tmp/jsii-kernel-5mkjIA/node_modules/aws-cdk-lib/lambda-layer-awscli/lib/layer.zip
         └─ [+] /tmp/jsii-kernel-XxMCrx/node_modules/aws-cdk-lib/lambda-layer-awscli/lib/layer.zip
[~] AWS::Lambda::Function Custom::CDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C81C01536 
 └─ [~] Metadata
     └─ [~] .aws:asset:original-path:
         ├─ [-] /tmp/jsii-kernel-5mkjIA/node_modules/aws-cdk-lib/aws-s3-deployment/lib/lambda
         └─ [+] /tmp/jsii-kernel-XxMCrx/node_modules/aws-cdk-lib/aws-s3-deployment/lib/lambda
[~] AWS::Lambda::LayerVersion fe-deployment/AwsCliLayer fedeploymentAwsCliLayer6A3D884B 
 └─ [~] Metadata
     └─ [~] .aws:asset:original-path:
         ├─ [-] /tmp/jsii-kernel-5mkjIA/node_modules/aws-cdk-lib/lambda-layer-awscli/lib/layer.zip
         └─ [+] /tmp/jsii-kernel-XxMCrx/node_modules/aws-cdk-lib/lambda-layer-awscli/lib/layer.zip

[eladb]

@Dreamescaper are you using v1.135.0?

[Dreamescaper]

@eladb
No, 2.1.0.

[eladb]

Okay that makes sense because we haven't propagated this change into the v2 branch yet.

[b3n3d1k7]

@eladb when will this fix be added to v2?

[b3n3d1k7]

@eladb when will this fix be added to v2?

Actually, after checking my problem in detail again, it seems like for me in v2 the problem was not the original-path metadata, but the pycache directory generated locally when installing the python dependencies for python lambda functions and layers using pip install -t in the DockerFile in Code.fromDockerBuild().
I'm not yet sure why, but when run within our gitlab pipeline the asset build stage does not generate the pycache directory.
This caused the asset hashes and thus S3-keys to change even though there where no code or dependency changes.
I fixed it by making sure pycache directories are removed in the asset target directory as the last step in the DockerFile:

FROM python:3.9-slim

WORKDIR /asset
COPY requirements.txt /asset

RUN pip install -r requirements.txt -t /asset

COPY . /asset
# remove __pycache__ from target directory to get consistent hashes for assets
RUN find /asset -type d -name __pycache__ -exec rm -r {} \+

So my problem was not really related to this issue.
However, I'll just leave this info here, so maybe it will help someone else with the same problem, who also stumbles across this while searching for a solution ;-)