servicecatalog: Product containing BucketDeployment: AWS CLI Layer asset stored in wrong bucket

[padaszewski]

Describe the bug

The product is made with the fromProductStack functionality and it contains the BucketDeployment Resource.
When trying to launch this product in the target account it fails to deploy because one part of the error:

Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist.

The object is correctly referenced in the template, however the assets bucket does not contain the corresponding .zip file.
After investigation the file was found in the standard cdk bootstrapping bucket (cdk-assets).
When the .zip file is copied to the product assets bucket then the product launches correctly.
This asset contains the AWSCliLayer files needed for the BucketDeployment.

Expected Behavior

The required assets is stored in the product stack assets bucket and the product launches.

Current Behavior

The required asset is stored in the cdk bootstrapping bucket and while launching of the product the product cannot find the asset in the assets bucket, because obviously this file is missing there.

Reproduction Steps

Full minimal example:

import * as cdk from 'aws-cdk-lib'
import { aws_s3 } from 'aws-cdk-lib'
import { Construct } from 'constructs'
import { CloudFormationTemplate, Portfolio } from 'aws-cdk-lib/aws-servicecatalog'
import { BlockPublicAccess, Bucket } from 'aws-cdk-lib/aws-s3'
import { BucketDeployment } from 'aws-cdk-lib/aws-s3-deployment'

export class ServiceCatalogPortfolioStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)

    const portfolio = new Portfolio(this, 'Portfolio', {
      displayName: 'MyPortfolio',
      providerName: 'MyTeam',
    })

    const assetsBucket = new Bucket(this, 'AssetsBucket', {
      bucketName: 'asset-bucket-my-fancy-unique-name',
    })

    const product = new cdk.aws_servicecatalog.CloudFormationProduct(this, 'DemoAppProduct', {
      productName: 'DemoAppProduct',
      owner: 'MrBug',
      productVersions: [
        {
          productVersionName: 'v2',
          cloudFormationTemplate: CloudFormationTemplate.fromProductStack(
            new AssetBucketDemoProductStack(this, 'AssetBucketDemoStack', { assetBucket: assetsBucket }),
          ),
        },
      ],
    })

    portfolio.addProduct(product)
    portfolio.shareWithAccount('targetAccount')
  }
}

class AssetBucketDemoProductStack extends cdk.aws_servicecatalog.ProductStack {
  constructor(scope: Construct, id: string, props?: cdk.aws_servicecatalog.ProductStackProps) {
    super(scope, id, props)

    const frontendBucket = new aws_s3.Bucket(this, 'FrontendDistributionBucket', {
      publicReadAccess: false,
      blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
    })

    const bucketDeployment = new BucketDeployment(this, 'FrontendBucketDeployment', {
      sources: [cdk.aws_s3_deployment.Source.asset('./dist')],
      destinationBucket: frontendBucket,
    })
  }
}

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.80.0

Framework Version

No response

Node.js Version

18.15.0

OS

Windows

Language

Typescript

Language Version

No response

Other information

No response

[padaszewski]

@wanjacki Could You take a look on this? Any idea for a workaround without manual copying?

[wanjacki]

You defined
const assetsBucket = new Bucket(this, 'AssetsBucket', { bucketName: 'asset-bucket-my-fancy-unique-name', })
as the bucket you are copying to.

You don't need any of that code in AssetBucketDemoProductStack. That code is creating a different bucket and using bucketdeployment to copy it to that other bucket.

All you have to do is define the asset in AssetBucketDemoProductStack:
https://docs.aws.amazon.com/cdk/v2/guide/assets.html
and ProductStack should take care of the rest to copy it over to that first bucket.

[JustusMarkert]

Hi @wanjacki,
thanks for your reply. In the full scenario we are trying to launch our application (consisting of Lambdas, RestApis and an Angular Frontend) via the Service Catalog. Its totally correct that three buckets are involved:

• First bucket is the bootstrapped CdkAssetsBucket (where all assets concerning pipelines and portfolio take place).
• Second Bucket is the AssetBucket wich is required by 'CloudFormationTemplate.fromProductStack' to store all Assets that are referenced in the resulting ProductStack template. We also grant read access for several AccountPrincipals here.
• The third bucket is part of our application and should be created in the target account on product launch. This bucket hosts the Angular Frontend resources wich are accessed via an proxy api (in combination with cognito user pool).

Also I want to point out, that the problem isn't the Asset containing the Anglular files. They are correctly stored in the second bucket (AssetBucket) and retrieved on product launch.
The problem we are facing is the retrival of the asset for the AwsCliLayer wich is created by the BucketDeployment in the background. The zip-file for this specific layer is stored in the first bucket, our bootstrap bucket. Interestingly the template contains the correct name of the file and also points to the correct bucket (AssetBucket). This causes the NoSuchKey S3 Error on product launch.

However - manually moving the zip-file (AwsCliLayer) from the boostrap bucket (1) to the correct assetBucket (2) makes the product launchable. All other assets are stored correctly - the only problem is this specify zip-file.

Thanks in advance
Justus

[wanjacki]

I see, so when an asset is detected to be created directly in ProductStack it will call:
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/aws-servicecatalog/lib/private/product-stack-synthesizer.ts#L21
which will create a BucketDeployment to copy over that asset to your specified AssetBucket.

So I am assuming the assets for AwsCliLayer is created outside of your ProductStack or indirectly such as a result of that FrontendBucketDeployment? Either way there's no way for the ProductStack to detect this AwsCliLayer asset unless it was directly specified.
Then likely our ProductStack never triggers that addFileAsset call to copy it over.

Not sure what the solution is, but maybe you can make an additional BucketDeployment yourself to copy it over yourself?

[padaszewski]

Hi @wanjacki
The AwsCliLayer is created as a part of BucketDeployment, which resides in the ProductStack. Every Asset created in the ProductStack should be stored in the specified assets bucket, but this is exactlz the exception for the AwsCliLayer.
This layer comes from here.
This makes the BucketDeployment resource unusable with the ProductStack, because the resulting products cannot be launched. The question is why the ProductStackSynthesizer does not recognizes this Layer as Asset needed for the ProductStack.

[wanjacki]

Hi @padaszewski,
The resources inside ProductStack don't get deployed until the product is launched, meaning your BucketDeployment won't run until you launch the Product and you can't launch the Product due to the asset being missing. I think the current ProductStack cannot detect any nested assets, I've seen others have similar issues.

Sorry unfortunately I'm no longer with AWS/Service Catalog so my ability to help is limited, but best of luck in figuring this out :/

[D-McK]

Would just like to bump this thread and say we are facing almost the exact same problem on our team (but using Python CDK). We have an assets bucket initialised and trying to utilise the bucket_deployment() method to serve a React based front-end stored locally.

When launching the Service Catalog product that is created, we are getting the same "NoSuchKey" error from the AwsCliLayer component of the bucket deployment method.

[ArneScherhag]

Does PR #26311 fix this issue?

It addresses #24317, where nested stack for AWS service Catalog failed in cdk synth.

[padaszewski]

Hi @ArneScherhag
The new version of the CDK with this fix needs to be released to test this on our end. However, the person responsible for this is currently on vacation. We will keep you updated here (probably around mid-August).

[D-McK]

Hi there, thought I would chime in since we have had chance to try the latest cdk version (v2.88.0) with our solution. Unfortunately we still receive the same aformentioned error of NoSuchKey when trying to access the build folder zip from the assets bucket in the bucket deployment