(aws-ecr-assets): DockerImageAsset fails to upload to ECR as the repo is not created

[cdimascio]

When creating a DockerImageAsset like below...

const asset = new aws_ecr_assets.DockerImageAsset(this, "MyAsset", {
  directory: path.join(SRC_DIR),
});

The image is created successfully at build time, however it fails to upload to an ECR repository. When the task enters a "stopped" status, the following error is reported.

Status reason CannotPullContainerError: Error response from daemon: 
repository XXXX.dkr.ecr.us-west-2.amazonaws.com/cdk-<xxxx>-container-assets-<account>-<region> 
not found: name unknown: The repository with name 'cdk-<xxxx>-container-assets-<account>'

It fails because the repo it's trying to publish to does not exist. It was never created.

What did you expect to happen?

The ECR repository referenced in the error message e.g XXXX.dkr.ecr.us-west-2.amazonaws.com/cdk-<xxxx>-container-assets-<account>-<region> would have been be created by CDK and subsequently the image would successfully upload to that ECR repo.

What actually happened?

The upload fails since the ECR repo was not created.

Also, during stack creation the Task status continuously iterates from pending, provisioning, to stopped. Each time it reaches status, stopped, it reports the the error above, then tries again. Ulitmately, the task goes intoa loop and cannot successfully start (i waited ~40+ minutes. it's possible that it would eventually give up).

Workaround

1. Delete the stack
2. Manually create an ECR repo with the name, cdk-<xxxx>-container-assets-<account>-<region> as specified in the error message.
3. Create the stack

For my use case, I refrenced the DockerImageAsset above in the taskImageOptions on ApplicationLoadBalancedFargateService, like so...

new ecs_patterns.ApplicationLoadBalancedFargateService(this, 'MyTask', {
      vpc,
      publicLoadBalancer: true,
      redirectHTTP: true,
      protocol: elb.ApplicationProtocol.HTTPS,
      domainName,
      domainZone,
      cpu: 256,
      desiredCount: 2,
      listenerPort: 443,
      taskImageOptions: {
        taskRole: apiServiceRole,
        containerPort: 3000,
        enableLogging: true,
        image: ecs.ContainerImage.fromDockerImageAsset(asset), // <- referenced asset here
        containerName: 'api-server',
      },
      loadBalancer,
      memoryLimitMiB: 512,
    });

Environment

• **CDK CLI Version :1.76.0 (build c207717)
• Framework Version:: 1.77.0
• Node.js Version:: v14.9.0
• OS : MacOS
• Language (Version):: TypeScript (3.9.7)

This is 🐛 Bug Report

[sbinyang]

Hi, I’m also seeing a similar issue while trying to create a lambda function using the new DockerImageFunction:

const fn = new lambda.DockerImageFunction(this, "AssetFunction", {
      code: lambda.DockerImageCode.fromImageAsset(path_to_src),
});

The image also created successfully at build, but failed to upload to an ECR repo with the following error. I ran it again after bootstrapping the account and it did not fix the issue.



Failure: Error: No ECR repository named 'cdk-{xxxx}-container-assets-{account}-{‘region} in account {account}. Is this account bootstrapped?

[olivboom]

+1 I'm also experiencing this issue and would appreciate a native fix within CDK instead of the given workaround.

[jmjava]

I am seeing this in both DockerImageFunction and DockerImageAsset. Are there any workarounds? I need to publish an image in my stack.

[eladb]

@cdimascio did you bootstrap your environment using cdk bootstrap with @aws-cdk/core:newStyleStackSynthesis context set to true?

[ahammond]

Reproduced this using CDK v2. We have never used any CDK v1 in this account.

[awhillas]

Anything happening with this? I'm still getting the massage after bootstrapping multiple times :(
When i bootstrap i get this message:

Trusted accounts for deployment: (none)
Trusted accounts for lookup: (none)
Using default execution policy of 'arn:aws:iam::aws:policy/AdministratorAccess'. Pass '--cloudformation-execution-policies' to customize.

Should i supply --cloudformation-execution-policies? What are the options here? SHould the "Trusted accounts" not be None?

[DJ-Icebear]

Our team is also having this issue. And if I understand correctly there no way to specify to which repo a built image is published when using ecs.ContainerImage.fromAsset(...)?

[vzolin]

I had the same issue, my workaround was deleting CDKToolkit from CloudFormation and it's S3 bucket and then bootstraping again.

[corymhall]

This should not be an issue currently as long as you have bootstrapped your account. We have other issues tracking adding support for pushing ECR images to a user non-bootstrap ECR repository (see #12597).

I'll leave this open for a little bit to allow for responses in case this is still an issue for anyone.

[ibrahimcesar]

+1 on @vzolin. I faced this issue, but while debbuging a problem, I manually deleted the ECR repo created on cdk bootstrap. Since I was not able to force a new bootstrap, I delete the previous CDKToolkit stack in CloudFormation.

I only find in the docs how to customize the bootstrap not force a override or even a full deletion of the stack and all resources since, for this work, I also had to empty and delete the S3 bucket created by cdk bootstrap.

[corymhall]

@ibrahimcesar before deleting the CDKToolkit stack or manually deleting the ECR repository did you try re-bootstrapping with cdk bootstrap? If so did you receive any error messages?

There is an option to run bootstrap with --force (e.g. cdk bootstrap --force) in order to force a re-bootstrap. This can sometimes get you passed some errors.

[ibrahimcesar]

@corymhall I only tried bootstrap without force. I didn't get an error per se, just a notice that my account was already bootstrapped. Next time I'll try with the --force flag. Thanks!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[abury]

I've come across this issue after starting a second CDK project. We bootstrapped in the first project, but that project didn't require a default ECR, so It wasn't created. Now trying to build this one with a new lambda.DockerImageFunction and seeing the error fail: No ECR repository named ....

Tried bootstrapping and force bootstrapping, neither have succeeded in creating the default ECR.. Nuking the original CDKToolkit CF Stack and s3 bucket and re-bootstrapping resolved the issue.

[willredington]

+1