Unable to push to ContainerAssetsRepository when using a trusted account to deploy

[joepjoosten]

Describe the bug

I've boostrapped my account with the following settings:

cdk bootstrap --trust <trusted-accountid> --trust-for-lookup <trusted-accountid> --cloudformation-execution-policies 'arn:aws:iam::aws:policy/AdministratorAccess' aws://<accountid>/eu-central-1

I get this error when i'm logged in with my trusted account to deploy a stack

fail: docker push <accountid>.dkr.ecr.eu-central-1.amazonaws.com/cdk-hnb659fds-container-assets-<accountid>-eu-central-1:<hash> 
exited with error code 1: denied: User: arn:aws:sts::<trusted-accountid>:assumed-role/<sso>/<username> is not authorized 
to perform: ecr:InitiateLayerUpload on resource: arn:aws:ecr:eu-central-1:<accountid>:repository/cdk-hnb659fds-container-assets-<accountid>-eu-central-1 
because no resource-based policy allows the ecr:InitiateLayerUpload action

Expected Behavior

I should be able to push an docker image asset to the ContainerAssetsRepository from a trusted account.

Current Behavior

Not supported, error shown

Reproduction Steps

1. Bootstrap

cdk bootstrap --trust <trusted-accountid> --trust-for-lookup <trusted-accountid> --cloudformation-execution-policies 'arn:aws:iam::aws:policy/AdministratorAccess' aws://<accountid>/eu-central-1

2. Stack with DockerImageAsset from aws-cdk-lib/aws-ecr-assets
3. Login with trusted account
4. cdk deploy

Possible Solution

Can be fixed by adding the trusted account(s) to the ContainerAssetsRepository policy

Additional Information/Context

No response

CDK CLI Version

2.143.0

Framework Version

No response

Node.js Version

20

OS

macos

Language

TypeScript

Language Version

TypeScript (5.4.5)

Other information

No response

[pahud]

If you bootstrap account A with trust on account B, this means:

1. The bootstrap would create 5 roles that allows account B root principal to assume.
2. For ecr image assets, your account A should have this role
   cdk-hnb659fds-image-publishing-role-ACCOUNT_A_ID-REGION and this role should have a trust policy with the account root principal of account B in the policy statement.
3. If you run cdk synth or cdk deploy from an identity of Account B, this identity would need the sts:AssumeRole on the cdk-hnb659fds-image-publishing-role-ACCOUNT_A_ID-REGION role.
4. the image assets should be pushed to ECR with cdk-hnb659fds-image-publishing-role-ACCOUNT_A_ID-REGION role.
5. cdk-hnb659fds-image-publishing-role-ACCOUNT_A_ID-REGION should have all required permissions, in my case:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ecr:PutImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:BatchCheckLayerAvailability",
                "ecr:DescribeRepositories",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:GetDownloadUrlForLayer"
            ],
            "Resource": "arn:aws:ecr:us-east-1:ACCOUNT_A:repository/cdk-hnb659fds-container-assets-ACCOUNT_A-us-east-1",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ecr:GetAuthorizationToken"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Can you verify above and see if there's anything missing?

[pahud]

Looking at your error message, looks like you are trying to push to the staging ecr repo with your current iam principal, instead of the cdk-hnb659fds-image-publishing-role-ACCOUNT_A_ID-REGION role. Can you explain how you push the images? Did you run cdk synth or cdk deploy for that?

[joepjoosten]

In my use case i was doing a cdk deploy on account A, from account B. The DockerImageAsset being build has in the Dockerfile a from clause from account C

FROM <account-c>.dkr.ecr.eu-central-1.amazonaws.com/<image-name>:<tag>

I had to add some docker config settings to get it to build the docker image locally:

{
  "credHelpers": {
    "<account-a>.dkr.ecr.eu-central-1.amazonaws.com": "ecr-login",
    "<account-b>.dkr.ecr.eu-central-1.amazonaws.com": "ecr-login",
    "<account-c>.dkr.ecr.eu-central-1.amazonaws.com": "ecr-login"
  }
}

Does this maybe make the cdk think that it's logged in with the wrong credentials, and using the wrong credentials to push?

[joepjoosten]

I can confirm that the role with the permission you showed is there in account A, and it has a trust relationship with the root account of account B. So the bootstrap step 1 through 5 is executed as you showed.