(cli): Cross-account asset publishing broken in v1.135

[msessa]

What is the problem?

CDK asset publishing doesn't correctly assume the target asset publishing role in certain cases and fails to deploy S3 artifacts.

The issue seems to have surfaced with the release of cdk 1.135.0 but was possibly introduced in earlier versions.

Initial investigation suggests this line isn't passing the stack's asset publishing role and instead making API calls as the base role.

When v1.135 is used to deploy against an account bootstrapped with stack < v9, this code branch will trigger the above account() call which causes this.host.aborted to become true and this line to never be reached.

Reproduction Steps

• Have role in AccountA
• Bootstrap AccountB with stack v8 ( cdk <= 1.134.0 ), and --trust role in AccountA
• Create CDK application with some S3 asset (lambda functions or similar), point environment to AccountB
• Synth application with @aws-cdk/core:newStyleStackSynthesis=true
• npx aws-cdk@1.135.0 deploy --debug -vvvvv

What did you expect to happen?

CDK deploy assumes the asset publishing role in AccountB and correctly uploads artifacts

What actually happened?

fail: Need to perform AWS calls for account <AccountB>, but the current credentials are for <AccountA>

CDK CLI Version

1.135.0

Framework Version

No response

Node.js Version

v14.16.0

OS

MacOS

Language

Typescript

Language Version

No response

Other information

Complete deploy log:

CDK toolkit version: 1.135.0 (build 3825f59)
Command line arguments: {
  _: [ 'deploy' ],
  debug: true,
  v: 5,
  verbose: 5,
  'outputs-file': 'cdk-outputs.json',
  O: 'cdk-outputs.json',
  outputsFile: 'cdk-outputs.json',
  app: 'cdk.out/',
  a: 'cdk.out/',
  'require-approval': 'any-change',
  requireApproval: 'any-change',
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  all: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  ci: false,
  execute: true,
  force: false,
  f: false,
  parameters: [ {} ],
  'previous-parameters': true,
  previousParameters: true,
  '$0': '../.npm/_npx/69094/bin/cdk',
  STACKS: [ 'cdk-issue-debug' ],
  'S-t-a-c-k-s': [ 'cdk-issue-debug' ]
}
cdk.json: {
  "app": "npx ts-node -P tsconfig.json --prefer-ts-exts src/main.ts",
  "context": {
    "@aws-cdk/core:newStyleStackSynthesis": "true",
    "aws-cdk:enableDiffNoFail": true,
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
    "@aws-cdk/core:enableStackNameDuplicates": true,
    "@aws-cdk/core:stackRelativeExports": true,
    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
    "@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
    "@aws-cdk/aws-kms:defaultKeyPolicies": true,
    "@aws-cdk/aws-s3:grantWriteWithoutAcl": true,
    "@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount": true,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
    "@aws-cdk/aws-efs:defaultEncryptionAtRest": true,
    "@aws-cdk/aws-lambda:recognizeVersionProps": true,
    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true
  },
  "output": "cdk.out",
  "build": "npx projen bundle",
  "watch": {
    "include": [
      "src/**/*.ts",
      "test/**/*.ts"
    ],
    "exclude": [
      "README.md",
      "cdk*.json",
      "**/*.d.ts",
      "**/*.js",
      "tsconfig.json",
      "package*.json",
      "yarn.lock",
      "node_modules"
    ]
  },
  "//": "~~ Generated by projen. To modify, edit .projenrc.js and run \"npx projen\"."
}
merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  context: {
    '@aws-cdk/core:newStyleStackSynthesis': 'true',
    'aws-cdk:enableDiffNoFail': true,
    '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': true,
    '@aws-cdk/core:enableStackNameDuplicates': true,
    '@aws-cdk/core:stackRelativeExports': true,
    '@aws-cdk/aws-ecr-assets:dockerIgnoreSupport': true,
    '@aws-cdk/aws-secretsmanager:parseOwnedSecretName': true,
    '@aws-cdk/aws-kms:defaultKeyPolicies': true,
    '@aws-cdk/aws-s3:grantWriteWithoutAcl': true,
    '@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount': true,
    '@aws-cdk/aws-rds:lowercaseDbIdentifier': true,
    '@aws-cdk/aws-efs:defaultEncryptionAtRest': true,
    '@aws-cdk/aws-lambda:recognizeVersionProps': true,
    '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': true
  },
  app: 'cdk.out/',
  build: 'npx projen bundle',
  watch: {
    include: [ 'src/**/*.ts', 'test/**/*.ts' ],
    exclude: [
      'README.md',
      'cdk*.json',
      '**/*.d.ts',
      '**/*.js',
      'tsconfig.json',
      'package*.json',
      'yarn.lock',
      'node_modules'
    ]
  },
  '//': '~~ Generated by projen. To modify, edit .projenrc.js and run "npx projen".',
  debug: true,
  assetMetadata: true,
  requireApproval: 'any-change',
  toolkitBucket: {},
  staging: true,
  outputsFile: 'cdk-outputs.json',
  bundlingStacks: [ '*' ],
  lookups: true
}
Determining if we're on an EC2 instance.
Does not look like an EC2 instance.
Toolkit stack: CDKToolkit
Setting "CDK_DEFAULT_REGION" environment variable to ap-southeast-2
Resolving default credentials
Looking up default account ID from STS
[AWS sts 200 0.181s 0 retries] getCallerIdentity({})
Default account ID: AccountA
Setting "CDK_DEFAULT_ACCOUNT" environment variable to AccountA
context: {
  '@aws-cdk/core:newStyleStackSynthesis': 'true',
  'aws-cdk:enableDiffNoFail': true,
  '@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId': true,
  '@aws-cdk/core:enableStackNameDuplicates': true,
  '@aws-cdk/core:stackRelativeExports': true,
  '@aws-cdk/aws-ecr-assets:dockerIgnoreSupport': true,
  '@aws-cdk/aws-secretsmanager:parseOwnedSecretName': true,
  '@aws-cdk/aws-kms:defaultKeyPolicies': true,
  '@aws-cdk/aws-s3:grantWriteWithoutAcl': true,
  '@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount': true,
  '@aws-cdk/aws-rds:lowercaseDbIdentifier': true,
  '@aws-cdk/aws-efs:defaultEncryptionAtRest': true,
  '@aws-cdk/aws-lambda:recognizeVersionProps': true,
  '@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021': true,
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '*' ]
}
--app points to a cloud assembly, so we bypass synth
Reading existing template for stack cdk-issue-debug.
Retrieved account ID AccountA from disk cache
Assuming role 'arn:aws:iam::AccountB:role/cdk-hnb659fds-deploy-role-AccountB-ap-southeast-2'.
[AWS cloudformation 200 0.291s 0 retries] describeStacks({ StackName: 'cdk-issue-debug' })
[AWS cloudformation 200 0.175s 0 retries] getTemplate({ StackName: 'cdk-issue-debug', TemplateStage: 'Original' })
cdk-issue-debug: deploying...
Retrieved account ID AccountA from disk cache
Assuming role 'arn:aws:iam::AccountB:role/cdk-hnb659fds-deploy-role-AccountB-ap-southeast-2'.
Waiting for stack CDKToolkit to finish creating or updating...
[AWS cloudformation 200 0.212s 0 retries] describeStacks({ StackName: 'CDKToolkit' })
[AWS ssm 200 0.236s 0 retries] getParameter({ Name: '/cdk-bootstrap/hnb659fds/version' })
[0%] start: Publishing ea955e1dd13eac7a289683a4c42a8f95466ebb1e7002c2b524369eab874ebd6c:AccountB-ap-southeast-2
Retrieved account ID AccountA from disk cache
Assuming role 'arn:aws:iam::AccountB:role/cdk-hnb659fds-file-publishing-role-AccountB-ap-southeast-2'.
[0%] check: Check s3://cdk-hnb659fds-assets-AccountB-ap-southeast-2/ea955e1dd13eac7a289683a4c42a8f95466ebb1e7002c2b524369eab874ebd6c.zip
[AWS s3 200 0.183s 0 retries] getBucketLocation({ Bucket: 'cdk-hnb659fds-assets-AccountB-ap-southeast-2' })
[AWS s3 200 0.169s 0 retries] listObjectsV2({
  Bucket: 'cdk-hnb659fds-assets-AccountB-ap-southeast-2',
  Prefix: 'ea955e1dd13eac7a289683a4c42a8f95466ebb1e7002c2b524369eab874ebd6c.zip',
  MaxKeys: 1
})
[AWS s3 403 0.147s 0 retries] getBucketEncryption({ Bucket: 'cdk-hnb659fds-assets-AccountB-ap-southeast-2' })
Call failed: getBucketEncryption({"Bucket":"cdk-hnb659fds-assets-AccountB-ap-southeast-2"}) => Access Denied (code=AccessDenied)
[50%] fail: Need to perform AWS calls for account AccountB, but the current credentials are for AccountA
[50%] start: Publishing 4ed6eb06863732b08308385e96b4da247bb35b3e8b6aca611ee88573da8a870a:AccountB-ap-southeast-2
Retrieved account ID AccountA from disk cache
[50%] check: Check s3://cdk-hnb659fds-assets-AccountB-ap-southeast-2/4ed6eb06863732b08308385e96b4da247bb35b3e8b6aca611ee88573da8a870a.json
[AWS s3 200 0.165s 0 retries] listObjectsV2({
  Bucket: 'cdk-hnb659fds-assets-AccountB-ap-southeast-2',
  Prefix: '4ed6eb06863732b08308385e96b4da247bb35b3e8b6aca611ee88573da8a870a.json',
  MaxKeys: 1
})
[100%] fail: Need to perform AWS calls for account AccountB, but the current credentials are for AccountA

 ❌  cdk-issue-debug failed: Error: Failed to publish one or more assets. See the error messages above for more information.
    at Object.publishAssets (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at CloudFormationDeployments.publishStackAssets (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:305:7)
    at CloudFormationDeployments.deployStack (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:202:5)
    at CdkToolkit.deploy (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:194:24)
    at initCommandLine (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/bin/cdk.ts:267:9)
Failed to publish one or more assets. See the error messages above for more information.
Error: Failed to publish one or more assets. See the error messages above for more information.
    at Object.publishAssets (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at CloudFormationDeployments.publishStackAssets (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:305:7)
    at CloudFormationDeployments.deployStack (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:202:5)
    at CdkToolkit.deploy (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:194:24)
    at initCommandLine (/Users/msessa/.npm/_npx/69094/lib/node_modules/aws-cdk/bin/cdk.ts:267:9)

[relm923]

This appears to be happening on 2.1.0 as well

[kimisme9386]

I encounter the same problem. The v1.134.0 work fine ,but the v1.135.0 has the problem.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.