(core): crossRegionReference only works for first destination region

[jsamuel1]

Describe the bug

When referencing a value from a source stack in two destination regions, only the first region is granted permissions on the CrossRegionExportWriterCustomerResourceProvider/Role, resulting in a stack deployment failure with insufficient permissions on ssm:ListTagsForResource in the second destination region.

Expected Behavior

Expected behaviour is for the IAM role to include each of the destination regions required.

Current Behavior

IAM Statement Changes:
Resource: "arn:aws:ssm:ap-southeast-1:082966487824::parameter/cdk/exports/*"
Effect: Allow
Action: ssm:DeleteParameters, ssm:GetParameters,ssm:ListTagsForResource,ssm:PutParameter
Principal: AWS:${Custom::CrossRegionExportWriterCustomResourceProvider/Role}

Error
8:48:55 am | CREATE_FAILED | Custom::CrossRegionExportWriter | ExportsWriterapsou...t2235C849A8E92266D
Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: User: arn:aws:sts::082966487824:assumed-role/sourceStack-
CustomCrossRegionExportWriterCustomRes-1L7WV0YX11H2H/sourceStack-CustomCrossRegionExportWriterCustomRes-XU6vlSVHMoXz is not authorized to perform: ssm:Lis
tTagsForResource on resource: arn:aws:ssm:ap-southeast-2:082966487824:parameter/cdk/exports/Region2Stack/sourceStackuseast1RefhostedZone861779BD20ED3C2C b
ecause no identity-based policy allows the ssm:ListTagsForResource action

Reproduction Steps

Sample Repo Code:

import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { Stack } from 'aws-cdk-lib';

const app = new cdk.App();
var sourceStack = new Stack(app, 'sourceStack', {  env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: 'us-east-1' } });
var hostedZone = new cdk.aws_route53.HostedZone(sourceStack, "hostedZone", { zoneName: 'test.local'});

var destRegion1Stack = new Stack(app, 'Region1Stack',  {  env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: 'ap-southeast-1' }, crossRegionReferences: true });
new cdk.aws_route53.TxtRecord(destRegion1Stack, "FirstARecord", { zone: hostedZone, recordName: 'ap-southeast-1', values: ['ap-southeast-1'] });

var destRegion2Stack = new Stack(app, 'Region2Stack',  {  env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: 'ap-southeast-2' }, crossRegionReferences: true });
new cdk.aws_route53.TxtRecord(destRegion2Stack, "SecondARecord", { zone: hostedZone, recordName: 'ap-southeast-2', values: ['ap-southeast-2'] });

Possible Solution

the IAM role should include permissions to all required regions.

Additional Information/Context

No response

CDK CLI Version

2.67.0 (build b6f7f39)

Framework Version

No response

Node.js Version

v18.13.0

OS

MacOs/Linux/Agnostic

Language

Typescript

Language Version

Typescript (v10.9.1)

Other information

No response

[pahud]

According to the description in #22008

Currently this will create a single ExportsWriter per region, but we could potentially update this to just use a single ExportsWriter which can write exports to a list of regions.

I believe it has not supported the write exports to multiple consuming regions yet. I can reproduce this in my account and is making it p1 feature request for now.

[mrpackethead]

I am also finding permission errors with the CrossRegionWriter.

If i update the attributes of a stack, and redeploy, I get this error.

❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named BuildCoreNetwork-CrowdstrikeEndpoints failed to deploy: UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [ExportsWriterapsoutheast2235C849A8E92266D]. ): Received response status [FAILED] from custom resource. Message returned: Error: Exports cannot be updated: 

    at throwIfAnyInUse (/var/task/index.js:4:10)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async handler (/var/task/index.js:3:54)
    at async Runtime.handler (/var/task/__entrypoint__.js:1:932) (RequestId: cfbddadf-d1b6-40a9-bb18-a92eddbc9c3e), Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: User: arn:aws:sts::108967544837:assumed-role/BuildCoreNetwork-Crowdstr-CustomCrossRegionExportW-7A0BHK7VO7QU/BuildCoreNetwork-Crowdstr-CustomCrossRegionExportW-kG0ITAeZRmZA is not authorized to perform: ssm:DeleteParameters on resource: arn:aws:ssm:ap-southeast-2:1xxxxxxxxxxx7:* because no identity-based policy allows the ssm:DeleteParameters action
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) (RequestId: e9de1172-5ff5-4c63-b46a-9407d574a4b1)
    at deployStacks (/home/ec2-user/.nvm/versions/node/v16.15.1/lib/node_modules/aws-cdk/lib/index.js:330:130488)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async CdkToolkit.deploy (/home/ec2-user/.nvm/versions/node/v16.15.1/lib/node_modules/aws-cdk/lib/index.js:330:146712)
    at async exec4 (/home/ec2-user/.nvm/versions/node/v16.15.1/lib/node_modules/aws-cdk/lib/index.js:385:51296)

At present the only way i've been able to resolve this, is to manually delete the SSM parameters, delete the stack and redeploy. Forutantly this was not a production stack so i had the ability to do that...

[joshmurrayeu]

As per the original issue, is there any workaround to manually add/create another CrossRegionExportWriter for the region that isn't authorised? I'm currently trying to get an application across three regions (eu-west-1, eu-west-2 and eu-north-1) and get a similar issue:

{corssRegionExportWriterRoleArn} is not authorized to perform: ssm:ListTagsForResource on resource: {ssmParameterArn}

[nomike]

I'm affected by this as well.

[LikeFury]

Is there an update on this issue? Anyway we can float this to the top?

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.