(core): Setting Bucket policy from cross-region stack causes cyclic reference

[JonWallsten]

Describe the bug

I'm trying to set a bucket policy from a cross-region stack and can't deploy due to the following error message:

We have created buckets in eu-west-1 and the Cloudfront is created in us-east-1 since it's a global resource.
The buckets policy needs the Cloudfront Distribution ID so it needs to be set after the creation of the bucket.

Expected Behavior

I expect to be able to set the policy since I use crossRegionReference: true

Current Behavior

It fails with the this error:
Error: 'InfraRegionalStack' depends on 'InfraGlobalApp' (InfraRegionalStack -> InfraGlobalApp/CloudFrontDistribution/CloudFrontDistribution/Resource.Ref). Adding this dependency (InfraGlobalApp -> InfraRegionalStack/S3Hosting/HostingBucket/Resource.RegionalDomainName) would create a cyclic reference.

This is the code used:

// Add bucket policy to make sure only CLoudfront can access it
const policyStatement = (bucketArn: string) =>
    new PolicyStatement({
        sid: 'AllowCloudFrontServicePrincipalReadOnly',
        effect: Effect.ALLOW,
        principals: [new ServicePrincipal('cloudfront.amazonaws.com')],
        actions: ['s3:GetObject'],
        resources: [`${bucketArn}/*`],
        conditions: {
            StringEquals: {
                'AWS:SourceArn': `arn:aws:cloudfront::${Stack.of(this).account}:distribution/${
                    this.distribution.distributionId
                }`
            }
        }
    });

// Add policies
props.hostingBucket.addToResourcePolicy(policyStatement(props.hostingBucket.bucketArn));

Another solution I tried was using the buckets properties to lookup a bucket and then use a different solution that I got to work before in another case:

// Add bucket policy to make sure only CLoudfront can access it
        const policyStatement = (bucketArn: string) =>
            new PolicyStatement({
                sid: 'AllowCloudFrontServicePrincipalReadOnly',
                effect: Effect.ALLOW,
                principals: [new ServicePrincipal('cloudfront.amazonaws.com')],
                actions: ['s3:GetObject'],
                resources: [`${bucketArn}/*`],
                conditions: {
                    StringEquals: {
                        'AWS:SourceArn': `arn:aws:cloudfront::${Stack.of(this).account}:distribution/${
                            this.distribution.distributionId
                        }`
                    }
                }
            });

        // Add policies
        const hostingBucket = Bucket.fromBucketAttributes(this, 'HostingBucket', {
            bucketName: props.hostingBucket.bucketName,
            region: AWS_REGION
        });
        const hostingBucketPolicy = new BucketPolicy(this, 'HostingBucketPolicy', {
            bucket: hostingBucket
        });
        const hostingBucketPolicyStatement = policyStatement(props.hostingBucket.bucketArn);
        hostingBucketPolicy.document.addStatements(hostingBucketPolicyStatement);

Resource handler returned message: "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."

Reproduction Steps

Create two different stacks in different regions, where one is us-east-1.
Create an S3 bucket in the regional stack and a Cloudfront distribution in us-east-1.
Pass the S3 bucket by reference to the us-east-1 stack and enable crossRegionReference.
Add the S3 bucket as an origin in Cloudfront.
Add a bucket policy to the S3 Bucket by using bucketRef.addToResourcePolicy()

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.93.0

Framework Version

No response

Node.js Version

18.14.1

OS

Windows 10 x64

Language

Typescript

Language Version

5.1.6

Other information

No response

[peterwoodworth]

This makes sense to throw an error here - Stack A which contains your bucket introduces a dependency to Stack B when props.hostingBucket.addToResourcePolicy(policyStatement(props.hostingBucket.bucketArn)); is called. This will add the policy in the same scope as the bucket, which is Stack A, and the policy statement contains a reference to a token not known until after deploy time (the distribution ID) from Stack B.

And of course, Stack B depends on Stack A because you're using the bucket in the distribution.

I think you're onto something in the second solution - but I'm not sure why that error would be throwing since buckets are global. In your template, does it mention anything other than the bucket's name?

[JonWallsten]

Is there a recommended way of overcoming this? The policy obviously need the distribution id from StackB, and the bucket is needed for the origin.

I tried different combination for the bucket lookup but got the same error in all cases.
These are the combination I've tried:
bucketArn
bucketRegionalDomainName
bucketArn+bucketRegionalDomainName
bucketName+region

The only time I got this to work was when I created a SSMParameterReader and SSMParameterWriter using AWS Custom Resource and wrote the bucket ARN into the parameter store right after it's creation and then used the parameter as the lookup. But I wanted to get rid of the those since they are always written and read even if nothing have changed leading to huge increase in deploy time.

After I wrote this yesterday, I moved all resources besides the WebAcl and the Certification used by Cloudfront to the regional bucket to get rid of the cross-region(and cross-stack) reference for the bucket. So now I only have a cross-region(and cross-stack) reference for the WebAcl and the Certificate. Those are not circular in anyway and can be updated without issues.
I've been having issues with so many resources when I used cross-region(and/or cross-stack) references (unable to set policies, unable to update Lambda @ Edge, unable to update Lambda Layers, unable to update certain properties) that moving it all to the same stack is the only solution that actually work kind of good. But having all infrastructure in the same stack is not something we really want. I guess Cloudfront being Global is the main issue here leading to a lot of other issues.

[peterwoodworth]

Is there a recommended way of overcoming this?

Unfortunately, outside of merging everything into one stack or using custom resources, I'm not aware of any workarounds to this.

[JonWallsten]

I see. Is there a way of using the custom resources so that they don't do a update each time they are read and written even do the values has not changed? I guess the CDK does that somehow. I took around 40-50 seconds per stack that either read or wrote parameters when I tried it out.

[pahud]

Looks like you need to build those resource in this sequence:

1. bucket in eu-west-1
2. cloudfront distribution(requires bucket ARN) in us-east-1
3. bucket policy(requires distribution ID and bucket ARN)

did you create the bucket policy from us-east-1 stack and encounter this error?

Resource handler returned message: "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."

[JonWallsten]

Looks like you need to build those resource in this sequence:

1. bucket in eu-west-1
2. cloudfront distribution(requires bucket ARN) in us-east-1
3. bucket policy(requires distribution ID and bucket ARN)

did you create the bucket policy from us-east-1 stack and encounter this error?

Resource handler returned message: "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."

@pahud:
We had so many different issues with cross-region references since we're using WAF, Cloudfront, Certificates, etc. so I ended up moving everything I could to the same region. So now Cloudfront is deployed in the same region as the buckets and this is not an issue anymore.