(cli): CDK Pipeline in account bootstrapped with 1.110.0 (bootstrap version 7): S3 Access Denied

[cogwirrel]

CDKPipeline fails with S3 Access Denied error when account has been bootstrapped with aws-cdk@1.110.0 (bootstrap version 7).

If I downgrade my CDK CLI to 1.108.0, bootstrap again (bootstrap version 6) and push a change through the pipeline it completes successfully.

Reproduction Steps

Minimal cdk pipeline repo here: https://github.com/cogwirrel/minimal-cdk-pipeline-ts

npm i -g aws-cdk@1.110.0
cdk bootstrap
cdk deploy
git remote add cc codecommit://MyRepo (requires git-remote-codecommit)
git push cc mainline

Observe error in the pipeline:

Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: ZE8ZB6NBMFEY4F6F; S3 Extended Request ID: t9dE+lUpj3K+CKD87YAPLOT0i2pD8CIqoTJv+KxBZQ6S84nM05bpPHzq0EhdNyjs8L00lSrR9wg=; Proxy: null)

What did you expect to happen?

Empty CDK Pipeline to deploy successfully

What actually happened?

"Prepare" step for the pipeline stage failed with an S3 Access Denied error.

Environment

• CDK CLI Version : 1.110.0
• Framework Version: 1.110.0
• Node.js Version: 14.17.0
• OS : macOS Big Sur (11.4)
• Language (Version): Typescript 4.3.2

Other

Possibly related to #15192 ?

---

This is 🐛 Bug Report

[cogwirrel]

Ah from a closer look at #15192 it seems like this allows for cross-account deployments but locks down same-account deployments :)

[cogwirrel]

Just tried deploying to a different account (with both bootstrapped with 1.110.0) and that worked ok :)

ie. I updated pipeline.ts in the example repo I linked with:

pipeline.addPipelineStage(new PipelineStage(app, 'Dev', {
  env: {
    account: '<different_account_id>',
    region: 'us-west-2',
  },
}));

So it does seem to only be when the target account for a pipeline stage is the same as the pipeline account :)

[engr-lynx]

I encountered the same issue both for new applications and old ones (previously working).

[ColeSiegelTR]

I am getting, this error S3 Access Denied on the prepare stage from the Cloud Formation execution role, despite granting that role access in the bucket policy and granting access to the encryption key. It is a cross account setup. Any suggestions how to troubleshoot further? Even when I turned off encryption for the bucket to test, the error persists.

[czubocha]

I've tested it and looks like the problem is in condition:

                Condition:
                     StringNotEquals:
                         s3:ResourceAccount:
                             Ref: 'AWS::AccountId'
               - Sid: PipelineCrossAccountArtifactsKey

In the Prepare step of pipeline input artifacts (Artifact_Build_Synth) need to be downloaded from PipelineArtifactsBucket. Without this condition in IAM policy, the permissions were granted but with it, they are not anymore (for the account where the pipeline is deployed of course).

@rix0rrr wrote in his PR

In-account bucket accesses are not directly permitted by the principal
roles; however, in-account accesses are enabled by resource policy
statements on the buckets themselves (which are sufficient to extend
the required permissions to the principal).

which is not really true in our case as there are no S3 resource policy statements allowing such access in PipelineArtifactsBucket.

So in my opinion to fix it we need to

• remove the above condition (but then the issue which caused the condition introduction will be still unsolved as DeploymentActionRole will still have access to all buckets in the target accounts)

OR

• add proper statements to the bucket policy of PipelineArtifactsBucket deployed in the same account as the pipeline.

[rix0rrr]

The statements need to go on the bucket. Can you share how you are defining the pipeline?

On June 27, 2021, GitHub Notifications ***@***.***> wrote: I've tested it and looks like the problem is in condition <https://github.com/aws/aws- cdk/commit/d04e28862a872ab90c00306193732c72a90c5e7c#diff- 4fdac38426c4747aa17d515b01af4994d3d2f12c34f7b6655f24328259beb7bfR414>: Condition: StringNotEquals: s3:ResourceAccount: Ref: 'AWS::AccountId' - Sid: PipelineCrossAccountArtifactsKey In the Prepare step of pipeline input artifacts (Artifact_Build_Synth) need to be downloaded from PipelineArtifactsBucket. Without this condition in IAM policy, the permissions were granted but with it, they are not anymore (for the account where the pipeline is deployed of course). @rix0rrr <https://github.com/rix0rrr> wrote in his PR <#15192 (comment)- 673444557> > In-account bucket accesses are not directly permitted by the > principal >  roles; however, in-account accesses are enabled by resource policy >  statements on the buckets themselves (which are sufficient to > extend >  the required permissions to the principal). > which is not really true in our case as there are no S3 resource policy statements allowing such access in PipelineArtifactsBucket. So in my opinion to fix it we need to * remove the above condition (but then the issue which caused the condition introduction will be still unsolved as DeploymentActionRole will still have access to all buckets in the target accounts) OR * add proper statements to the bucket policy of PipelineArtifactsBucket deployed in the same account as the pipeline.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15307 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe- auth/AAD77AUARDJPZCDPL6ULRILTU4IN5ANCNFSM47JLMYUQ>.

[czubocha]

https://github.com/czubocha/aws-cdk-pipelines-go-serverless/blob/master/cdk/pipeline.go
here it is, this is a very minimal setup

[rix0rrr]

Oh interesting. I have to say I'm confused. I have basically the same setup as that and in my case the permissions ARE added to the bucket.

Can you share the template of the case where they aren't being added?

[cogwirrel]

Ah does it work for you with the repro steps I provided?

[alexpulver]

I assume that the bucket policy should be created in the pipeline template. If so, I don't see it as well. To reproduce:

git clone -b future https://github.com/alexpulver/aws-cdk-sam-chalice
cd aws-cdk-sam-chalice
python3.7 -m venv .venv
source .venv/bin/activate
./scripts/install-deps.sh
export AWS_ACCESS_KEY_ID="***"
export AWS_SECRET_ACCESS_KEY="***"
export AWS_SESSION_TOKEN="***"
cdk --version  # Currently set to 1.107.0
cdk synth
grep AWS::S3::BucketPolicy cdk.out/AwsCdkSamChalice-Pipeline.template.json
vi package.json  # Update to 1.110.0
./scripts/install-deps.sh
cdk --version
cdk synth
grep AWS::S3::BucketPolicy cdk.out/AwsCdkSamChalice-Pipeline.template.json

[rix0rrr]

iiiiiiiiiiiiiiinteresting.

$  grep AWS::S3::BucketPolicy cdk.out/PipelineStack.template.json 
      "Type": "AWS::S3::BucketPolicy",

https://github.com/rix0rrr/test-cicd/blob/master/lib/pipeline-stack.ts

[rix0rrr]

AHA!

It breaks if you DO give an account on the stage, and the account is the same as the pipeline's account! If you leave out the account, then the permissions get properly added to the bucket.

[PierreKiwi]

Faced this issue today. Is the recommended fix to remove the account from the stage itself?

[rix0rrr]

For now, yes. We will fix it shortly.

[aax]

I have this problem, too. And like @czubocha I don't specify any account anywhere (neither in the app nor stage nor stack). Is your fix still working in this configuration, @rix0rrr ?

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[fbdo]

Hi all, @rix0rrr I tried today to deploy a pipeline using the newly released version 1.111.0 and I can see see the same S3 Access Denied error. I already tried to clean up/bootstrap again and no success.

[peterwoodworth]

Reopening because multiple customers are still experiencing this issue in the latest version #15406

[rix0rrr]

There are multiple causes for "S3 Access Denied".

I saw someone mention in the other thread they deleted and recreated their bootstrap stack.

Doing that will permanently break Pipelines.

See the pipelines README on how to unbreak yourself in that situation

[fbdo]

That was my case, I followed the instructions in the README and it started to work. Sorry for the false alarm, and thanks!

[czubocha]

My case was that I upgraded CDK CLI but I didn't upgraded CDK version in go.mod (I'm using Go bindings). Thanks for the fix, issue can be closed.

[alexpulver]

My case was resolved by this after upgrading the Python CDK dependencies

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.