[terraform] Give AWSS3FullAccess to s3 IAM users

[nandajavarma]

Description

This PR essentially reverts this one. Even though with AWS CLI the permission seems enough, mc cli that we use for preflights(probably for other ops in Gitpod, not sure) requires a lot more permissions to do S3 ops. This is made clean in their documentation here.

Related Issue(s)

Fixes #13017

How to test

You can run the pipeline using:

werft run github -j .werft/eks-installer-tests.yaml -a deps=external -a domain=tests.doptig.com

Release Notes

NONE

Documentation

Werft options:

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-integration-tests=all
  Valid options are all, workspace, webapp, ide

[nandajavarma]

/hold for preview test to pass https://werft.gitpod-dev.com/job/gitpod-custom-nvn-fix-s3-preflights.1

[mrzarquon]

@nandajavarma since we are handing this out to customers as example reference architecture, I don't think we should be including extremely broad permissions that only need to exist for our CI jobs to work properly.

Double checking - is this the S3 preflight check done in our CI or in KOTS? In either case, it these are the minimum required permissions for our services to function, requiring them to be more permissive just because of how we want to validate them isn't a good idea.

[adrienthebo]

@nandajavarma since we are handing this out to customers as example reference architecture, I don't think we should be including extremely broad permissions that only need to exist for our CI jobs to work properly.

Double checking - is this the S3 preflight check done in our CI or in KOTS? In either case, it these are the minimum required permissions for our services to function, requiring them to be more permissive just because of how we want to validate them isn't a good idea.

I believe that this is the preflight check in KOTS.

That said, I believe our reference architecture documentation uses a reduced permission set, will double check that.

[nandajavarma]

Have created these follow up issues to fix the preflight check and reduce the permissions:

• [kots] fix the s3 preflights to use aws s3 cli instead of mc #13332
• [terraform] Reduce the s3 IAM role permissions from AWSS3FullAccess #13333

@adrienthebo @mrzarquon Do you think we can ship this now for the sake of the release and take up the follow-ups after?

[mrzarquon]

@nandajavarma since the preflight check in the wild, I agree we shouldn't update it and rush it out. But we should change the permission profile to be what is linked in the minio docs instead of just enabling all access. s3:HeadBucket is likely the permission we need to add, as MC is polling that to see if it has permission / access to the provided bucket. While this is their provided example, can we test with just adding the headbucket permission to the limited scope I provided:

statement {
    actions   = ["s3:HeadBucket"]
    resources = "*"
    effect    = "Allow"
  }

And then if that doesn't work, fall back to their provided example:

statement {
    actions   = ["s3:*"]
    resources = [
                "${aws_s3_bucket.gitpod-storage[count.index].arn}/*",
                aws_s3_bucket.gitpod-storage[count.index].arn
    ]
    effect    = "Allow"
  }
statement {
    actions   = ["s3:HeadBucket"]
    resources = "*"
    effect    = "Allow"
  }

[adrienthebo]

Since the Terraform module isn't shipped as part of the KOTS release and we're running on a schedule this week, let's merge this change, cut the release branch, and then clean up the IAM permissions right after the release. We take on minimal liability with this and merging this change keeps the release on track.

Edit: thinking about this for a moment longer, even if someone picks up the Terraform module with the widened permission set, they'll get the permission reduction after updating the Terraform module.

🚢

[adrienthebo]

/unhold for tests passing in https://werft.gitpod-dev.com/job/gitpod-custom-nvn-fix-s3-preflights.8