🐛 Bug Report - S3 bucket encryption

[wanieldilson]

Describe the problem

As part of the Terraform upgrade, the AWS provider needed to be upgraded also. Upgrading the AWS provider meant making a lot of changes to the S3 bucket resources as per the new syntax.

The KMS key used to be part of the s3 bucket module but is now split out into an individual resource. This has caused buckets that were previously encrypted to enter a state of flux whereby the bucket shows as unencrypted but the objects within it remain encrypted with the old KMS key (now deleted).

S3 buckets will not successfully encrypt with the new KMS key.

[wanieldilson]

Option 1:

• Create a new encrypted s3 bucket for Thanos store
• Use a script to move every object to the new encrypted s3 bucket so everything is encrypted with the right key
• Update the Thanos store arguments in the staff-infrastructure-monitoring-deployments repo to point to the new bucket and associated KMS key

[wanieldilson]

S3 resources created with the latest AWS provider must have a lifecycle block configured to avoid issues with Terraform state. We were seeing this manifest itself in the form of encryption and other aspects of S3 not being reflected in the Terraform state.

In response to this, terraform was reverting any changes it didn't see in the state, which meant our resources were being changed on every pipeline run.

After adding the lifecycle block in PR #186 the S3 buckets were provisioned as expected. There is some cleanup to do as per #187