s3 remote state config stores credentials

[blalor]

Terraform version: v0.7.11

tl;dr: terraform remote config with the S3 backend and access_key and secret_key provided via -backend-config stores the credentials in the remote state persisted in S3; subsequent terraform plan fails with Error reloading remote state: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records if credentials are no longer valid.

Possibly related to: #7179, #1964, #2824
Definitely related to: #4376

I'm attempting to build out some complex environments using multiple AWS accounts. In the scope of a plan or apply, I require the following credentials (which could be for multiple accounts):

• persisting state in S3
• managing resources in critical, PCI-scoped AWS account
• managing resources in non-PCI-scoped AWS account

Furthermore, all AWS credentials are being dynamically generated/retrieved from Vault.

When preparing an apply during a CD pipeline, after cloning the repo containing the Terraform config, I use a wrapper script around the terraform commands which:

1. retrieves AWS credentials for all accounts from Vault
2. exports variables like TF_VAR_access_key, TF_VAR_secret_key, TF_VAR_remote_state_access_key, TF_VAR_remote_state_secret_key
3. runs terraform get to retrieve modules
4. runs terraform remote config to link project to remote state
5. runs terraform plan

The plan step fails, with an error like:

Failed to read state: Error reloading remote state: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: BF137E90D940C598

The full terraform remote config looks like:

terraform \
    remote config \
    -backend=s3 \
    -backend-config=bucket=$MY_BUCKET \
    -backend-config=key=state/env-dev.tfstate \
    -backend-config=region=us-east-1 \
    -backend-config=encrypt=true \
    -backend-config="access_key=${TF_VAR_remote_state_access_key}" \
    -backend-config="secret_key=${TF_VAR_remote_state_secret_key}"

If I re-run the terraform remote config, a subsequent plan or remote pull succeeds.

There are two actual bugs here that I can see:

1. remote backend credentials provided with terraform remote config are stored in the remote state
2. the credentials provided with terraform remote config are lost if the remote state contains credentials

[hashibot]

Hello! 🤖

This issue relates to an older version of Terraform that is no longer in active development, and because the area of Terraform it relates to has changed significantly since the issue was opened we suspect that the issue is either fixed or that the circumstances around it have changed enough that we'd need an updated issue report in order to reproduce and address it.

If you're still seeing this or a similar issue in the latest version of Terraform, please do feel free to open a new bug report! Please be sure to include all of the information requested in the template, even if it might seem redundant with the information already shared in this issue, because the internal details relating to this problem are likely to be different in the current version of Terraform.

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.