Provider block assume_role correctly assumes role for init, but not for plan

[josuevalverde12]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.6
+ provider registry.terraform.io/hashicorp/aws v3.26.0

Affected Resource(s)

Terraform aws provider block:

provider "aws" {}

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

If I run Terraform with role "arn:aws:iam::AAAAAAA:role/Role_A" that has the correct cross-account permissions to assume the required role "arn:aws:iam::BBBBBBBB:role/Role_B", the init succeeds but the plan fails. The plan ends up reverting back to using the previously assumed "arn:aws:iam::AAAAAAA:role/Role_A". This terraform modifies resources in account BBBBBBBB, which also contains the state file S3 bucket.

I have verified that running my Terraform configuration with Role_B ends in a successful init, plan, apply.

# Primary target region for this environment
provider "aws" {
  alias      = "primary"
  region     = "us-east-1"

  assume_role {
    role_arn = "arn:aws:iam::BBBBBBBB:role:role/Role_B"
  }
}

# Secondary region for replicas of S3 buckets
provider "aws" {
  alias      = "replica"
  region     = "us-west-2"

  assume_role {
    role_arn = "arn:aws:iam::BBBBBBBB:role:role/Role_B"
  }
}

Expected Behavior

• Terraform plan succeeds after successful init, using Role_B not Role_A.

Actual Behavior

• Terraform plan fails with AWS AccessDenied errors, saying Role_A does not have permissions to modify resource in account BBBBBBBB

Steps to Reproduce

1. Set up "arn:aws:iam::AAAAAAA:role/Role_A" with Administrator IAM policy attached.
2. Set up "arn:aws:iam::BBBBBBBB:role:role/Role_B" with Administrator IAM policy attached and a trust relationship on action sts:AssumeRole with account AAAAAAA.
3. Locally, assume "arn:aws:iam::AAAAAAA:role/Role_A"
4. Write terraform configuration that creates resources in account BBBBBBBB, with backend S3 config also in BBBBBBBB.
5. Configure aws provider to assume_role "arn:aws:iam::BBBBBBBB:role:role/Role_B". Like in my example above.
6. Run init. Succeeds.
7. Run plan. AccessDenied errors.

Important Factoids

• With TF_LOG=trace, I am able to see that an initial call to STS is made on plan and the role "arn:aws:iam::BBBBBBBB:role:role/Role_B" is properly assumed using credentials from role "arn:aws:iam::AAAAAAA:role/Role_A". However, before the plan is finished, I see another STS call that fetches credentials from the EC2 MetaData and re-assumes role "arn:aws:iam::AAAAAAA:role/Role_A", failing the rest of the plan.
• I've tried this with aws provider version 3.x as well since the authentication chain is modified from 2.x to 3.x`. Same errors occur.
• Also tried this with Terraform0.12.29

References

Could not find any other issues on these errors.

[sidewinder12s]

@josuevalverde12 wanna give this a try with version 3.25.0 of the provider?

.26 landed an update to the AWS SDK that had some changes to how credentials get resolved.

[josuevalverde12]

Hey @sidewinder12s, thanks for the reply. I gave it a try with 3.25.0 and same results :(
Would the Terraform version have anything to do with this or is it purely a provider-driven interaction?

[sidewinder12s]

I think provider auth is purely in the provider. If you were having issues with the S3 state backend that'd be closer to Terraform version from my experience, but I haven't used the assume_role features of the provider before either. Was just guessing since I ran into a bug in the updated SDK included in 3.26.

[josuevalverde12]

Bump. Can anyone replicate? Still encountering this issue.

[YakDriver]

I believe this to be the same issue as #17353 (or at least related).

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.