AWS Provider 4.9 AWS IAM instance profile not updated on manually changed AWS IAM role

[bashoKa]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

terraform -v
Terraform v1.1.9
on darwin_arm64

• provider registry.terraform.io/hashicorp/aws v4.9.0
• provider registry.terraform.io/hashicorp/random v3.1.2

Affected Resource(s)

• aws_iam_instance_profile
• aws_iam_role

Related

• instance_profile.role not updated when role detached #25646
• [Bug]: Changes to aws_iam_instance_profile are not detected #32671
• Fix aws_iam_instance_profile detects that the instance profile is not associated with IAM Role. #34099
• iam_instance_profile not updating roles #1270
• IAM instance profile not created fast enough to modify EC2 instance #838

Terraform Configuration Files

resource "aws_iam_instance_profile" "brt" {
  name_prefix = "aws-brt"
  role        = aws_iam_role.brt.name
}

resource "aws_iam_role" "brt" {
  name               = "aws-brt"
  assume_role_policy = data.aws_iam_policy_document.brt_instance_assume_role.json
}

Actual Behavior

1. Manual deletion of the AWS IAM role
2. Manual rebuilding of the AWS IAM role
3. Instance profile will not be updated to the new role on terraform apply
4. Warning in AWS console: "The instance profile xxxx-profile associated with the environment has no role. Please attach a role to the instance profile."
5. Changing the AWS IAM role name to some random value && terraform apply
6. Changing the role name back to original value && terraform apply will update the instance profile

Steps to Reproduce

1. Manual deletion of the AWS IAM role
2. Manual rebuilding of the AWS IAM role
3. Instance profile will not be updated to the new role on terraform apply
4. Warning in AWS console: "The instance profile xxxx-profile associated with the environment has no role. Please attach a role to the instance profile."
5. Changing the AWS IAM role name to some random value && terraform apply
6. Changing the role name back to original value && terraform apply will update the instance profile

Important Factoids

References

Similar bug happened already before:

• iam_instance_profile not updating roles #1270

Thanks guys for your investigation and please let me know if you need further details.

[justinretzolk]

Hey @bashoKa 👋 Thank you for taking the time to raise this! So that we have all of the necessary information in order to investigate, would it be possible to supply debug logs (redacted as needed) as well?

[bashoKa]

Hi

Sure, just running it with debug mode?

best

[justinretzolk]

Hey @bashoKa 👋 That's correct. That can be done by setting an environment variable of TF_LOG=TRACE.

[bashoKa]

One thing I noticed preparing the trace:
I wasn't able to attach the created role to an ec2 instance:
Value (arn:aws:iam::123:instance-profile/awsbrt) for parameter iamInstanceProfile.arn is invalid. Invalid IAM Instance Profile ARN
But this happens right away after the creation of role and profile

[macropin]

We've hit this again. These are the steps to manually reattach a role to an instance profile:

# list all the instance profiles in the account
$ aws iam list-instance-profiles

# reattach the role required to the instance profile
$ aws iam add-role-to-instance-profile --instance-profile-name "instance1.foo.example.com_20220617071515949800000001" --role-name  instanceRole

I think this is a simple lifecycle issue caused by a boundary evaluation bug (> should be >=). Can someone please look at the code I linked in #25646 and confirm?

[vishwa-trulioo]

I can confirm that this is issue there on provider 4.27.0 as well.

My observation is, The following error shows up if the desired capacity has a deviation at the time when running the TF apply. But, if the desired capacity has no diff, then it won't throw errors.
However, this issue doesn't happen on 4.14.0.

Errors:

...
  # module.myasg.aws_autoscaling_group.this[0] will be updated in-place
  ~ resource "aws_autoscaling_group" "this" {
      ~ desired_capacity          = 2 -> 1
        id                        = "myasg"
        name                      = "myasg"
        # (22 unchanged attributes hidden)
      ~ launch_template {
            id      = "lt-05c1f947e191ff630"
            name    = "mylaunchtpl"
          ~ version = "55" -> "56"
        }
        # (9 unchanged blocks hidden)
    }
Plan: 0 to add, 2 to change, 0 to destroy.
module.myasg.aws_autoscaling_group.this[0]: Modifying... [id=myasg]
╷
│ Error: waiting for Auto Scaling Group (myasg) capacity satisfied: 13 errors occurred:
│ 	* Scaling activity (bb960aa2-4fc7-3683-7214-19e7d4b2d771): Failed: Value (myasg-role) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name. Launching EC2 instance failed.
│ 	* Scaling activity (18f60aa2-414a-69e4-d9bb-497515b792ab): Failed: Value (myasg-role) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name. Launching EC2 instance failed.
│ 	* Scaling activity (02960aa2-39b1-9321-93fa-456708f879ab): Failed: Value (myasg-role) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name. Launching EC2 instance failed.

This was reported in the past and fixed. Here are the ticket numbers. But, it seems it needs a bit more tuning.

1. Invalid IAM Instance Profile name terraform#15341
2. IAM instance profile not created fast enough to modify EC2 instance #838

[Nuru]

Just hit this bug in provider version 5.9.0

[circa10a]

Hitting this bug in v4.67.0

[JoshuaRowland]

This issue exists in version 5.4.0.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.43.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.