Terraform is trying to detach a removed IAM Instance Profile

[thiagonache]

Summary

Terraform always try to detach instance profile which was already removed.

Config

resource "aws_s3_bucket" "s3_bucket" {
    bucket = "myuniquenamebucket201604080137"
    acl = "private"
}

resource "aws_iam_instance_profile" "S3ReadOnlyAccess" {
    name = "S3ReadOnlyAccess"
    roles = ["S3ReadOnlyAccess"]
}

resource "aws_iam_role" "S3ReadOnlyAccess" {
    name = "S3ReadOnlyAccess"
    path = "/"
    assume_role_policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Principal": {"AWS": "*"},
            "Effect": "Allow",
            "Sid": ""
        }
    ]
}
EOF
}

resource "aws_iam_policy" "S3ReadOnlyAccess" {
    name = "S3ReadOnlyAccess"
    path = "/"
    description = "Allow read only access to s3"
    policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [ "s3:Get*", "s3:List*" ],
      "Effect": "Allow",
      "Resource": "${aws_s3_bucket.s3_bucket.arn}"
    }
  ]
}
EOF
}

resource "aws_iam_policy_attachment" "S3ReadOnlyAccess" {
    name = "S3ReadOnlyAccess-attachment"
    roles = ["S3ReadOnlyAccess"]
    policy_arn = "${aws_iam_policy.S3ReadOnlyAccess.arn}"
}

Error

Error applying plan:

1 error(s) occurred:

* aws_iam_role.S3ReadOnlyAccess: Error deleting IAM Role S3ReadOnlyAccess: NoSuchEntity: Role S3ReadOnlyAccess in Instance Profile S3ReadOnlyAccess cannot be found.
    status code: 404, request id: dceca376-fd43-11e5-a64a-dd7324be381e

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Code

// Roles cannot be destroyed when attached to an existing Instance Profile
        resp, err := iamconn.ListInstanceProfilesForRole(&iam.ListInstanceProfilesForRoleInput{
                RoleName: aws.String(d.Id()),
        })
        if err != nil {
                return fmt.Errorf("Error listing Profiles for IAM Role (%s) when trying to delete: %s", d.Id(), err)
        }

        // Loop and remove this Role from any Profiles
        if len(resp.InstanceProfiles) > 0 {
                for _, i := range resp.InstanceProfiles {
                        _, err := iamconn.RemoveRoleFromInstanceProfile(&iam.RemoveRoleFromInstanceProfileInput{
                                InstanceProfileName: i.InstanceProfileName,
                                RoleName:            aws.String(d.Id()),
                        })
                        if err != nil {
                                return fmt.Errorf("Error deleting IAM Role %s: %s", d.Id(), err)
                        }
                }
        }

For sure roles cannot be deleted with IAM Profiles associated, but because terraform is managing the IAM Profile also, there's no guarantee it stills exists when API is called. Every time I got the same error.

Fix

PR

I'll open a new Pull Request with my suggestion and I'll update this issue. But, basically, what I'm doing is calling API before removing to double check if that Instance Profile still exists, and then delete it.

depends_on

We can also use depends_on to fix it.

Plan

We could change the plan to wait for IAM Instance Profile deletion and then remove the IAM Role. If we do this, that code block can be deleted

What do you guys think about it?

[thiagonache]

Opened PR #6083 for evaluation

[kpeder]

Regarding this comment:
#6045 (comment)
I would guess that the use case is out of scope for Terraform, which would expect to fully manage the object via explicit reference to its state file.
I have executed a 'terraform refresh' after manually adding an extra policy to an IAM role and found that the attachment is not recognized or added to the state. Potentially contrary to expectation, terraform does not enumerate any newly attached policies that were not instantiated by terraform (i.e. applied by terraform and added to the state). It seems that only resources that are explicitly listed in the state file are refreshed, regardless of their attachment to a terraform-managed resource.
See the PR discussion for visual dependency analysis with 'terraform graph'.

[kpeder]

Update on the PR here to resolve this issue. Terraform is not identifying resources that are attached by string reference instead of resource attribute. Correcting the template results in terraform mapping the dependencies properly and destroying resources in the proper order.
#6083

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.