aws_iam_role_policy_attachment resource produced new value for was present but now absent

[Matty9191]

Terraform version:

Terraform v0.12.19

• provider.aws v2.47.0
• provider.http v1.1.1
• provider.local v1.4.0

Affected Resource(s):

aws_iam_role_policy_attachment

Terraform Configuration:

resource "aws_iam_role" "ClusterAutoscalerRole" {
  assume_role_policy = data.aws_iam_policy_document.ClusterAutoscalerRole_policy.json
  name               = "${var.eks_cluster_name}_ClusterAutoscalerRole"
}

data "aws_iam_policy_document" "ClusterAutoscalerRole_policy" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.eks_cluster.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:kube-system:cluster-autoscaler"]
    }

    principals {
      identifiers = ["${aws_iam_openid_connect_provider.eks_cluster.arn}"]
      type        = "Federated"
    }
  }
}

resource "aws_iam_role_policy_attachment" "ClusterAutoScaler_polattach" {
  policy_arn = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:policy/${var.eks_cluster_name}_ClusterAutoscaler_policy"
  role       = aws_iam_role.ClusterAutoscalerRole.name
}

resource "aws_iam_policy" "ClusterAutoScaler_policy" {
  name   = "${var.eks_cluster_name}_ClusterAutoScaler_policy"
  policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}
POLICY
}

Error received:

I am consistently getting the following error when attempting to attach a policy to a role. @camlow325 reported a similar issue in 10549, and mentioned this may be an eventual consistency issue. Is similar retry logic needed here?

Error: Provider produced inconsistent result after apply

When applying changes to
module.eks_control_plane.aws_iam_role_policy_attachment.ClusterAutoScaler_polattach,
provider "registry.terraform.io/-/aws" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Expected Behavior:

I was expecting the policy to be attached without issue.

Steps to reproduce:

Running terraform apply produces this issue consistently.

Additional information:

Glad to provide additional information to help debug this issue

[ghost]

I experience this issue with terraform v0.12.21 and aws provider version 2.52.0.
The policy actually is attached to the role.

[ghost]

I experience this issue with terraform v0.12.21 and aws provider version 2.52.0.
The policy actually is attached to the role.

It appears that I had the issue because I used uppercase in the policy_arn, but in fact the policy name was all lowercase.

[bondsbw]

I got the same error when using an AWS managed policy, but with an ARN that contained the wrong partition. I applied the arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore managed AWS policy in GovCloud, but I should have applied arn:aws-us-gov:iam::aws:policy/AmazonSSMManagedInstanceCore.

Using the correct partition fixed the issue and did not result in this error. The fix may simply be a more specific error message.

AWS provider v2.58.0

(Copied here from #8751; this is a more relevant issue)

[strofimovsky]

It took me a while to find a single character difference in a policy name, but it was it.

It seems like API IAM API treats managed policy names as case insensitive, while Terraform looks for a case-sensitive match. API docs don't mention case sensitivity at all. Either way it should be consistent at least with reality (a case insensitive match).

[vadixidav]

@bondsbw Thank you! I think a good resolution here would be to print a better error out when the ARN partition is incorrect. This seems like it would be a common issue.

Also, a useful trick:

data "aws_region" "current" {}

locals {
  is_govcloud   = length(regexall("us-gov-.*", data.aws_region.current.name)) > 0
  arn_partition = local.is_govcloud ? "aws-us-gov" : "aws"
}

Then for an ARN: "arn:${local.arn_partition}:iam::aws:policy/..."

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.