plan and template data source with aws_elasticsearch_domain for policy keeps complaining it has changed when it has not

[plombardi89]

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

[plombardi@plombardi-lt0 managed-infrastructure]$ bin/terraform version
Terraform v0.7.4

Affected Resource(s)

• aws_elasticsearch_domain

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

variable "name" {
  description = "the name of the Elasticsearch cluster"
}

variable "proxy_ip" {
  description = "the IP address of the proxy/NAT stable address to restrict access to Elasticsearch"
}

variable "data_instance_type" {}
variable "data_instance_count" {}

variable "master_instance_type" {}
variable "master_instance_count" {}

data "template_file" "policy" {
  template = "${file("${path.module}/policy.json")}"

  vars {
    proxy_ip = "${var.proxy_ip}"
  }
}


resource "aws_elasticsearch_domain" "elasticsearch" {
  domain_name = "${var.name}"
  elasticsearch_version = "2.3"

  advanced_options {
    "rest.action.multi.allow_explicit_index" = "true"
  }

  ebs_options {
    ebs_enabled = true
    volume_type = "gp2"
    volume_size = 20
  }

  cluster_config {
    instance_type            = "${var.data_instance_type}"
    instance_count           = "${var.data_instance_count}"
    dedicated_master_enabled = true
    dedicated_master_type    = "${var.master_instance_type}"
    dedicated_master_count   = "${var.master_instance_count}"
    zone_awareness_enabled   = true
  }

  access_policies = "${data.template_file.policy.rendered}"

  snapshot_options {
    automated_snapshot_start_hour = 0  // midnight UTC
  }
}

IAM Policy Template

{"Version":"2012-10-17","Statement":[{"Action":"es:*","Principal":"*","Effect":"Allow","Condition":{"IpAddress":{"aws:SourceIp":["${proxy_ip}"]}}}]}

Expected Behavior

Plan should not keep claiming the template has changed (it has not).

Actual Behavior

Plan keeps complaining that the template has changed and forcing an update of the policy on AWS.

Steps to Reproduce

1. terraform plan

[apparentlymart]

Hi @plombardi89. Sorry for the issue here.

Would it be possible for you to share the diff output (scrubbed of any sensitive info) from terraform plan on your subsequent runs when you were expecting it to be empty? This would help me to understand precisely what Terraform thinks has changed.

A frequent issue with IAM policies in Terraform is that the AWS services like to "normalize" them when returning them from the API, which causes diffs in spite of the meaning being the same. You mentioned in your report specifically that the template is what is showing as changed, but I'm curious as to whether it's actually the access_policies attribute itself that's coming back as different, due to some normalization of your policy document.

[plombardi89]

@apparentlymart Working on something else at the moment but I'll try and get what you asked for in a little bit.

[plombardi89]

@apparentlymart

~ module.tracing_es_develop.aws_elasticsearch_domain.elasticsearch
    access_policies: "{\"Statement\":[{\"Action\":\"es:*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":\"52.202.51.119\"}},\"Effect\":\"Allow\",\"Principal\":\"*\",\"Resource\":\"arn:aws:es:us-east-1:914373874199:domain/tracing-develop/*\"}],\"Version\":\"2012-10-17\"}" => "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Action\":\"es:*\",\"Principal\":\"*\",\"Effect\":\"Allow\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"52.202.51.119\"]}}}]}"

~ module.tracing_es_prod.aws_elasticsearch_domain.elasticsearch
    access_policies: "{\"Statement\":[{\"Action\":\"es:*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":\"52.202.51.119\"}},\"Effect\":\"Allow\",\"Principal\":\"*\",\"Resource\":\"arn:aws:es:us-east-1:914373874199:domain/tracing-prod/*\"}],\"Version\":\"2012-10-17\"}" => "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Action\":\"es:*\",\"Principal\":\"*\",\"Effect\":\"Allow\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"52.202.51.119\"]}}}]}"

[apparentlymart]

Thanks @plombardi89!

Here's what Terraform read back from the API as the current state:

{
    "Statement": [{
        "Effect": "Allow",
        "Action": "es:*",
        "Principal": "*",
        "Resource": "arn:aws:es:us-east-1:914373874199:domain/tracing-develop/*"
        "Condition": {
            "IpAddress": {
                "aws:SourceIp": "52.202.51.119"
            }
        },
    }],
    "Version": "2012-10-17"
}

...and here's what your template generated in the config:

{
    "Statement": [{
        "Effect": "Allow",
        "Action": "es:*",
        "Principal": "*",
        "Condition": {
            "IpAddress": {
                "aws:SourceIp": ["52.202.51.119"]
            }
        }
    }]
    "Version": "2012-10-17",
}

Notice that the API response includes the extra Resource constraint. This constraint is implied by the fact that this is a policy on the ElasticSearch domain, but the API wants to return it as an explicit member of the policy. So this looks like the same issue as #7763, #5067 and #3634.

Unfortunately this one is proving hard to resolve because it requires us to "predict" an appropriate ARN value for Resource. It's certainly possible to systematically calculate this value, but ARNs are more complicated than they look and so we've generally been resistant to trying to synthesize them in this way. That might end up being the only way this gets fixed though.

In the mean time, the workaround would be for you to explicitly specify the Resource ARN in your policy, which will then allow Terraform to see it as unchanged and not re-apply the change.

Since we already have several other issues open for this, I'm going to close this one to consolidate the discussion. Please feel free to re-open it if you think there's something unique about this case that isn't covered by the others.

Thanks again for the bug report!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.