data "aws_iam_policy_document" and other resources are always read during apply

[dreinhardt89]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Versions

*Terraform v0.14.11
*hashicorp/aws v3.61.0

Affected Resource(s)

• aws_iam_policy_document
• aws_lambda_function

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "aws_elasticsearch_domain_policy" "master-company-es-5" {
  domain_name     = aws_elasticsearch_domain.master-company-es-5.domain_name
  access_policies = data.aws_iam_policy_document.master-company-es-5.json
}

data "aws_iam_policy_document" "master-company-es-5" {
  statement {
    resources = [aws_elasticsearch_domain.master-company-es-5.arn]

    principals {
      type = "AWS"

      identifiers = [
        "arn:aws:iam::AWS_ACCOUNT:root",
        aws_iam_user.master-company-user.arn
      ]
    }

    actions = [
      "es:AddTags",
      "es:DescribeElasticsearchDomain",
      "es:DescribeElasticsearchDomains",
      "es:DescribeElasticsearchDomainConfig",
      "es:ListDomainNames",
      "es:ListTags",
      "es:RemoveTags",
      "es:ESHttpDelete",
      "es:ESHttpGet",
      "es:ESHttpHead",
      "es:ESHttpPost",
      "es:ESHttpPut",
    ]

    condition {
      test     = "IpAddress"
      variable = "aws:SourceIp"

      values = [
        var.rack-vpccidr,
      ]
    }

    effect = "Allow"

    sid = "system-access"
  }
}

data "aws_lambda_function" "datadog-forwarder-arn" {
  function_name = aws_cloudformation_stack.datadog_forwarder.name
}

resource "aws_cloudwatch_log_subscription_filter" "datadog-log-subscription-filter-ads-linkedin" {
  name            = "datadog_log_subscription_filter"
  log_group_name  = "LOG_GROUP_NAME"
  destination_arn = data.aws_lambda_function.datadog-forwarder-arn.arn
  filter_pattern  = ""
}

TF Plan Output

  # data.aws_iam_policy_document.***-es-5 will be read during apply
  # (config refers to values not yet known)
 <= data "aws_iam_policy_document" "***-es-5"  {
      ~ id      = "2519247774" -> (known after apply)
      ~ json    = jsonencode(
            {
              - Statement = [
                  - {
                      - Action    = [
                          - "es:RemoveTags",
                          - "es:ListTags",
                          - "es:ListDomainNames",
                          - "es:ESHttpPut",
                          - "es:ESHttpPost",
                          - "es:ESHttpHead",
                          - "es:ESHttpGet",
                          - "es:ESHttpDelete",
                          - "es:DescribeElasticsearchDomains",
                          - "es:DescribeElasticsearchDomainConfig",
                          - "es:DescribeElasticsearchDomain",
                          - "es:AddTags",
                        ]
                      - Condition = {
                          - IpAddress = {
                              - aws:SourceIp = [
                                  - "10.30.0.0/16",
                                ]
                            }
                        }
                      - Effect    = "Allow"
                      - Principal = {
                          - AWS = [
                              - "arn:aws:iam::OUR_AWS_ACCOUNT:root",
                              - "arn:aws:iam::OUR_AWS_ACCOUNT:user/ninja-***",
                            ]
                        }
                      - Resource  = "arn:aws:es:us-east-1:OUR_AWS_ACCOUNT:domain/***-es5"
                      - Sid       = "system-access"
                    },
                ]
              - Version   = "2012-10-17"
            }
        ) -> (known after apply)
      - version = "2012-10-17" -> null

      ~ statement {
          - not_actions   = [] -> null
          - not_resources = [] -> null
            # (4 unchanged attributes hidden)


            # (2 unchanged blocks hidden)
        }
    }

# data.aws_lambda_function.datadog-forwarder-arn will be read during apply
  # (config refers to values not yet known)
 <= data "aws_lambda_function" "datadog-forwarder-arn"  {
      ~ architectures                  = [
          - "x86_64",
        ] -> (known after apply)
      ~ arn                            = "arn:aws:lambda:us-east-1:OUR_AWS_ACCOUNT:function:datadog-forwarder" -> (known after apply)
      + code_signing_config_arn        = (known after apply)
      ~ dead_letter_config             = [] -> (known after apply)
      ~ description                    = "Pushes logs, metrics and traces from AWS to Datadog." -> (known after apply)
      ~ environment                    = [
          - {
              - variables = {
                  - "DD_"                       = "datadoghq.com"
                  - "DD_API_KEY_SECRET_ARN"     = "arn:aws:secretsmanager:us-east-1:OUR_AWS_ACCOUNT:secret:DdApiKeySecret-OUR_DD_API_KEY"
                  - "DD_ENHANCED_METRICS"       = "false"
                  - "DD_FETCH_LAMBDA_TAGS"      = "true"
                  - "DD_S3_BUCKET_***ME"         = "datadog-forwarder-forwarderbucket-p2zdpdcf9syp"
                  - "DD_SITE"                   = "datadoghq.com"
                  - "DD_TAGS_CACHE_TTL_SECONDS" = "300"
                  - "DD_USE_PRIVATE_LINK"       = "false"
                  - "DD_USE_VPC"                = "false"
                }
            },
        ] -> (known after apply)
      ~ file_system_config             = [] -> (known after apply)
      ~ handler                        = "lambda_function.lambda_handler" -> (known after apply)
      ~ id                             = "datadog-forwarder" -> (known after apply)
      ~ invoke_arn                     = "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:OUR_AWS_ACCOUNT:function:datadog-forwarder/invocations" -> (known after apply)
      + kms_key_arn                    = (known after apply)
      ~ last_modified                  = "2021-10-05T17:35:46.000+0000" -> (known after apply)
      ~ layers                         = [
          - "arn:aws:lambda:us-east-1:OUR_AWS_ACCOUNT:layer:Datadog-Forwarder:7",
        ] -> (known after apply)
      ~ memory_size                    = 1024 -> (known after apply)
      ~ qualified_arn                  = "arn:aws:lambda:us-east-1:OUR_AWS_ACCOUNT:function:datadog-forwarder:$LATEST" -> (known after apply)
      ~ reserved_concurrent_executions = 100 -> (known after apply)
      ~ role                           = "arn:aws:iam::OUR_AWS_ACCOUNT:role/datadog-forwarder-ForwarderRole-1JAFORCKRMRC9" -> (known after apply)
      ~ runtime                        = "python3.7" -> (known after apply)
      + signing_job_arn                = (known after apply)
      + signing_profile_version_arn    = (known after apply)
      ~ source_code_hash               = "SOURCE_CODE_HASH" -> (known after apply)
      ~ source_code_size               = 133 -> (known after apply)
      ~ tags                           = {
          - "dd_forwarder_version" = "3.39.0"
        } -> (known after apply)
      ~ timeout                        = 120 -> (known after apply)
      ~ tracing_config                 = [
          - {
              - mode = "PassThrough"
            },
        ] -> (known after apply)
      ~ version                        = "$LATEST" -> (known after apply)
      ~ vpc_config                     = [] -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # aws_cloudformation_stack.datadog_forwarder will be updated in-place
  ~ resource "aws_cloudformation_stack" "datadog_forwarder" {
        id               = "arn:aws:cloudformation:us-east-1:OUR_AWS_ACCOUNT:stack/datadog-forwarder/105ae3e0-2531-11ec-9302-0e4b061813b3"
        name             = "datadog-forwarder"
      ~ parameters       = {
          ~ "DdApiKey"     = "****" -> "***"
            # (1 unchanged element hidden)
        }
        tags             = {}
        # (6 unchanged attributes hidden)
    }

  # aws_cloudwatch_log_subscription_filter.datadog-log-subscription-filter-account-set-crm must be replaced
-/+ resource "aws_cloudwatch_log_subscription_filter" "datadog-log-subscription-filter-account-set-crm" {
      ~ destination_arn = "arn:aws:lambda:us-east-1:OUR_AWS_ACCOUNT:function:datadog-forwarder" -> (known after apply) # forces replacement
      ~ id              = "cwlsf-2452665111" -> (known after apply)
        name            = "datadog_log_subscription_filter"
      + role_arn        = (known after apply)
        # (2 unchanged attributes hidden)
    }

Expected Behavior

None of the resources were changed so the tf plan should indicate no infrastructure changes

Actual Behavior

Terraform always states the data blocks will be read during apply and some of the resources (using the aws_lambda_function arn are recreated during apply).

Steps to Reproduce

1. terraform plan
2. terraform apply
3. terraform plan (without making any changes)

[justinretzolk]

Hey @dreinhardt-terminus 👋 Thank you for taking the time to file this issue. It sounds like you may be using values from the data source that can't be determined until the apply phase, as is described in the data resource behavior documentation, but it's a bit difficult to say with certainty with the information we have now.

Can you update the issue description with the output as well, as noted in the issue template so that we have all of the necessary information to investigate?

[dreinhardt89]

Hey @dreinhardt-terminus 👋 Thank you for taking the time to file this issue. It sounds like you may be using values from the data source that can't be determined until the apply phase, as is described in the data resource behavior documentation, but it's a bit difficult to say with certainty with the information we have now.

Can you update the issue description with the output as well, as noted in the issue template so that we have all of the necessary information to investigate?

I added in the tf plan output.

[justinretzolk]

Hey @dreinhardt-terminus 👋 Thank you for the update. I've got a bit of a breakdown here that I hope will help some.

data "aws_lambda_function" "datadog-forwarder-arn"

Based on the configuration, this data source is dependent on aws_cloudformation_stack.datadog_forwarder.name. In the plan logs, we see that this resource is being updated in place:

 # aws_cloudformation_stack.datadog_forwarder will be updated in-place

Because this dependent resource is being updated in place, Terraform will defer reading the data source until the apply phase. The resource "aws_cloudwatch_log_subscription_filter" resource is in turn dependent on the data source for the destination_arn argument, leading to the destroy/recreate.

What's unclear based on the information provided is why aws_cloudformation_stack.datadog_forwarder is being updated - is that happening on every run, which would give an indication as to why the data source is read during apply every time?

data "aws_iam_policy_document" "master-company-es-5"

This data source refreshing is a bit more unclear to me. I do see that there are three interpolations: aws_elasticsearch_domain.master-company-es-5.arn, aws_iam_user.master-company-user.arn, and var.rack-vpccidr, but the plan output doesn't seem to indicate a change in any of these. Are any of these values things that would not be known at plan time, which would cause the change? Alternatively, are any of those resources being changed in any way, which would lead to the same situation as mentioned above?

[dreinhardt89]

For aws_cloudformation_stack.datadog_forwarder updating it looks like the DdApiKey parameter is always updating in place which may be contributing to this. I'll look into this, I may be creating this incorrectly.

As for data "aws_iam_policy_document" "master-company-es-5" I put the additional configs below. I don't see anything that would not be known at plan time and non of the below resources are being changed in the terraform config.

resource "aws_elasticsearch_domain" "master-company-es-5" {
  domain_name           = "master-company-es5"
  elasticsearch_version = "6.8"

  advanced_options = {
    "rest.action.multi.allow_explicit_index" = "true"
  }

  cluster_config {
    instance_type  = "m3.medium.elasticsearch"
    instance_count = "3"
  }

  ebs_options {
    ebs_enabled = "true"
    volume_type = "io1"
    volume_size = "80"
    iops        = "1000"
  }

  snapshot_options {
    automated_snapshot_start_hour = 23
  }

  tags = merge(
    local.dev_billing_tags,
    {
      ServiceName = "MasterCompany"
      Domain      = var.convox_unique_prefix
    }
  )
}

resource "aws_iam_user" "master-company-user" {
  name = "${var.convox_unique_prefix}-master-company"
  path = "/"
  tags = merge(
    local.dev_billing_tags,
    {
      ServiceName = "MasterCompany"
    }
  )
}

variable "rack-vpccidr" {
  default = "10.30.0.0/16"
}

[justinretzolk]

Hey @dreinhardt-terminus I think this issue may be relevant to what you're seeing: #10300

As far as the additional configs that you provided, are any of those showing up in the plan output that may have been omitted what you provided previously? And is that data source being read more of an "this is not nice to look at" issue, or is it causing similar resource replacement on every run?

[dreinhardt89]

For the aws_cloud_formation_stack resource I got that working by implementing the sensitive parameters a different way (followed this as an example: https://github.com/scribd/terraform-aws-datadog/blob/master/logs_monitoring.tf).

As for the aws_iam_policy_document it triggers an update in place for aws_elasticsearch_domain_policy.

  # aws_elasticsearch_domain_policy.master-company-es-5 will be updated in-place
!   resource "aws_elasticsearch_domain_policy" "master-company-es-5" {
!       access_policies = jsonencode(
            {
-               Statement = [
-                   {
-                       Action    = [
-                           "es:RemoveTags",
-                           "es:ListTags",
-                           "es:ListDomainNames",
-                           "es:ESHttpPut",
-                           "es:ESHttpPost",
-                           "es:ESHttpHead",
-                           "es:ESHttpGet",
-                           "es:ESHttpDelete",
-                           "es:DescribeElasticsearchDomains",
-                           "es:DescribeElasticsearchDomainConfig",
-                           "es:DescribeElasticsearchDomain",
-                           "es:AddTags",
                        ]
-                       Condition = {
-                           IpAddress = {
-                               aws:SourceIp = "10.30.0.0/16"
                            }
                        }
-                       Effect    = "Allow"
-                       Principal = {
-                           AWS = [
-                               "arn:aws:iam::911070201873:root",
-                               "arn:aws:iam::119585928394:user/ninja-master-company",
                            ]
                        }
-                       Resource  = "arn:aws:es:us-east-1:119585928394:domain/master-company-es5"
-                       Sid       = "system-access"
                    },
                ]
-               Version   = "2012-10-17"
            }
        ) -> (known after apply)
        id              = "esd-policy-master-company-es5"
        # (1 unchanged attribute hidden)
    }

[dreinhardt89]

Looks like I am also getting the below but I don't know what override_main_response_version is.

resource "aws_elasticsearch_domain" "master-company-es-5" {
  domain_name           = "master-company-es5"
  elasticsearch_version = "6.8"

  advanced_options = {
    "rest.action.multi.allow_explicit_index" = "true"
  }

  cluster_config {
    instance_type  = "m3.medium.elasticsearch"
    instance_count = "3"
  }

  ebs_options {
    ebs_enabled = "true"
    volume_type = "io1"
    volume_size = "80"
    iops        = "1000"
  }

  snapshot_options {
    automated_snapshot_start_hour = 23
  }

  tags = merge(
    local.dev_billing_tags,
    {
      ServiceName = "MasterCompany"
      Domain      = var.convox_unique_prefix
    }
  )
}

  # aws_elasticsearch_domain.master-company-es-5 will be updated in-place
!   resource "aws_elasticsearch_domain" "master-company-es-5" {
!       advanced_options      = {
-           "override_main_response_version"         = "true" -> null
            # (1 unchanged element hidden)
        }
        id                    = "arn:aws:es:us-east-1:119585928394:domain/master-company-es5"
        tags                  = {
            "Domain"      = "ninja"
            "Environment" = "Development"
            "ManagedBy"   = "https://github.com/GetTerminus/infra-terminus-ninja"
            "ServiceName" = "MasterCompany"
        }
        # (8 unchanged attributes hidden)

[justinretzolk]

Hey @dreinhardt-terminus 👋 Thanks again for the updates! Since aws_elasticsearch_domain.master-company-es-5 is showing the update in place that you pointed out in your most recent comment, that explains why the data.aws_iam_policy_document.master-company-es-5 is showing as known after apply, and thus triggering the update in place for aws_elasticsearch_domain_policy.master-company-es-5 -- the aws_iam_policy_document.master-company-es-5 data source is dependent on aws_elasticsearch_domain.master-company-es-5 here:

 resources = [aws_elasticsearch_domain.master-company-es-5.arn]

[dreinhardt89]

So @justinretzolk do we have a snake eating it's own tail situation? Is there a workaround for this? override_main_response_version updates every time is there a way to prevent that? again I'm not sure what that parameter does.

[justinretzolk]

Hey @dreinhardt-terminus, I don't know that I'd necessarily say it's a snake eating it's own tail, as if aws_elasticsearch_domain.master-company-es-5 is not changed during a plan, I wouldn't expect the data source to be known after apply, and thus the other resources wouldn't show a diff either. Ultimately, I think the final piece of the puzzle is determining what's going on with that advanced_options block.

In looking at the documentation for the advanced_options block, it looks like it's key/value pairs. Here is where those values are pulled in during the plan/read process.

I'm curious - is it possible that this is being set outside of Terraform? If it were, Terraform would read that during the plan phase and try to remove it as it's not defined in the configuration. Some reference for that value can be found on this AWS document. If this value is something that you find should be set, you could update the aws_elasticsearch_domain.master-company-es-5 resource definition to include the following and it should stop the perpetual diff.

advanced_options  = {
  "override_main_response_version" = "true"
}

[dreinhardt89]

After adding the above advanced_option we no longer get a perpetual update in-place.

[justinretzolk]

Thank you for the udpate @dreinhardt-terminus! Since this seems to have ultimately been a combination of a few things that were configuration related, and doesn't seem to be a (new) bug, we're going to go ahead and close this issue for now. If you feel we've done this in error, please do let us know.

[knyar]

FWIW, I am seeing this too: a change to the override_main_response_version is being displayed on every Terraform run. Not sure if this happens for everyone, or just when advanced_options is not empty (I also have "rest.action.multi.allow_explicit_index" set there).

I do realise that I can set a value for override_main_response_version to make the diff go away, but if all users are now expected to provide a value for the that key, perhaps it should be somehow documented?

[justinretzolk]

@knyar Thank you for the additional context - I'm going to reopen this and mark it as a bug so that we can look into it as time permits, since multiple separate reports seems to indicate that it's not something that's a one-off configuration issue.

[justinretzolk]

On second thought, @knyar, would you mind opening a fresh issue with that information? Since it's a bit different that the initial issue that was reported here, I'd hate to cause additional confusion by having your issue description buried so far down.

[knyar]

On second thought, @knyar, would you mind opening a fresh issue with that information? Since it's a bit different that the initial issue that was reported here, I'd hate to cause additional confusion by having your issue description buried so far down.

Sure, I've created #21318

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.