json config structure in the plan is incomplete, with missing locals and references

[frittsy]

Terraform Version

Terraform v1.9.5
on darwin_arm64

Use Cases

As an organization we write IaC policies using OPA/Rego by querying an output Terraform plan that has been converted into JSON format. When a resource or provider references a local value, such as local.some_constant, the Terraform plan does not include the expanded value of that local, so it's impossible to query if the value is acceptable.

Attempted Solutions

As a minimal example:

locals {
  role_arn = "arn:aws:iam::123456789012:role/IAM-Role-Manager"
}

provider "aws" {
  region = "us-east-1"

  assume_role {
    role_arn = local.role_arn
  }

  skip_credentials_validation = true
  skip_requesting_account_id  = true
  skip_metadata_api_check     = true
  access_key                  = "mock_access_key"
  secret_key                  = "mock_secret_key"
}

With the following commands:

terraform init
terraform plan -out=out.tfplan
terraform show -json out.tfplan > tfplan.json

Produces the following (shortened) output:

"configuration": {
  "provider_config": {
    "aws": {
      "name": "aws",
      "full_name": "registry.terraform.io/hashicorp/aws",
      "expressions": {
        "assume_role": [
          {
            "role_arn": {
              "references": [
                "local.role_arn"
              ]
            }
          }
        ]
      }
    }
  }
}

Where the value of local.role_arn is not available anywhere in the plan. Whereas variables with their known values do get included in the plan, so if I were to refactor this into var.role_arn, then I could feasibly query it.

I understand that locals can be complex expressions that are more than just constant values, but I'm hoping that it's possible to evaluate as much as possible and include what is known into the plan output.

Proposal

Admittedly I don't know the full implications of including these values in the plan output. I'm guessing that local values are always evaluated at runtime, and I think that behavior should remain the same. I would want this to be a purely informational change, if that is possible. If it would need to be opt-in behavior, could it be a flag such as:?

terraform plan -out=FILE -output-locals=true

References

No response

[crw]

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!

[liamcervante]

Hi @frittsy, are you able to expand more on exactly what it is you want to check about the value? The local value is going to be whatever the configuration sets it to be, which you have complete control over. I'm curious how the local value is producing a result that needs to be validated externally.

You can also use custom conditions within Terraform to ask Terraform to validate the values directly, instead of exposing them in the plan and then performing your own validation on them later. If the local value is being produced via references to inputs, you can validate the inputs directly (same with module outputs, and resources and data sources). Or you can simply use check blocks to validate the local value is what you need it to be and Terraform will produce warnings during the plan if everything is not as expected.

Thanks!

[liamcervante]

I just read your description more closely:

As an organization we write IaC policies using OPA/Rego by querying an output Terraform plan that has been converted into JSON format

If this is a value you want to use externally beyond just validation, you can create an output for it which will be included in the state and plan files.

[frittsy]

@liamcervante I did notice that output values are included in the plan, but there's a couple issues with that:
First, if I updated my configuration to output a local value like this:

locals {
  role_arn = "arn:aws:iam::123456789012:role/IAM-Role-Manager"
}

output "provider_assumed_role" {
  value = local.role_arn
}

Then the output Terraform plan includes:

"planned_values": {
  "outputs": {
    "provider_assumed_role": {
      "sensitive": false,
      "type": "string",
      "value": "arn:aws:iam::123456789012:role/IAM-Role-Manager"
    }
  }
}

But there's nothing in the plan that tells me local.role_arn maps to the output provider_assumed_role. So again, going back to my original post, this part of the plan

"references": [
  "local.role_arn"
]

becomes impossible to resolve. And I can't assume that every local will have an output of the same name in every Terraform configuration.

Additionally, even if outputting every local was a viable solution, we have hundreds of Terraform configurations/repositories in our organization that I would need to refactor to make that happen. At that point I could also just do what I said in the description and refactor some of these locals into vars, which do get included in the plan.

My goal here is that I should be able to take any Terraform plan and perform external validation on it, regardless of how it was written and whether these values are a local, variable, output, etc.

[jbardin]

Hi @frittsy,

Yes, it's a known issue that json config representation does not give you a complete representation of the config structure without separately parsing the real configuration. I think finding a way to restructure the plan output to allow this type of static analysis would be preferable to just dumping the local values into the plan (which isn't really feasible without some major changes, since locals, variables, and non-root outputs are always temporary values and never persisted to a place for serialization).

As far as validation within Terraform is concerned, the local value has no side effect of its own, so there's no validation to be done at that point. Validation is intended to happen at the point the value is used, which in this case would be within the resource configuration. The plan data will include the change in value for the role_arn attribute regardless of where that comes from. That value can be validated within Terraform using preconditions and postconditions, or externally in the planned changes.

So aside from doing validation directly in the configuration, would having a better configuration structure help with what you are trying to do?

[frittsy]

@jbardin thanks for the explanation. I figured it would be more involved than it seems on the surface, but I had to try.

The only thing I want to clarify is:

locals, variables, and non-root outputs are always temporary values and never persisted to a place for serialization

Because from my testing of:

variable "role_arn" { }

with the command

terraform plan -out=out.tfplan -var 'role_arn=arn:aws:iam::123456789012:role/IAM-Role-Manager'

ends up outputting this in the plan JSON

"variables": {
  "role_arn": {
    "value": "arn:aws:iam::123456789012:role/IAM-Role-Manager"
  }
}

which gave me hope that local values could be treated in a similar manner. But I understand and trust you that it is not that simple.

---

Your one comment

Validation is intended to happen at the point the value is used, which in this case would be within the resource configuration. The plan data will include the change in value for the role_arn attribute regardless of where that comes from

made me realize that my problem is actually only in the provider_config of the output plan. You're correct that local values get evaluated in resources, but in providers they're left as only references instead of constant_value. Which after reading this configuration representation section, I understand why it's left unevaluated a bit more. It's just unfortunate that there's no evaluated copy of the providers.

For an easy reproduction of what I'm talking about, this configuration:

locals {
  ecr_repository_name = "local-example"
  region              = "us-east-1"
}

resource "aws_ecr_repository" "this" {
  name = local.ecr_repository_name
}

provider "aws" {
  alias  = "local"
  region = local.region
}

provider "aws" {
  alias  = "constant"
  region = "us-east-1"
}

produces this plan output. And again, if I were to refactor this into var.region instead of local.region, I could resolve that reference.

[frittsy]

I'm afraid that my comment is losing track of what exactly it is I'm trying to achieve + my real issue.

I'm trying to write an IaC policy in OPA that looks for usage of a deprecated IAM role in AWS providers. Unfortunately it became commonplace in our org to use local values in the role_arn field of the AWS provider, so because of the missing information in the plan output, I can't validate what role is actually being assumed in that provider.

If I can only achieve this by parsing the Terraform configuration itself (perhaps with something like conftest), then I can try to work with that. But up until this point, we've written all our policies against the output plan JSON, so I was hoping I could continue using that.

[jbardin]

Thanks for extra detail @frittsy! For the provider configuration I think a more complete config representation would help to attempt to check the value, but it would still require walking back through all references, and in the case that the value is composed of multiple inputs via some expression you still would not have access to the final value anyway.

Using a variable would give you access to validations, and in some cases I've seen modules used only for their validation aspect to avoid exposing these as possible root input variables. You can pass the value into a module which performs strict type and value checking on the inputs, then outputs the same value back for use elsewhere in the module.

There's also less impetus on creating ways to validate provider configurations like this, because complex use cases often rely on providers to fetch their credentials from external sources, via environment variables, common config paths, input variables, etc. (though I understand that this could be use for non-credential validations which affect provider behavior as well).

[frittsy]

These are all good ideas but all of these proposed solutions involve refactoring the Terraform configurations to some extent, which is unrealistic for us because based on a rough code search we would need to update nearly 250 repos. And even if I did get them all updated to use variables with validation or a module with strict checks, that doesn't stop someone from writing a new configuration in the "bad" way or stripping out the validation. Which is why I'm implementing these policies and checks externally, in a CI/CD pipeline with OPA, so that regardless of how the configuration is written I can detect these violations.

I don't think there's any hope to include more data in the output Terraform plan, so I'm okay with closing this issue personally, but if you think there's any value in leaving this open then that's fine with me too. I appreciate all the input ❤️

[jbardin]

Even if the extra plan information wouldn't solve every case for provider configuration validation, the incomplete configuration structure in the plan is still a valid concern for writing external policies. I don't think we have a duplicate open for that, so I'll leave this here for now.

Thanks!