Terraform 0.13 complains that all list elements must have the same type, while 0.12 does not.

[marcosdiez]

The issue I am describing fails on Terraform v0.13.3 and works on Terraform v.0.12.29

I have a terraform module that adds routes to aws transit gateway (it's not provider related, though).

These routes come either from other modules or can be added manually.

This is how it looks like:

        target_subnets = [
          module.some_aws_vpc.route_target,
          module.another_aws_vpc.route_target,
          {
            "name" = "my_custom_route"
            "value" = {
              "cidr_block" = "172.22.64.0/21"
              "tg_attachment_id" = var.some_custom_tg_attachment_id
            }
          }
        ]

route_target, on the other hand, is defined as such:

    output "route_target" {
      value = {
        name = var.name
        value = {
          cidr_block       = var.default_route_to_transit_gateway ? local.routable_cidr_block : "0.0.0.0/0"
          tg_attachment_id = aws_ec2_transit_gateway_vpc_attachment.attachment.id
        }
      }
    }

In other words, it's the same structure. Terraform 12 is fine with it, Terraform 13 gives me the following error message:

The given value is not suitable for child module variable "target_subnets"
defined at ../modules/transit_gateway_routes/transit_gateway_routes.tf:1,1-26:
all list elements must have the same type.

I consider this a regression. If this can't be simply fixed, can we have a "cast" command ?

Thanks you!

[alisdair]

Thanks for reporting this @marcosdiez. I can't reproduce the issue without more information.

Can you provide a small configuration that I can use to reproduce the error message?

[marcosdiez]

Hi.

Sorry for the delay.
I was able to do a minimalist version that reproduces the issue:

it needs 3 files:

• main.tf
• m1/m1.tf
• m2/m2.tf

To test, one must type:
rm -f terraform.tfstate;terraform12 init && terraform12 apply && terraform13 init && terraform13 apply

with terraform 0.12.x it will work as expected.
with terraform 0.13.3, it will fail.

===========================BEGIN OF main.tf ======================

module "m1" {
  source = "./m1"

  name             = "m1_name"
  cidr_block       = "10.0.1.0/24"
  tg_attachment_id = "tg-12345"
}

output "m1" {
    value = module.m1.route_target
}

module "m2" {
    source = "./m2"

    target_subnets = [
        module.m1.route_target,
        {
            name = "manual_entry"
            value = {
                cidr_block       = "10.0.2.0/24"
                tg_attachment_id = "tg-999999"
            }
        }
    ]
}

output "m2" {
    value = module.m2.target_subnets
}

===========================END OF main.tf ======================

===========================BEGIN OF m1/m1.tf ======================

variable "name" {
    type = string
}
variable "cidr_block" {
    type = string
}
variable "tg_attachment_id" {
    type = string
}

output "route_target" {
    value = {
        name = var.name
        value = {
            cidr_block = var.cidr_block
            tg_attachment_id = var.tg_attachment_id
        }
    }
}

===========================END OF m1/m1.tf ======================

===========================BEGIN OF m2/m2.tf ======================

variable "target_subnets" {
  type    = list
  default = []
}

output "target_subnets" {
    value = var.target_subnets
}

===========================END OF m2/m2.tf ======================

[alisdair]

Thank you, this is super helpful! I'm able to reproduce with these steps, and also confirm that the issue is present in Terraform 0.14.0-dev.

[alisdair]

An initial debugging session indicates that Terraform is trying to convert the target_subnets argument to m2 from a tuple to a list, and the tuple element types are dynamic and the expected object({…}) type. The output from module m1 is an unknown value of dynamic type at the time this is happening, which is surprising.

[alisdair]

While investigating this a bit further, I found a workaround that might be useful to you. If you specify the full type of the input variable to m2, Terraform's type checking works correctly:

variable "target_subnets" {
  type    = list(object({
    name = string
    value = object({
      cidr_block = string
      tg_attachment_id = string
    })
  }))
  default = []
}

output "target_subnets" {
    value = var.target_subnets
}

Hopefully this applies to your real configuration, too!

I currently still think this is a bug, and I'm going to keep looking into it for now.

[mdiez-modus]

Hey, this is a nice trick. Although I also think this is a bug, it does solve all my problems with the additive of making my modules have strict typing.

Dumb question, are these "complex types" in the terraform documentation ? I can't remember seeing them.

[alisdair]

Yes, there's documentation on structural types on this page. The example for defining a full object type isn't very prominent, though, so perhaps we could improve that.

[marcosdiez]

Thank you for showing me yet another trick!

[alisdair]

After some debugging, I have more of an idea of what's happening here.

When the type of the module input is just list, this is equivalent to list(any), i.e. a list where the elements are of any single type. Terraform validates that the elements are all unifiable to a single type, and eventually reaches this block of code in the cty library.

As noted in the comments there, "this is a special case where the caller wants us to find a suitable single type that all elements can convert to, if possible." The problem comes when one of the elements is unknown/dynamic, as is the case with the output from module m1. This results in hitting this block of code:

// If the list element type after unification is still the dynamic
// type, the only way this can result in a valid list is if all values
// are of dynamic type
if listEty == cty.DynamicPseudoType {
	for _, tupleEty := range tupleEtys {
		if !tupleEty.Equals(cty.DynamicPseudoType) {
			return nil
		}
	}
}

Because the other element in the list is of a known type, object({…}), this fails and returns nil. The result is the error that is in the original report. As previously noted, adding a concrete list element type avoids this branch altogether, and since we can defer conversion of unknown/dynamic values until later, no error occurs.

The above block of code was added (by me 🥴) in response to a panic caused by using null as a tuple element. It's clearly not working in this situation, but I'm not sure how to suggest moving forward.

I think this is enough digging to warrant the "explained" label on this ticket, although whoever picks it up will still have to figure out a way forward that doesn't break existing behaviour in cty or Terraform itself.

[ocervell]

I'm running into this too, but there is no way out in my situation ...

I have a YAML config that I'm deserializing with Terraform describing my exporters: it is a list of objects that can contain string and list elements.

config:
  ...
  exporters:
  - class: Bigquery
    project_id: ${PROJECT_ID}
    dataset_id: ${BIGQUERY_DATASET_ID}
    table_id: ${BIGQUERY_TABLE_ID}

  - class: Stackdriver
    project_id: ${STACKDRIVER_HOST_PROJECT_ID}
    metrics:
      - error_budget_burn_rate
      - sli_measurement
      - slo_target

  - class: Datadog
    api_key: ${DATADOG_API_KEY}
    app_key: ${DATADOG_APP_KEY}
  
  - class: Dynatrace
    api_token: ${DYNATRACE_API_TOKEN}
    api_url: ${DYNATRACE_API_URL}

If I specify:

variable "config" {
  description = "SLO Configuration"
  type = object({
    ...
    backend   = any
    exporters = any
}

This results in the error above:

The given value is not suitable for child module variable "exporters" defined
at
../../../../../terraform/modules/terraform-google-slo/modules/slo-pipeline/variables.tf:21,1-21:
all list elements must have the same type.

I've tried specifying exporters = list(any), exporters = list(object({})), exporters = list, exporters = any .... but not of them work.

Question: in the example above, how would you define the exporters variable so that Terraform does not fail ?

Some workarounds that could be added to Terraform to allow such inputs:

• any should not behave as if there is only 1 type in the object pass. E.g: list(any) should accept arbitrary list objects

• ability to pass a list of maps with any type using list(object(any))

• ability to define optional arguments and default values for any sub fields when using complex types

• ability to ignore variable type checking (possible ?)

[alisdair]

Question: in the example above, how would you define the exporters variable so that Terraform does not fail ?

This is not currently possible. All objects must have the same type, and at the moment that means they must have valid values for each of the attributes.

For now, the workaround is to adjust your data such that each element has all of the attribute names present, with suitable empty values (e.g. null or []) where appropriate.

I have better news for the future, though! In 0.14.0, we are releasing an experimental optional object argument feature, which we hope to stabilize by 0.15.0. If you have time to try that out and give feedback, that would be much appreciated.

[Varun-garg]

For now, the workaround is to adjust your data such that each element has all of the attribute names present, with suitable empty values (e.g. null or []) where appropriate.

I am having issues while trying to do the same like maps of AWS resources. In @alisdair 's example, I can get defaults for aws_subnet. But for other resources, do we have dig into each resource and make defaults for them?

I am trying to upgrade from 0.12 to 0.15, but because of this issue, am facing issues getting to 0.13.

One of the examples:

variable "sensitive_nat_gateway_ips" {
 type = map
}
...
module "vpc-us-east-1" {
  sensitive_nat_gateway_ips = {
                                "us-east-1b": "ip_1",
                                "us-east-1c": "ip_2",
                                "us-east-1d": "ip_3",
                                "us-east-1e": "ip_4"
                              }
}

module "vpc-us-west-1" {
  sensitive_nat_gateway_ips = {}
}

...

data "aws_eip" "sensitive_nat_gateway_ips" {
  for_each    = var.sensitive_nat_gateway_ips

  public_ip   = each.value
}

...

resource "aws_nat_gateway" "sensitive_gateways" {
  for_each               = { for nat_details in local.sensitive_nat_gateway_details: nat_details.availability_zone => nat_details }
  depends_on             = [ aws_internet_gateway.main_vpc_gateway ]

  allocation_id          = lookup(lookup(data.aws_eip.sensitive_nat_gateway_ips, each.value.availability_zone, {}), "id", "") --> THIS FAILS
  subnet_id              = each.value.nat_public_subnet_details.subnet_id

  tags = {
    Name                 = "sensitive_nat_gateway_${replace(each.key, "-", "_")}"
    managed-by           = "terraform"
  }
}

I tried to do the following hack

data "aws_eip" "null_aws_eip" {

}

and replaced {} with data.aws_eip.null_aws_eip, but it fails during validation. Any help would be really appreciated.

[alisdair]

@Varun-garg We use GitHub issues for tracking bugs and enhancements, rather than for questions. While we can sometimes help with certain simple problems here, it's better to use the community forum where there are more people ready to help. The GitHub issues here are monitored only by our few core maintainers.

Please consider asking your question in the community forum. Thanks!

[apparentlymart]

Hi again, all!

Terraform v1.3.0 includes a final, stable version of the "optional attributes" feature that @alisdair mentioned as an experiment in an earlier comment.

This now allows describing an object type constraint which will accept input objects that lack particular attributes, in which case Terraform will automatically fill them in with placeholder default values during type conversion to still achieve a value of the specified type constraint.

Looking at the comment with an example of an exporters attribute which expects a list of objects, I think the following would be a reasonable way to describe the data structure illustrated by example in the YAML document:

variable "config" {
  type = object({
    # ...
    exporters = list(object({
      class      = string
      project_id = optional(string)
      dataset_id = optional(string)
      metrics    = optional(set(string), [])
      api_key    = optional(string)
      app_key    = optional(string)
      api_token  = optional(string)
      api_url    = optional(string)
    }))
  })
}

I imagine that the intent here was to select a different subset of attributes depending on the value of class. That is not something the Terraform type system was designed to allow: it expects all elements of a collection to be of the same type and therefore conversely expects that objects of different types will be in different collections. Therefore it might actually make more sense in this particular case to shape this in a different way with a separate attribute for each kind of "exporter", but I'm showing the above just because it's a concrete example of the optional object attributes syntax that might be useful to others following this discussion.

Although it was not intentional that the unification behavior changed as a result of fixing the panic in zclconf/go-cty#56, we don't plan to reintroduce the panic just to recover the previous implicit type inference behavior, and we typically consider it better to write out explicit type constraints anyway... any in practice seems to just lead to confusion later on because it relies on some tricky type inference heuristics that can make it appear as if a particular expression is valid even though what's really happening is Terraform has guessed subtly incorrectly what you meant.

Therefore, given how long the new behavior has been in place and given that the change was made to avoid a crash, I don't expect the previous behavior to be restored exactly as it was in Terraform v0.12. If you are encountering a problem where Terraform cannot automatically infer what collection element type you intend to use, the usual answer will be to write a more specific type constraint so that Terraform can tell exactly what type you're intending, instead of trying to guess. If you have particularly tricky examples that you're not sure how to specify, please start a topic about it in the community forum where we'll be happy to help with specific situations and suggest how to model them within the design of Terraform's type system.

Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.