Accessing maps and lists with 0.7.0

[lbernail]

I've just started migrating some stacks to terraform 0.7.0 and I have some trouble understanding the difference of behavior between lookup / element and direct access using brackets ( mapvar[key], listvar[index])

Here is a simple example with a map of lists:

variable "azs" {
  type = "map"
  default = {
      "eu-west-1" = [ "eu-west-1a", "eu-west-1b", "eu-west-1c" ]
  }
}

When I try accessing using lookup and element:
availability_zone = "${element(lookup(var.azs,var.region),count.index)}"

I get the following error:
element: argument 1 should be type list, got type string

Which is probably related to an error in lookup that is not caught because if I remove the element block and just look at lookup(var.azs,var.region) I get this error:

lookup: lookup() may only be used with flat maps, this map contains elements of type list

However if I access the map using brackets, it works fine:

availability_zone       = "${element(var.azs[var.region],count.index)}"

I also tried to use the bracket notation to access the resulting list:

availability_zone       = "${var.azs[var.region][count.index]}"

but got a parse error: Error reading config for aws_subnet[public]: parse error: syntax error

I looked at the documentation to understand the expected behavior of element, lookups and access using brackets but did not find any reference to the bracket notation (I mostly looked into variables and interpolation and may have missed it)

Some of these behaviors may be related to parsing or designed to ensure backward compatibility but it would be great if it was explained somewhere (I would be willing to help with this documentation)

[lbernail]

I just found another difference in behavior

If I have the following map:

variable "ami_basenames" {
    type = "map"
    default = {
        "debian-8.4" = "debian-jessie-amd64-hvm*"
        "ubuntu-16.04:eu-west-1" = "ubuntu/images-milestone/hvm-ssd/ubuntu-xenial-16.04*"
    }
}

When I try to lookup the name of the AMI based on the distribution using lookup it works fine:

lookup(var.ami_basenames,"${var.distrib}-${var.distrib_version}")

However if I try an access with the bracket notation:

var.ami_basenames["${var.distrib}-${var.distrib_version}"]

I get a surprising error:

unknown variable: var.dockerhost_distrib

If I use a single variable it works OK (I get a key does not exist un map in this example)

var.ami_basenames["${var.distrib}"]

[gazmania]

Hi. I'm also seeing similar issues. I want to be able to return arbitrary data structures out of a module. Now maps/lists can be returned that's a massive step forward, but I then have no reliable way to access the data due to the inconsistencies of the access functions and syntax:

e.g.

    output "test_crazy_out" {
      value = "${map(
            "brie", list(map("texture",list("creamy","smooth"))),
            "cheddar", list(map("texture",list("smooth","mild"))),
            "edam", list(map("texture",list("boring")))
          )}"
    }

I would expect to be able to access any part of that structure. So to get the primary texture of cheddar I would use

${module.mymod.test_crazy_out["cheddar"][0]["texture"][0]}

The above doesn't work. I can get to the second level of nesting with

${element(module.mymod.test_crazy_out["cheddar"],0)}

but can't go beyond that as a subsequent lookup() around the element refused to produce an output, and using a ["texture"] instead of a lookup gives a syntax error.

[sebolabs]

I wanted to a map with sg rules as a param to a miscroservice module and I get this error when trying to pass list of cidr blocks as a value of a key
[...] sg_ingress_rule { cidr_blocks = ["172.16.100.0/24","172.16.101.0/24"] protocol = "-1" from_port = "80" to_port = "80" } [...]
"${lookup(var.sg_ingress_rule,"cidr_blocks")}"
* lookup: lookup() may only be used with flat maps, this map contains elements of type list
please make it possible!
it would make everything easier as I already can work with list of maps and iterate through them

[dkniffin]

Here's another example of inconsistent behavior between brackets and element:

If I have the following variable:

variable "endpoints" {
    type = "list"
    default = [
      {
        path = "abc"
        http_method = "GET"
      },
      {
        path = "def"
        http_method = "POST"
      }
    ]
}

I was expecting to be able to access the maps inside of the list via:

element(var.endpoints, count.index)

But that threw:

element: element() may only be used with flat lists, this list contains elements of type map in

If I do the same thing with brackets, it works:

var.endpoints[count.index]

I did just notice the note in the docs "This function only works on flat lists", but that's very counter-intuitive. There's no reason it shouldn't work on nested lists or lists of maps, etc.

[gbougeard]

@dimfeld how do you access to your map keys?
var.endpoints[count.index]["path"] does not look to be correct/working.

[apparentlymart]

Hi everyone! Sorry things are so confusing here.

The [ ... ] syntax was added to try to smooth over some of the oddness caused by the use of functions for these behaviors, since Terraform's type system doesn't currently allow a function to return a different type depending on its input arguments and thus, as you've seen, the functions are all defined to return strings.

However, there are some further issues with the interpolation language right now that prevent the chaining of [...][...] sequences as @gbougeard found. hashicorp/hil#42 was intended to address that, but its complexity meant that it wasn't able to land in 0.9. It's now likely to be addressed as part of a more holistic look at how we deal with complex data types in Terraform, rather than trying to fix it in isolation as I did there, so there will be some future work to address this.

For now the usual recommendation is to use [...] for an inner access that returns a non-string value and then use a function around it, giving rather-odd combinations like this:

    value = "${element(var.foo["baz"], 2)}"

The use of the index syntax for the inner access allows it to return a type other than string, and then the use of a function for the outer works around the fact that the interpolation language doesn't like chaining index operations.

This situation is known to be far from optimal and will be addressed in a future release. Notably, this workaround only accommodates data structures two levels deep.

I'm going to close this now not because it's not something that will be fixed but because this issue was originally a question, which I think has now been addressed, and the underlying issues are already well-known and planned to be fixed. Thanks for the great discussion here, everyone!

[betabandido]

@apparentlymart Is there any update on this issue? We often encounter this issue when attempting to pass nested maps or lists to modules.

[deitch]

For now the usual recommendation is to use [...] for an inner access that returns a non-string value and then use a function around it, giving rather-odd combinations like this

@apparentlymart so then how do I retrieve an element from, say, a list embedded in a map embedded in a list?

variable "foo" {
    type = "list"
    default = [
      {
        bar = ["a","b","c"]
      },
      {
        bar = ["x","y","z"]
      }
    ]
}

I cannot use var.foo[0]["bar"][1] to get "b". because the sequential [..][..] doesn't work, and. I cannot use lookup(var.foo[0],"bar")because that returns a list, and lookup() only accepts a string!

[chriswhelix]

@deitch you can get around this problem with locals, e.g.:

locals {
  subnet_account_env_lookup = {
    "12345" = {
      production = ["subnet-123", "subnet-456"]
    }
  }
  subnet_env_lookup = "${local.subnet_account_env_lookup[var.account]}"
  subnets = "${local.subnet_env_lookup[var.environment]}"
}

That of course assumes you only actually need one entry out of the whole tree for a given invocation. But you could use the same technique for throwaway lookups near the resource that requires them, for example:

locals {
  subnets_in_foo_acct = "${local.subnet_account_proj_lookup[var.account_for_foo]}"
  subnets_for_foo = "${local.subnets_in_foo_acct["foo"]}"
}
resource "aws_alb" "foo" {
  subnets =  "${local.subnets_for_foo}"
}

I only just figured this out, but as far as I can see it provides a very generally applicable way around this limitation.

[deitch]

Thanks @chriswhelix . So you are saying that even though I cannot nest var.foo["a"][2]["bar"]6], and I cannot use functions either because of the strings requirement, I can replicate it using locals, since the result of local.<something> can be any data type? And thus I can build up a tree of locals to get the one I want?

That might actually solve some of may headaches. It still feels impossibly heavy compared with var.foo["a"][${count.index}][${var.name}], but is a step up.

[athak]

@deitch for most cases it should be a valid workaround but you can't use count with locals yet, so if your use case involves count, locals is not applicable. It bit me a few weeks earlier while transitioning some modules to 0.10.

[ghost]

Hey @apparentlymart

Is it possible to provide a bit more explanation on that syntax:

value = "${element(var.foo["baz"], 2)}"

It's not really clear to me what foo is and what the index is referring to. I have a list, with a map and then a list again, but I'm running into problems with the interpolation syntax.

resource "aws_security_group_rule" "default_cidr_rules" {
  count = "${length(var.security_group_rules_sg)}"

  from_port                = "${lookup(var.security_group_rules_sg[count.index], "from_port")}"
  to_port                  = "${lookup(var.security_group_rules_sg[count.index], "to_port")}"
  protocol                 = "${lookup(var.security_group_rules_sg[count.index], "protocol")}"
  type                     = "${lookup(var.security_group_rules_sg[count.index], "type")}"
  description              = "${lookup(var.security_group_rules_sg[count.index], "description")}"
  source_security_group_id = "${lookup(var.security_group_rules_sg[count.index], "cidr_blocks")}"

  security_group_id = "${aws_security_group.default.id}"
}

Variable definition:

  security_group_rules_cidr = [
    {
      from_port = 22
      to_port = 22
      protocol = "tcp"
      type = "ingress"
      description = "Corporate Network."
      cidr_blocks = ["${var.corp_nw_ip}"]
    }
  ]

Unfortunately lookup doesn't work with nested lists. I am hoping that this syntax might solve my problem, but I've tried a lot of variants on your example, which don't seem to work. So it would be great if you could give an example with the variable definition.

[marranz]

Similar problems heres, when using the service data resource for kubernetes: (output of the console bellow)

data.kubernetes_service.web-staging.spec [ { cluster_ip = 100.69.15.127 external_ips = [] external_name = load_balancer_ip = load_balancer_source_ranges = [] port = [ { name = node_port = 32260 port = 80 protocol = TCP target_port = 80} ] selector = { app = web env = staging} session_affinity = None type = NodePort} ]

`data.kubernetes_service.web-staging.spec[0]
{
cluster_ip = 100.69.15.127 external_ips = [] external_name = load_balancer_ip = load_balancer_source_ranges = [] port = [ { name = node_port = 32260 port = 80 protocol = TCP target_port = 80} ] selector = { app = web env = staging} session_affinity = None type = NodePort}

lookup(data.kubernetes_service.web-staging.spec[0],"cluster_ip")
100.69.15.127

lookup(data.kubernetes_service.web-staging.spec[0],"port")
lookup: lookup() may only be used with flat maps, this map contains elements of type list in:`

[gsvijayraajaa]

Similar problem here,

container_definitions = {
	"nginx" = {

				"type" = "large"
				"image" = "631.dockerhub.com/nginx:latest"
				"port" = 80
				"containerMountPoint" = "/etc/nginx"
				"environment" = [
						                { "name" = "a", "value" = "b" },
							        { "name" = "c", "value" = "d" }
						            ]
		}
}

I need to access environment key . There will be multiple nested maps in container_definitions.
lookup(var.container_definitions["nginx"],"environment")
Using lookup I get : lookup() may only be used with flat maps, this map contains elements of type list.

[ahulab]

Here's a workaround for anyone that winds up here:

resource "aws_security_group_rule" "default_cidr_rules" {
  count = "${length(var.security_group_rules_sg)}"

  from_port                = "${lookup(var.security_group_rules_sg[count.index], "from_port")}"
  to_port                  = "${lookup(var.security_group_rules_sg[count.index], "to_port")}"
  protocol                 = "${lookup(var.security_group_rules_sg[count.index], "protocol")}"
  type                     = "${lookup(var.security_group_rules_sg[count.index], "type")}"
  description              = "${lookup(var.security_group_rules_sg[count.index], "description")}"
  source_security_group_id = "${lookup(var.security_group_rules_sg[count.index], "cidr_blocks")}"

  security_group_id = "${aws_security_group.default.id}"
}

Variable definition:

  security_group_rules_cidr = [
    {
      from_port = 22
      to_port = 22
      protocol = "tcp"
      type = "ingress"
      description = "Corporate Network."
      cidr_blocks = ["${var.corp_nw_ip}"]
    }
  ]

Unfortunately lookup doesn't work with nested lists. I am hoping that this syntax might solve my problem, but I've tried a lot of variants on your example, which don't seem to work. So it would be great if you could give an example with the variable definition.

Define your list as a string using a delimiter, then split the string back into a list when you're doing your lookup.

So for the above example:

resource "aws_security_group_rule" "default_cidr_rules" {
  count = "${length(var.security_group_rules_sg)}"

  from_port                = "${lookup(var.security_group_rules_sg[count.index], "from_port")}"
  to_port                  = "${lookup(var.security_group_rules_sg[count.index], "to_port")}"
  protocol                 = "${lookup(var.security_group_rules_sg[count.index], "protocol")}"
  type                     = "${lookup(var.security_group_rules_sg[count.index], "type")}"
  description              = "${lookup(var.security_group_rules_sg[count.index], "description")}"
  source_security_group_id = "${split(",", lookup(var.security_group_rules_sg[count.index], "cidr_blocks"))}"

  security_group_id = "${aws_security_group.default.id}"
}

with the var definition:

security_group_rules_cidr = [
    {
      from_port = 22
      to_port = 22
      protocol = "tcp"
      type = "ingress"
      description = "Corporate Network."
      cidr_blocks = "10.0.0.0/8,192.168.1.1/24,8.8.8.8/32"
    }
  ]

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.