count and for_each for modules

[AnthonyWC]

Is it possible to dynamically select map variable, e.g?

Currently I am doing this:

vars.tf

locals {
  map1 = {
    name1 = "foo"
    name2 = "bar"
  }
}

main.tf

module "x1" {
  source = "../"
  parameter = "${local.map1["name1"]}"
}

module "x2" {
  source = "../"
  parameter = "${local.map1["name2"]}"
}

This works but it's repetitive/DRY to hardcode the key name.

Ideally I want to able to do something like this:

module "x1" {
  source = "../"
  parameter = "${local.map1[$var.select]}"
}

Where I can dynamically alter the key variable in the same file. I thought about using null_data_source:

data "null_data_source" "test" {
  inputs = {
    current = "${var.selector}"
  }
}

parameter  = "${local.map1[data.null_data_source.test.outputs["current"]]}"

But don't think I will able to inject different variable to selector value since I can't use locals within module block.

[apparentlymart]

Hi @AnthonyWC!

I'm not sure I understood correctly your question, but you can use any reference in the [ ... ] index brackets that you could use outside of them in the same context. For example, you could add a new local value for which name to select, like this:

locals {
  map1 = {
    name1 = "foo"
    name2 = "bar"
  }
  selector = "name1"
}

module "x1" {
  source = "../"
  parameter = "${local.map1[local.selector]}"
}

If I'm not answering the write question here, it'd help to have a more concrete, real-world example so that it's easier to understand what you're trying to achieve. It can be hard to figure out your intent with just generic names like "map1", "name1", etc.

[AnthonyWC]

Ok so originally I had something like this:

module "x1" {
  source = "../"
  tag_Name = "value1"
}

module "x2" {
  source = "../"
  tag_Name = "value2"
}

I want to reduce the number of copying, so I change it:

locals {
  tag_Name_map = {
    name1 = "value1"
    name2 = "value2"
  }
}

module "x1" {
  source = "../"
  tag_Name = "${local.tag_Name_map[name1]}"
}

module "x2" {
  source = "../"
  tag_Name = "${local.tag_Name_map[name2]}"
}

But I think I should able to go further with something like this:
(pseudo-code)

module "x1" {
  source = "../"
  tag_Name = "${local.tag_Name_map[$var.name]}"
}

module "x2" {
  source = "../"
  tag_Name = "${local.tag_Name_map[$var.name]}"
}

Where I can select the local map variable based on another variable on a per module basis. Not sure if locals is the right way to achieve it.

[apparentlymart]

I think I am still not understanding properly... in your second example you shown indexes name1 and name2, which you replaced both with $var.name in the final example. It seems like you're looking for something that would allow $var.name to be "name1" for module x1 and "name2" for module x2, but Terraform has no way to know that x1 maps to "name1" and x2 names to "name2".

I think maybe what you want to do is insert the module's own name in there? So in that case your tag map would look like this:

locals {
  tag_Name_map = {
    x1 = "value1"
    x2 = "value2"
  }
}

In that case, what you want to do here is not possible at this time. Terraform's configuration format prefers being explicit over implicit, and also being direct rather than indirect. While of course this is subjective, the design principles for the language actually consider your first example to be better because the value can be seen directly inside the module block, rather than needing to refer elsewhere. Unless that value is also going to be used in another location, factoring it out into a separate local map doesn't actually reduce the amount of copying... it just places the same information in a different place.

I think the closest thing to what you want here would be a future feature to address the request in #953. Other work has prevented us from prototyping that further since my last comment here, but we want to eventually support both count and the for_each argument discussed in #17179, which could allow you to write something like this:

locals {
  tag_Name_map = {
    name1 = "value1"
    name2 = "value2"
  }
}

# Not yet implemented and details may change before release
module "x" {
  # Create an instance of this module for each element in locals.tag_Name_map
  for_each = "${locals.tag_Name_map}"

  source = "../"
  tag_Name = "${each.value}" # use each value from the map
}

[AnthonyWC]

Thanks for the detailed response. I was thinking maybe it was possible to somehow set the value of $var.name differently within each module. Maybe with inline template if it was allowed to be set differently within each module (which doesn't look to work that way currently).

The other direction may be to look outside of Terraform and use something else to dynamically generate the terraform file, which separate out the templating logic and won't affect Terraform itself. Maybe something like a lightweight version of pongo2 for Terraform or some other project like this one (https://github.com/mjuenema/python-terrascript).

[apparentlymart]

I'm not personally familiar with python-terrascript, but indeed code generation is a reasonable approach if you are working on a system that has lots of repetitive, systematic elements that you can generate from some high-level representation. For example, I've seen this used to generate various aws_iam_* resources (or rather, modules containing them) based on a CSV file of users, because the tabular form of a CSV file can be more convenient in some situations.

The module function in that program does seem like it could help you achieve what you want here, though I have not tested it. I assume that it will generate a JSON-formatted Terraform configuration like this (which should be named something.tf.json rather than just something.tf as for the native syntax):

{
  "module": {
    "x1": {
      "source": "../",
      "tag_Name": "foo"
    },
    "x2": {
      "source": "../",
      "tag_Name": "bar"
    }
  }
}

Once something like that for_each mechanism I described before is implemented, it will be possible to do more operations like this within Terraform itself, but we support the JSON format because we expect there will always be situations where code generation is preferable.

[kuwas]

@apparentlymart

We're currently also doing something similar to @AnthonyWC, running a templating engine to preprocess input and generate Terraform from them.

I'm sure this comes up often, but are there any plans for Terraform to introduce a native way to do this? I dont meant as a part of HCL which will handle the actual runtime options. Instead, I was thinking of something similar to the way that Ansible processes templates to generate python before using it for execution, Terraform can generate HCL from templates.

[jakauppila]

@pselle Now with resource for_each implemented, I was curious what it might look like for module for_each? Is most of the heavy lifting done at this point or is it still quite a large effort?

[pselle]

@jakauppila Effort is hard to say, but modules and resources are handled in different areas of Terraform -- additionally, resources already have count whereas modules don't as yet (thus, count and for_each for modules are both a bit greenfield).

As described in the 0.12 preview last year, I expect some groundwork has already been made (necessary changes to the statefile, reserving those keywords ...).

I hope that gives some better picture of where the project is at with regards to modules and for_each, even if it's not the most hoped-for answer :)

[kuwas]

Any timelines on the implementation of this?

[AnupamaSoma]

we need a list based variable for both domain and path, please suggest how to use loops in terraform

[jspiro]

Can we get any kind of committment or anti-committment? Not having for_each on modules is blocking a lot of user-friendliness and consistency we want. We're halfway there with resource for_each, so it's particularly frustrating.

I'm pondering creating a local exec script that generates the module definitions from a template as a halfway step. There will still need to be some state mv migration when the full support is made available, but it should accomplish the same result.

[kuwas]

Can we get any kind of committment or anti-committment? Not having for_each on modules is blocking a lot of user-friendliness and consistency we want. We're halfway there with resource for_each, so it's particularly frustrating.

I'm pondering creating a local exec script that generates the module definitions from a template as a halfway step. There will still need to be some state mv migration when the full support is made available, but it should accomplish the same result.

@jspiro We're using a separate templating engine to generate terraform files with the modules, then we commit them into the repo, so that Terraform Cloud can just read from the compiled directory. When module for_each support drops, a simple one to one state mv shouldn't be too hard, and no local-exec is required. We're also considering of adding the compile step as a pre-commit hook.

Maybe you can try a similar workaround to prevent the "hack", namely the local-exec, from getting into your state file.

[jspiro]

Makes sense. In my case, I am trying to create an abstraction layer for users to not have to know terraform or github or do any cloning–I want them editing YAML files in Github directly, and it resulting in changes. I have this working well for resources, but not modules.

Which templating system did you pick?

[kuwas]

@jspiro We chose nunjucks, which is a js version of jinja2. I work for npm and since we are a js shop and have some legacy ansible, this choice made sense.

Another thing we've thought about doing is introduce a GitHub actions workflow that takes care of templating portion automatically. If you're looking to create an abstraction layer, maybe create some CI/CD pipelines that does this:

CI Pipeline

• User makes some changes to configuration yaml on a branch
• CI platform listens for push events on non-master branches
• CI platform reads from yaml configuration and runs templating engine
• CI platform push back to the branch with the compiled terraform files
• User opens pull request from their branch to master

CD Pipeline

• Terraform Cloud listens for events on master branch and the compiled terraform folder as "working directory"
• Terraform Cloud handles remote plan / apply

[terlar]

An alternative work-around, that potentially requires some more work (on the module side) that I have used is to create a fake for_each. By adding a variable called _for_each inside the module, then in each resource inside the module utilize this _for_each variable as the native for_each. In the end gather all the resources under a namespace inside the output.

This have worked well and the down-side is that the module becomes slightly more complicated (always working with list of things).

[sahilarora535]

@mitchellh @apparentlymart As great a tool terraform is(more so after the 0.12.6 release with for_each for resources), this issue for having for_each for modules is something which I am sure a lot of users want be solved and added as a feature in terraform. Do you have more visibility on plans for this, and if this is something being actively tracked?

[sdickhoven]

i have a use case for this for creating multiple vpcs using a complex data structure to steer the specifics. e.g.

vpcs = {
  east = {
    provider_alias = "us_east_1"
    prefixes = {
      primary = "10.192.0.0/16"
    }
    tiers = {
      public      = "primary"
      backend     = "primary"
      persistence = "primary"
    }
    ipv6               = true
    mesh               = true
    availability_zones = ["a", "b", "c"]
  }
  west = {
    provider_alias = "us_west_1"
    prefixes = {
      primary = "10.193.0.0/16"
      eks     = "10.224.0.0/16"
    }
    tiers = {
      public      = "primary"
      private     = "primary"
      backend     = "primary"
      persistence = "primary"
      eks         = "eks"
    }
    ipv6               = true
    mesh               = true
    availability_zones = ["a", "b", "c"]
  }
}

i would like to do

module "vpc" {
  for_each = var.vpcs
  source = "./modules/vpc"
  providers = {
    aws = format("aws.%s", each.value["provider_alias"])  #<<<< i just realized that this is not possible
  }
  name = each.key
  ...
}

i already have for_each "loops" inside this module for e.g. availability_zones, tiers, etc. so the only workaround i can think of would be to not package the (extensive) vpc config into a module and instead use the setproduct() function to explode "two-dimensional" data into a flat list. e.g.

resource "aws_subnet" "my_subnet" {
  for_each = { for key, val in var.vpcs : join("/", setproduct([key], val["availability_zones"])) => {...} }
  provider = ...
}

this would create

• aws_subnet.my_subnet["east/a"]
• aws_subnet.my_subnet["east/b"]
• aws_subnet.my_subnet["east/c"]
• aws_subnet.my_subnet["west/a"]
• aws_subnet.my_subnet["west/b"]
• aws_subnet.my_subnet["west/c"]

not ideal. :)

i just realized that

  providers = {
    aws = format("aws.%s", each.value["provider_alias"])
  }

is not possible.

so i'll have to do something like this:

module "vpc__us_east_1" {
  for_each = { for name, config in var.vpcs : name => config if config["provider_alias"] == "us_east_1" }
  source = "./modules/vpc"
  providers = {
    aws = aws.us_east_1
  }
  name = each.key
  ...
}

module "vpc__us_west_1" {
  for_each = { for name, config in var.vpcs : name => config if config["provider_alias"] == "us_west_1" }
  source = "./modules/vpc"
  providers = {
    aws = aws.us_west_1
  }
  name = each.key
  ...
}

so, i definitely need for_each for modules for what i'm trying to do. :)

we're in the process of re-spinning our aws infrastructure from scratch and being able to use for_each in modules would be fantastic!

terraform 0.12 makes a lot of things possible that weren't possible before and starting from scratch means that we can make the most of all the new features.

any indication as to schedule for this feature would be greatly appreciated. i'm under pressure to deliver something so unless for_each for modules will be supported within the next week or two i'll have to go forward with a workaround.

thanks.

[sdickhoven]

another use case is driving terraform config via data structs when using 3rd party modules. i'd like to build a data struct that defines all the inputs for my eks clusters and then use the gruntworks' eks module to actually instantiate them.
not having a module for_each would imply that i have to disassemble the 3rd party module in order to do this.

[jowrjowr]

Another use case I'm looking at is enabling AWS guard duty on every region.

This requires multiple providers, and I need to iterate over a list of regions for a module invocation (due to OTHER restrictions on the provider resource).

A for_each makes this easy. No for_each means this is tedious.

[pavel-khritonenko]

Workaround: I ended up creating the new module for the whole set of resources for particular value of list/map and then copying definition like below instead of creating a map and iterate over its keys:

module "m1" {
source = "../module"
var1 = "m1"
...
}

module "m2" {
source = "../module"
var1 = "m2"
...
}

[Cricen]

Ran into this again today and I'm really surprised this isn't a bigger deal. Any update on @jspiro commitment question?

[atecce]

we also could use this. in particular it would be great for k8s deployments across multiple clusters simultaneously

[conet]

@digitalfiz I am intentionally using count. count is used for number of copies I want to create for that module. I am setting the value of xyz_enable as either 1 or 0. When its set to 1, the module is deployed and its neglected when the variable value is 0.

Does this actually work? I'm asking because what you describe as a working feature seems to cover part of this issue.

[digitalfiz]

@conet. Yes passing any arbitrary variable into a module to tell it to turn things in the module on or off or even to control the count of said resources in the module will work. This issue is for something on a much higher level to remove any of the issues you would get from passing your own variable in.

Anyone right now could mimic this issues request at least partially if your module is simple enough. But if your module is already making use of the count feature on resources it becomes much harder to merge that with an arbitrary count passed in at the module level.

As an example say you have a module that creates an ecs service and all the things needed for it (sg, albs, taskdefs, etc...). This service can either be a worker or a web app so inside you're module you use a variable like enable_alb or something that is a bool and that sets count on the alb resource to 1 or 0 to turn the alb on and off. Now lets say you want to be able to use the count variable on the module to iterate over a list of services for your app. You now have to add a count to all your resources in the module and also make sure you still maintain the on/off in the count for the things in relationship to using an alb or not.

This problem is exacerbated when you start using modules in your modules, like if we wanted an alb module to reuse in other cases outside of this ecs module.

You also have a big headache of a mess when it comes to outputs.

If the count was handled by terraform at the module level you wouldn't need to worry about the counts on all the resources in the module other than for turning resources on or off.

[conet]

OK, that is what I thought, propagating the count to the entire resource tree behind a module is a pain that is why I haven’t considered it as an option. I agree that having it at the module level is useful.

[KptnKMan]

Also just ran into this issue again, and found both threads on the count and for_each here.
I too am hoping we can get some commitment on this, so that I'm not forced to create more gnarly deployments.
As it is, I'll have to implement the best case workaround used by @mohitm108.
Modules are supposed to be a core feature of Terraform, especially with the recent module registry, how is this not a thing?

Edit: Thumbs down @blalor and @dreamrace ? Really? You don't think this should be fixed asap? This issue of count and for_each in modules has been an issue since 2015.

[hildoer]

@KptnKMan Terraform is going to do this. They are close, or so they claim.

https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each/

[sheldonhull]

@KptnKMan Terraform is going to do this. They are close, or so they claim.

https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each/

I don't think that post is relevant to this as that's the preview blog post prior to release. As of now, I'm not sure the issue has been solved for using for_each on a module to iterate regions or resources. You still have to repeat yourself and can't use for_each to loop on a module itself.

[PLockhart]

I recently hit this problem, and am working around this lack of functionality by using a template file. It is working well so far.
https://www.reddit.com/r/Terraform/comments/cwp0d4/terraform_multi_region_question_can_i_just_use/fhkrvdt/?utm_source=share&utm_medium=ios_app&utm_name=iossmf

[vikas027]

I wish this gets implemented soon :) @apparentlymart

[ArseniiPetrovich]

The original (#953) issue has been around for more than five years already. It's anniversary, let's celebrate!

Seriously, guys, when it is going to be implemented?

[apparentlymart]

As noted in the comment I left above, implementation is in progress right now.

[ghost]

any update on this ?

[ghost]

Sooner then later terraform journey leads you to using module of modules of modules etc ...
for_each is crucial to have for making first level module act plural in same way as singular (current state).

locals {
  my_values = [
    {
      name = "one",
      set  = 1
    },
    {
      name = "two",
      set  = 2
    }
  ]
}

module "this" {
  source = "./module"

  for_each = local.my_values
  map_value    = each.value
}

---

Terraform team, thank you for v0.12.x and ongoing work with enhancement!
We eager to see this functionality into new releases.

[mightyguava]

Will this include support for looping over regions to simplify multi-region provisioning? I.e. being able to loop over providers as well?

[jakebiesinger-onduo]

@mightyguava OOC, what would the expected syntax for that be? Something like:

provider "google" {
  alias = "goog-us-east1"
  region = "us-east1"
}
provider "google" {
  alias = "goog-us-west1"
  region = "us-west1"
}
locals {
  regions = toset(['us-east1', 'us-west1'])
  providers = {
    us-east1 = google.goog-us-east1
    us-west1 = google.goog-us-west1
  }
}
module "vpc" {
 for_each = local.regions
  providers = {
    google = local.providers[each.key]
  }
  ...
}

[mightyguava]

Yes! That looks amazing. We deploy most of our infra in 2 regions in an active-passive configuration. So being able to instantiate both regions using the same module block would be a huuuge win. It's also our primary use case for for_each on modules.

I think it might be slightly nicer if you could iterate over the providers map directly rather than needing to have a regions set, but that's minor.

[jakebiesinger-onduo]

(FYI I'm not on the terraform team; just an interested bystander)

Yes, good point. I think no reason you theoretically couldn't iterate such a map. The problem is you currently can't build such a map, since it seems referencing google.goog-us-east1 is only valid syntax inside of a module.providers block.

You have my +1 that this would be awesome :)

[apparentlymart]

Hi @mightyguava,

That feature is not in scope for this issue, but I'd encourage you to open a new feature request issue to capture that use-case.

[mightyguava]

Thanks, added issue #24476. I think this is a natural extension to for_each over modules and would add a lot of value to the feature. (it was also my implicit assumption that this would work)

[pselle]

Hello folks, I've locked this issue to collaborators to reduce noise for the many subscribed. The work on this is in progress right now and is in the 0.13 release milestone.

[danieldreier]

I'm very excited to announce that beta 1 of terraform 0.13.0 will be available on June 3rd. Module count and for_each will be included in that release, and we're very interested in everyone's feedback during the beta period. I've pinned an issue with more details about the beta program, and posted a discuss thread for folks who want to talk about it more.

[pselle]

We turned on count and for_each for modules in master as of #24461. As @danieldreier said, this will be in the 0.13.0 release, with the beta coming out next week.

I'm closing this issue to reflect that the work is done and will be released soon!

[danieldreier]

Terraform 0.13.0 beta 1 launched today with count and for_each for modules! I hope people who have been following this request will take time to try out the beta and give feedback.