Implement imports RFC

[yorinasub17]

https://terragrunt.gruntwork.io/docs/rfc/imports/

NOTE: In the RFC, we mention implementing import as a separate block. To expedite development, we recommend implementing it as part of include by enhancing its capability in backwards compatible ways.

Recommended phased implementation (expectation is that each of bullets is a separate PR):

1. [IMPLEMENTED] Expose the underlying config values under a reference var. E.g., being able to say include.foo.locals.

   • This would replace read_terragrunt_config, and has the advantage of providing a way to solve read_terragrunt_config with dependency is broken #1128 because we have more control over when parsing happens during a block decode, than a function eval.

2. [(a) IMPLEMENTED ; (b) IMPLEMENTED] Add the ability to configure the way included config merges. Right now include does an automatic shallow merge. In addition to shallow merge, user should have the ability to specify (a) no merge (full replacement for read_terragrunt_config), and (b) deep merge.

3. [IMPLEMENTED]Add ability to configure the included config to automatically fetch and expose dependencies.

4. [IMPLEMENTED]Add the ability to have multiple include blocks in one child config.

5. Add the ability to include multiple levels. Note that a potential yak shave is that this may break the include related functions, like get_parent_terragrunt_folder and path_relative_to_include. We will need to respecify how these work when multiple levels are included.

[Shocktrooper]

I see that this ticket has been marked as high priority and there are only 10 of those in total at the time of this comment so I believe it is one of the next few in the queue but do you have a general timeframe for this being implemented by chance @yorinasub17 ?

[brikis98]

high-priority are tickets we'll try to prioritize internally at Gruntwork. We do not have a timeline on them.

[yorinasub17]

First feature in the list is now implemented in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.30.4

[yorinasub17]

Deep merge with cross include dependency reference support is now available: https://github.com/gruntwork-io/terragrunt/releases/tag/v0.31.3

[yorinasub17]

Multiple include blocks (still single level) is now supported starting https://github.com/gruntwork-io/terragrunt/releases/tag/v0.32.0

This feature should allow significantly more DRY terragrunt configs even without multiple include levels. Refer to the updated use case documentation for more details.

[chrisurf]

@yorinasub17 what is the current status on adding the ability to include multiple levels? THX

[yorinasub17]

@chrisurf It needs some more design cycles to think through what it would mean to the functions. That said, with multiple include support addressing a few core use cases of DRY Terragrunt, I don't think we are likely to implement it anytime soon.

[norman-zon]

@yorinasub17 Any news on nested includes?
We are evaluating Terragrunt to get our Terraform code DRY and when experimenting with different layouts, I always bumped against the restriction of not being able to have more levels of include, which IMO breaks the principle of getting the code DRY quite a bit.

[david-heward-unmind]

i tend to agree with the above comment. single include level prevents breaking the parent into re-useable include blocks. including those blocks at a child level makes no sense since it would break DRY.

any chance we can get this moving?

[GergelyKalmar]

Indeed, due to this limitation it does not seem to be possible to include anything in a root config at all, which is a problem. In fact, if we do

include "root" {
  path = find_in_parent_folders()
}

as recommended in https://terragrunt.gruntwork.io/docs/features/keep-your-terragrunt-architecture-dry/, then there cannot be an include in any of the parent folder configurations anymore.

[OloJuChang]

Is this still being worked on?

[corinz]

+1

[kevcube]

Indeed, due to this limitation it does not seem to be possible to include anything in a root config at all, which is a problem. In fact, if we do

include "root" {
  path = find_in_parent_folders()
}

as recommended in https://terragrunt.gruntwork.io/docs/features/keep-your-terragrunt-architecture-dry/, then there cannot be an include in any of the parent folder configurations anymore.

@GergelyKalmar I recommend naming your root "root.hcl" (or anything else) and then doing

include "root" {
  path = find_in_parent_folders("root.hcl")
}

the default parameter of the find_in_parent_folders function is "terragrunt.hcl" but the name of included files is arbitrary

[GergelyKalmar]

I'm not sure if that helps, from what I understand the limitation is that only a single level of includes is allowed. It does not matter what the root config is called, you will still get an error if there is any other include anywhere else.

[joaocc]

Hi. Any updates on this development? Is it still being pursued? Thanks

[cveld]

Would be great if nested includes were implemented. Currently we have laid out an environment/component strategy where we need to repeat the generic component configuration in every component:

include "root" {
  path = find_in_parent_folders()
}

# Include the common configuration for the component. The common configuration contains settings that are common
# for the component across all environments.
include "common" {
  path   = "${dirname(find_in_parent_folders())}/infrastructure/${basename(get_terragrunt_dir())}/terragrunt.hcl"
  expose = true
}

This common include would perfectly fit inside the root include.

Tree:

│   terragrunt.hcl
|
├───environments
│   └───dev
|       |   env.hcl (environment specific cross component config)
|       |   
│       ├───kv
|       |      terragrunt.hcl (environment specific component config)
|       |   
│       └───rg
|              terragrunt.hcl (environment specific component config)
|
└───infrastructure
    ├───kv
    |      terragrunt.hcl (common component config)
    |      main.tf
    |
    └───rg
           terragrunt.hcl (common component config)
           main.tf

Full example in git:
https://github.com/cveld/terraform-tester/tree/main/Examples/terragrunt-environments-with-components

[sergiojoker11]

Any update on this? Brain hurts trying to workaround this limitation. Does anybody have a less ugly solution?

[amkartashov]

Here is approach I use for merging cfg.yaml files in nested directories of terragrunt stacks:

https://github.com/amkartashov/terragrunt-nested-configuration

In short, all cfg.yaml files are read from top directory to the stack directory and merged with:

this is not actual code, this is effectively generated code

cfg = merge([
  try(yamldecode(file("TOP_DIR/cfg.yaml")), {}),
  try(yamldecode(file("TOP_DIR/prod/cfg.yaml")), {}),
  try(yamldecode(file("TOP_DIR/prod/ecs/cfg.yaml")), {}),
  try(yamldecode(file("TOP_DIR/prod/ecs/cluster/cfg.yaml")), {}),
]...)

[amkartashov]

another idea is to use terramate to generate nested configuration https://github.com/amkartashov/terramate-terragrunt-example

this approach is more flexible, because terramate is responsible for merging configuration files, not merge() - so it allows custom modification and inheritance of nested configuration settings.

[elopsod]

@yorinasub17 what is the current status on adding the ability to include multiple levels?
Thank you!

[Lillecarl]

@elopsod https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#read_terragrunt_config this might be a viable workaround.

[yhakbar]

Hey folks,

I've been asked to look at this, and I think there's a good argument to be made to close this out as won't do:
#3517 (comment)

If folks agree, I'll cut a PR to mark the RFC as won't do in the docs site.