[RFC] Imports: soliciting feedback from community

[yorinasub17]

This is an RFC that introduces import blocks for addressing limitations of configuration reuse with include.

I would like feedback from the entire community on this, as this has the potential to address a lot of limitations that were proposed throughout the lifetime of terragrunt, but is fairly opinionated in the design principle.

NOTE: It may be easier to read in the rendered form. Here is the direct link.

[yorinasub17]

Use case that would benefit heavily from this: #744 (comment)

[brikis98]

Thanks for the great write-up Yori. I love going through all the concrete use cases to make sure it's solving the right problems 👍

Of the "reuse" options on the table—that is, the existing include, globals, read_terragrunt_config, manually reading JSON/YAML files, and your new import idea—I think import is probably the best bet, with read_terragrunt_config not too far behind (though I'd want to go through all the explicit use cases with it to see what they look like). However, there is one more option to consider that isn't explicitly about "reuse", which is the proposal I made here: #759.

It seems like with that approach, most of the code reuse issues go away. Everything in one environment is in one file (or, more likely, one set of .hcl files in a single folder), you can use locals to share variables, you can use module.xxx expressions to express dependencies, you don't need too many special Terragrunt helpers, and so on. In short, it's almost vanilla Terraform, which I'd argue is a very good thing, but with some extra functionality for better permissions management and avoiding everything ending up in one giant state file. If we're going to make a large change, this approach still strikes me as worth considering. How does it handle the code reuse use cases we are looking at with merge?

[tomasbackman]

Yes, very good write up and summary. Im really looking forward to having any of these solutions come to life as the current state is a bit annoying with the limitations in reusability.
Personally I would be happy with any of the proposed solutions. Most important for me is that they do not take too long to get.. terragrunt is after all a great wrapper that makes terraform easier/better. So letting these limitations stay too long feels worse (and contradicting the goal more) for me than getting the slightly "wrong" implementation (that still solves the issue.. =)

[yorinasub17]

Thanks for the feedback so far!

However, there is one more option to consider that isn't explicitly about "reuse" ...

Ah I was thinking of that as orthogonal, but you bring up a good point that it is related. When I get back to more thinking on this, I'll update with a section discussing this alternative.

I think import is probably the best bet, with read_terragrunt_config not too far behind (though I'd want to go through all the explicit use cases with it to see what they look like)

Given that read_terragrunt_config will probably be the easiest to implement, and to @tomasbackman point about speed of delivery and not letting this problem linger, I think it is worth baking that a bit more.

---

Given the above two and the feedback I've received, next steps on this RFC are for me to:

• Update section on read_terragrunt_config as a function to walk through the use cases
• Add section on RFC: single terragrunt.hcl per environment? #759
• Reorganize alternatives so that it highlights that the three top runners are: import, read_terragrunt_config, and RFC: single terragrunt.hcl per environment? #759

[jfunnell]

I agree with @brikis98 for what it's worth.. Flattening the structure and allowing locals to be used between modules would solve most problems I'm experiencing, though I guess there could be more complicated setups that might still need some nested, fancy shared variable hierarchy that your proposal seems to cover.

[yorinasub17]

UPDATE:

• 542509f : Add walkthroughs of each scenario for read_terragrunt_config. Note that after walking through the examples, the only way I see this avoiding the verbosity is if we convert all the blocks to attributes.

• 2b1039d : Include commentary on how RFC: single terragrunt.hcl per environment? #759 addresses the details in this RFC.

[brikis98]

Thanks for updating the RFC with details from that new proposal! I left a few thoughts/comments.

See also #759 (comment) for a thought I had...

[brikis98]

Why would locals be scoped to a file? The idea with the new proposal is that all .hcl files in a single folder are processed as if they are in a single file, just as Terraform processes all .tf files in a single folder as if they are in a single file.

[yorinasub17]

There is an advantage to having vars scoped to the file so that you don't have to keep picking new names for a thing like region.

[brikis98]

Agreed, but I think in net, treating all .hcl files in a folder as if they were in one .hcl file, just as Terraform does with .tf files, is the better trade-off.

[yorinasub17]

I'll update to note that it is preferred to have scope of locals shared, but mention the possibility of having file scoped vars, its advantages and disadvantages.

[brikis98]

Yea, if you're deploying the same things across regions, you'd use a different local for each region.

[brikis98]

Pretty sure locals and everything should be shared across all .hcl files in a folder, just like how Terraform does it for .tf files.

[yorinasub17]

See above for why one would want variables that are scoped to the file. E.g function scope vs global scope to avoid namespace clashes.

[jakauppila]

Looking at the suggestions, I'd be inclined to agree that import or read_terragrunt_config is the way way to go.

Our workaround for the for the differing inputs between configurations is to simply have two separate "top level" .hcl files that we include depending on the configuration so I think either solution would let us move away from that.

@brikis98 Flattening the structure as you proposed in #759 seems counter-intuitive to our use-case as I described in #759 (comment)

[jfunnell]

Hm, I guess I can agree there is still possibly a need for imports even with a flattened structure, after seeing some examples.. Orthogonal enough to warrant implementing both, at least.

I think read_terragrunt_config is really simple and non-magical, and would compliment both the existing layout and a unified module approach.

[yorinasub17]

Thanks for the feedback everyone! I think based on the feedback and new thoughts around #759, it sounds to me like #759 is where we want to be heading. However, given how big of a change that really is, I am not sure that will get done anytime soon and, as @tomasbackman mentioned, there is a pressing need to solve this problem sooner rather than later.

Given that, I propose that we should take an incremental approach here, implementing the features that are low cost and partially solve the problem, and slowly work towards more complete solutions.

I think:

• Given the simplicity of read_terragrunt_config, it is probably a nice addition to have alongside any of the other implementations and thus we should implement this.
• import with deep_merge is probably a good short/medium term solution to have inbetween status quo and RFC: single terragrunt.hcl per environment? #759 .
• Although it is not decided yet, my guess is that we will eventually converge towards a version of RFC: single terragrunt.hcl per environment? #759 in the long term.

I'm going to start implementing read_terragrunt_config, but further feedback on that conclusion is very welcome!

EDIT: PR for read_terragrunt_config - #1051

[brikis98]

Yea, the more I think about it, the more I believe the direction described in #759 (comment) is the way to go: turn Terragrunt into a preprocessor for more or less pure Terraform code. So many of the features we've built into Terragrunt already exist in Terraform (e.g., locals, helper functions, dependencies, etc), so many others become unnecessary with pure Terraform code (e.g., code generation of provider / backend blocks), and it does a better job of paving the way for Terraform to adopt these features we all want natively.

But yes, it'll take some time to do that, so +1 on putting in some reasonable stop gap solutions for now.

[yorinasub17]

UPDATE: I have updated the RFC to indicate that:

• This is now in development
• We will implement read_terragrunt_config, import blocks, and the single terragrunt.hcl file approach in that order.
• Added relevant PR and release links

[yorinasub17]

@brikis98 any objections to merging this in now?

[brikis98]

No objection, go for it! Thx Yori!

[yorinasub17]

Thanks for sanity check! Going to merge it in now.

[dudicoco]

@yorinasub17 is there a PR open for the import blocks feature?

[yorinasub17]

Unfortunately, not yet.

[xcrezd]

@yorinasub17 Any news on import feature?

[arash-bizcover]

@yorinasub17 is there any way we can reference a hcl file from another git repo either with include, read_terragrunt_config or import
My use case is I don't want to keep some generic hcl files in the applications repos.

[yorinasub17]

We currently don't have plans to implement remote references for import, but a PR to add support for that is welcome.

[michelzanini]

The import idea is brilliant. It will help many use cases we have.

I am not quite sure yet about #759. I would prefer to keep using the hierarchical folders. If you do get to implement #759 at some point, please make sure we can still do hierarchical folders. Mixing it all up in one single file should be optional.

As mentioned by @arash-bizcover , a remote import would be a great addition. We have repeated top level terragrunt.hcl files for multiple repositories. We need to keep syncing them in different repos. If we could import them all from a single Git repo it would be really good.

For example, consider something like this:

import "shared" {
  config_path = "git@github.com:my-company/shared-terragrunt.git/terragrunt.hcl?ref=master"
  merge = true
}

inputs = {
  
}

In this case config_path would support git references similar to the way source supports.

[jfunnell]

I am not quite sure yet about #759. I would prefer to keep using the hierarchical folders. If you do get to implement #759 at some point, please make sure we can still do hierarchical folders. Mixing it all up in one single file should be optional.

It's not one file, it's as many as you want. You just dont need a subfolder per module. Please read the whole proposal because most people, including myself, have propogated this misinformation

[damon-atkins]

How would you implement the following if the functions are deprecated
key = "${basename(get_parent_terragrunt_dir())}/${path_relative_to_include()}/terraform.tfstate"

[yorinasub17]

How would you implement the following if the functions are deprecated

I seem to recall a conversation in one of the issues regarding path_relative_to_include in the almost 1 year that passed since I wrote this (wow time flies) where this turns out to be a critical feature for many users and most people prefer the current approach vs. the more explicit approach proposed here, so we will most likely keep this around until there is a better replacement.

As for get_parent_terragrunt_dir, the intention is to have get_terragrunt_dir return that path. That is, in import mode, get_terragrunt_dir will always return the dir of the config you wrote it in, not the dir of the importing config. We currently aren't thinking about introducing a replacement for get_importing_terragrunt_dir, but if there are enough use cases for it, we will most likely add it in.

[Matausi29]

[digorgonzola]

Has there been any progress on this RFC?