Terragrunt can't find outputs in dependency

[gercograndia]

Describe the bug

A while ago, we upgraded to terragrunt version v0.58.2 in an azure environment, and were already noticing things were quite a bit slower than before.

But last week we ran into a confusing issue, where a particular module coulnd't resolve the outputs of a dependency, while it was definitely there.

ERRO[0910]  221:     dbxuc_schema_system = dependency.databricks_config.outputs.uc_audit.schema.id
ERRO[0910]
ERRO[0910] This object does not have an attribute named "outputs".

It must be said: this module had (by nature) quite a few dependencies, but all of them resolved fine except this one.

Inspecting the output of the said module showed that the outputs were definitely there. However, the terragrunt output command takes about 3 minutes, while just running plain terraform output from the .terragrunt-cache directory completes almost instantly (less than a second).

The debug log mentioned that the dependency is skipped because it is disabled.

time=2024-08-02T14:45:39+02:00 level=debug msg=Skipping outputs reading for disabled dependency databricks_config

The dependency declaration itself didn't have a enabled attribute set, but the terragrunt.hcl of the dependency does have a skip statement.

Anyways, while testing earlier versions of terragrunt we noticed the following:

• v0.55.1 is the last version that is both performant as well as properly resolving the dependencies.
• both v0.55.2 and v0.55.3 has the described issue, but are still 'quick'.
• in v0.55.5 another strange error popped up, but the plan is what we would expect:

ERRO[0008]  110:   groups = dependency.ad_groups.outputs.azuread_groups
ERRO[0008]
ERRO[0008] There is no variable named "dependency".

• in v0.55.6 that aforementioned error is gone, but the main issue (can't find the outputs) popped up. Also, it became very slow from this release onwards.

We tried also some newer versions but that wasn't helpful either:

• In latest version (v0.66.1) we got the following error:

ERRO[0167]   on xxxxx/terragrunt.hcl line 67:
ERRO[0167]   67:         [for g in try(v.member_of, []) : try(dependency.ad_groups.outputs.azuread_groups[g].id, null)]
ERRO[0167]
ERRO[0167] with
ERRO[0167] v.member_of set to null
ERRO[0167] .
ERRO[0167] A null value cannot be used as the collection in a 'for' expression.
ERRO[0167] Error: Iteration over null value

• v0.60.x no new issues, but the main 'outputs' issue was still there.

We didn't put more effort into the newer versions yet, we might do that. We're also more than willing to test out a few things if that's going to be helpful.

Our CICD plan route decreased from 36 minutes (using v0.58.3) to 10 minutes (using v0.55.1)

Steps To Reproduce

Just changing the terragrunt version and run terragrunt plan.

Expected behavior

It should resolve the outputs (and not be that slow :-)

Versions

• Terragrunt version: v0.58.3
• OpenTofu/Terraform version: terraform v1.5.7
• Environment details (Ubuntu 20.04, Windows 10, etc.): macos

Additional context

n/a

[gercograndia]

Maybe in addition:

I reduced the terragrunt.hcl to remove all other dependencies so that only the 'faulty' dependency remained.

However, the issue still remained, so I guess it has not so much to do with the sheer number of dependency declarations.

[kjonca]

Seems we are also hit by this issue (terragrunt 0.66.1). Interesting thing is: when I temporarily change to "skip=false" in dependency, then run "terragrunt plan" (which I believe fetches the output of dependency), then, after restoring to "skip=true" - output is visible and is used.

[gercograndia]

Seems we are also hit by this issue (terragrunt 0.66.1). Interesting thing is: when I temporarily change to "skip=false" in dependency, then run "terragrunt plan" (which I believe fetches the output of dependency), then, after restoring to "skip=true" - output is visible and is used.

Hmm interesting. Let me do some tests here as well.

However, I woud argue that if you have a skip = true in a dependent module, then it would make sense to not fetch the outputs right? I must say that I am not very keen on this skip option, it gives quite a few issues.

[kjonca]

"dependent module, then it would make sense to not fetch the outputs right? " It's complicated. :) We use this to make some configuration tests. And skips is used to tell "we do not want to test this big and complicated module" now, but we want to check modules which depends on it.

[levkohimins]

@gercograndia, @kjonca, Guys, can you provide the simplest possible configuration to reproduce the issue?

[gercograndia]

Hi @levkohimins,

I'll see if we can reproduce it (tomorrow) in a as simple as possible setup but in our case it will never be easy.

I think the situation is as follows:

• module_1 has a dependency on module_2 and wants to use the outputs.
• module_2 has a skip attribute, and uses the output of a dependency to module_3 to decide whether or not to skip.

I think the situation of @kjonca is different: trying to get hold of the outputs when skip = true.

[levkohimins]

Thanks @gercograndia, If you can't provide a simple configuration, that's not a problem, as long as it reproduces the issue.

[gercograndia]

Here you go @levkohimins

Assuming both module-2 and module-3 are applied, run a terragrunt plan in module-1.

Tested with terragrunt v0.58.3, which shows the issue (just as the latest, v0.66.1). v0.55.1 is doing it fine.

terragrunt-issue-3321.zip

What this config does NOT show though, is the huge performance degradation introduced in v0.55.6.

[levkohimins]

Thank you very much @gercograndia. Let's solve the "no outputs" issue first, and then I'll try to look at the performance.

[gercograndia]

Great, thanks a lot in advance.

[gercograndia]

Any updates here guys?

Got bitten by it again yesterday, as we had one last skip lingering around and that caused that a dependency existed without any issues, was creating (so not skipping) nicely but refused to report its outputs.

Took a while before I realised again that this was the same issue again, because first you start to check all your own configs. Pretty confusing bug actually.

Was on version v0.68.8 btw.

[yhakbar]

Hey @gercograndia ,

Take a look at the new exclude block we've introduced, which replaces the skip attribute (specifically take a look at the all_except_output value that can be provided to the actions attribute). We built it out specifically to be a more flexible version of the skip attribute, so any feedback you can provide would be appreciated.

If you aren't aware, it's part of a larger suite of changes that you might also be interested in.

[gercograndia]

Hi, was not aware of the exclude block, nor with this 'larger suite of changes'. But looks promising, keep up the good work!

I consider skip as deprecated now, good riddance!