Terraform v0.13.0 Panic When Using Module Composition

[adamhorden]

Terraform Version

> terraform_git -v
Terraform v0.13.0-dev
commit 6fbd3942ea6a2e456d43467a64d34fe81dd08274

Terraform Configuration Files

I have had to just add crash.log to this issue report. This is an interesting panic, as it seems to be that if I have greater than two modules called from a root module I cause this panic. I can always try and strip out internal configuration to produce a test case so I am hoping crash.log will help you debug this panic.

If required I could tar up the config but it is multiple modules all in multiple git repos so hoping the stack trace will suffice.

Crash Output

https://gist.github.com/adamhorden/9730f0415d5166844ee1a947d8486f81

panic: no expansion has been registered for module.he_aws_tf_mod_he_aws_tf_app_itf.module.he_aws_tf_mod_he_aws_tf_app_itf_stk_be.module.he_aws_tf_mod_aws_cc_app_itf_stk

[apparentlymart]

Thanks for reporting this, @adamhorden!

Here's the isolated stack trace, for easier reference by someone working on this bug later:

panic: no expansion has been registered for module.he_aws_tf_mod_he_aws_tf_app_itf.module.he_aws_tf_mod_he_aws_tf_app_itf_stk_be.module.he_aws_tf_mod_aws_cc_app_itf_stk

goroutine 648 [running]:
github.com/hashicorp/terraform/instances.(*expanderModule).moduleInstances(0xc0011804e0, 0xc0000950d0, 0x1, 0x1, 0xc0018a8b50, 0x2, 0x4, 0xc0013981a0, 0x29349c0, 0xc000d1f140)
	/Users/adhorden/adhorden/terraform_git/instances/expander.go:267 +0xb72
github.com/hashicorp/terraform/instances.(*expanderModule).moduleInstances(0xc001180320, 0xc0000950c0, 0x2, 0x2, 0xc0015a3b50, 0x1, 0x4, 0xc00182a0a0, 0x9d, 0x2200000029)
	/Users/adhorden/adhorden/terraform_git/instances/expander.go:279 +0x2c9
github.com/hashicorp/terraform/instances.(*expanderModule).moduleInstances(0xc000d1e320, 0xc0000950b0, 0x3, 0x3, 0xc0015a3b50, 0x0, 0x4, 0x44db616, 0x40, 0x36)
	/Users/adhorden/adhorden/terraform_git/instances/expander.go:279 +0x2c9
github.com/hashicorp/terraform/instances.(*Expander).ExpandModule(0xc000d1e340, 0xc0000950b0, 0x3, 0x3, 0x0, 0x0, 0x0)
	/Users/adhorden/adhorden/terraform_git/instances/expander.go:116 +0x108
github.com/hashicorp/terraform/terraform.(*nodeExpandModuleVariable).DynamicExpand(0xc000db8400, 0x2fae6e0, 0xc00108c750, 0x1, 0x1, 0x1)
	/Users/adhorden/adhorden/terraform_git/terraform/node_module_variable.go:42 +0x94
github.com/hashicorp/terraform/terraform.(*Graph).walk.func1(0x29169c0, 0xc000db8400, 0x0, 0x0, 0x0)
	/Users/adhorden/adhorden/terraform_git/terraform/graph.go:84 +0x4d8
github.com/hashicorp/terraform/dag.(*Walker).walkVertex(0xc000e330e0, 0x29169c0, 0xc000db8400, 0xc000e0fb00)
	/Users/adhorden/adhorden/terraform_git/dag/walk.go:387 +0x357
created by github.com/hashicorp/terraform/dag.(*Walker).Update
	/Users/adhorden/adhorden/terraform_git/dag/walk.go:309 +0x11d7

I understand that this is a complicated configuration that's hard for you to reduce into a minimal reproducible example. As a compromise, if you could share some details about the heirarchy of modules that you have that might help to understand what we can see in the logs and the stack trace.

One way to quickly get an easily-shareable summary of the modules would be to run terraform init -upgrade and then share its full output, which includes summary information about the modules and providers that this configuration is using.

[adamhorden]

@apparentlymart I am trying to produce a reproducible test case. I can make this fail across many of our projects. In fact non of our projects will run with a v0.13 release.

I have attached a gist of init:

https://gist.github.com/adamhorden/24e7f8b72a73ba2f08d0bbbb5f9471e7

I am still working on a reproducible test case.

The best I could do is checkout all the modules, update the source link to these, sanitise any data, and tar it and then encrypt with the HashiCorp GPG key as a worst case if I fail at a reproducible test case 🤔 .

Thanks again for all of the hard work that has gone into Terraform v0.13 😬 .

EDIT:

I was thinking you need to see the hierarchy while I work on a reproducible test case:

I ran tree against various modules so you can see how this is made up from the root module.

https://gist.github.com/adamhorden/b437a4f39e563ae6340b72eda52adaa4

This is only 10%, I had to commented out all the other modules for testing but the pattern getting to the panic is still the same.

[apparentlymart]

Thanks! I think I'm going to re-add our "waiting-response" label here because I think your smaller reproduction case is likely to be needed to fully understand this, but someone else on the team who is more familiar with the new module processing might 🤞 be able to offer an explanation based just on the stack trace, if we're lucky! 😄

[adamhorden]

I have a test case @apparentlymart 💥 🤓 .

https://github.com/adamhorden/terraform_bug_002/tree/master/he_aws_tf_app_itf/he_aws_tf_mod_he_aws_tf_app_itf

There is a README.md, same as my last report with a full steps taken to get a panic.

One hierarchy down is full modules.

This will always create the panic I am reporting. I am starting to think that assume_role in proxy providers might be causing this 🤔 .

Hope this test case helps. It is extensive as a test case. Sorry I could not cut it down any more. It was racking my head for half a day how to come up with a repeatable test case.

[jurgenweber]

Looks like I am seeing the same:

Terraform v0.13.0-beta1
$ tf init
Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching ">= 2.52.0, ~> 2.53, ~> 2.53"...
- Finding hashicorp/kubernetes versions matching ">= 1.11.1"...
- Finding hashicorp/local versions matching ">= 1.4.*"...
- Finding hashicorp/null versions matching ">= 2.1.*"...
- Finding hashicorp/random versions matching ">= 2.1.*"...
- Installing hashicorp/aws v2.64.0...
- Installed hashicorp/aws v2.64.0 (signed by HashiCorp)
- Installing hashicorp/kubernetes v1.11.3...
- Installed hashicorp/kubernetes v1.11.3 (signed by HashiCorp)
- Installing hashicorp/local v1.4.0...
- Installed hashicorp/local v1.4.0 (signed by HashiCorp)
- Installing hashicorp/null v2.1.2...
- Installed hashicorp/null v2.1.2 (signed by HashiCorp)
- Installing hashicorp/random v2.2.1...
- Installed hashicorp/random v2.2.1 (signed by HashiCorp)

$ terraform validate
panic: no expansion has been registered for module.cluster

goroutine 1765 [running]:
github.com/hashicorp/terraform/instances.(*expanderModule).moduleInstances(0xc0021d0080, 0xc000a80720, 0x2, 0x2, 0xc00235eb50, 0x0, 0x4, 0x10ee3b8, 0x0, 0x32)
	/home/circleci/project/project/instances/expander.go:267 +0xb72
github.com/hashicorp/terraform/instances.(*Expander).ExpandModule(0xc0021d00a0, 0xc000a80720, 0x2, 0x2, 0x0, 0x0, 0x0)
	/home/circleci/project/project/instances/expander.go:116 +0x108
github.com/hashicorp/terraform/terraform.(*nodeExpandModuleVariable).DynamicExpand(0xc000e43dc0, 0x2face20, 0xc001c9c8f0, 0x1, 0x1, 0x1)
	/home/circleci/project/project/terraform/node_module_variable.go:42 +0x94
github.com/hashicorp/terraform/terraform.(*Graph).walk.func1(0x2916760, 0xc000e43dc0, 0x0, 0x0, 0x0)
	/home/circleci/project/project/terraform/graph.go:84 +0x4d8
github.com/hashicorp/terraform/dag.(*Walker).walkVertex(0xc0006541e0, 0x2916760, 0xc000e43dc0, 0xc0029c9bc0)
	/home/circleci/project/project/dag/walk.go:387 +0x357
created by github.com/hashicorp/terraform/dag.(*Walker).Update
	/home/circleci/project/project/dag/walk.go:309 +0x11d7

edit: added full crash log.

crash.log

[ipmb]

Here's the closest I got to a minimal test case, https://github.com/ipmb/terraform-issue-25114

The VPC module here is a copy of https://github.com/terraform-aws-modules/terraform-aws-vpc/ with the version limitation removed to workaround terraform-aws-modules/terraform-aws-vpc#453

[ficofer]

Team,

We are experiencing the same/similar issue with one of our modules (using terraform version v0.13.0-beta1)

Here is the full crash.log https://gist.github.com/ficofer/3350a93db889f2aa64b9c8b41a2590c4

This is the short output:

2020/06/04 19:18:09 [DEBUG] Starting graph walk: walkValidate
2020/06/04 19:18:09 [TRACE] dag/walk: visiting "module.prometheus.module.thanos_query_cluster.local.tcp_protocol (expand)"
2020/06/04 19:18:09 [TRACE] vertex "module.prometheus.module.thanos_query_cluster.local.tcp_protocol (expand)": starting visit (*terraform.nodeExpandLocal)
2020/06/04 19:18:09 [TRACE] vertex "module.prometheus.module.thanos_query_cluster.local.tcp_protocol (expand)": expanding dynamic subgraph
2020/06/04 19:18:09 [TRACE] vertex "module.prometheus.module.thanos_query_cluster.local.tcp_protocol (expand)": visit complete
panic: no expansion has been registered for module.prometheus

goroutine 263 [running]:
github.com/hashicorp/terraform/instances.(*expanderModule).moduleInstances(0xc0005da320, 0xc000360280, 0x2, 0x2, 0xc000855b20, 0x0, 0x4, 0x7eb68b0, 0x10, 0x4)
	/home/circleci/project/project/instances/expander.go:267 +0xb72
github.com/hashicorp/terraform/instances.(*Expander).ExpandModule(0xc0005da340, 0xc000360280, 0x2, 0x2, 0x0, 0x0, 0x0)
	/home/circleci/project/project/instances/expander.go:116 +0x108
github.com/hashicorp/terraform/terraform.(*nodeExpandLocal).DynamicExpand(0xc000ac6300, 0x2face20, 0xc000a64000, 0x1, 0x1, 0x1)
	/home/circleci/project/project/terraform/node_local.go:70 +0x94
github.com/hashicorp/terraform/terraform.(*Graph).walk.func1(0x28d24a0, 0xc000ac6300, 0x0, 0x0, 0x0)
	/home/circleci/project/project/terraform/graph.go:84 +0x4d8
github.com/hashicorp/terraform/dag.(*Walker).walkVertex(0xc000178c60, 0x28d24a0, 0xc000ac6300, 0xc00065c000)
	/home/circleci/project/project/dag/walk.go:387 +0x357
created by github.com/hashicorp/terraform/dag.(*Walker).Update
	/home/circleci/project/project/dag/walk.go:309 +0x11d7

Adding more information here:

This is the crash log when I run one of the inner modules called from the parent module that failed above.

https://gist.github.com/ficofer/e0179331b4a904bdc833454f88219fa8

We discover that this module is the one failing calling other two submodules, moduleA and moduleB

If I comment out moduleA and leave moduleB it works.
If I comment out moduleB and leave moduleA it works.

If I left both of them together it fails, it's worth noting that the modules does not have any dependent variables in between each other.

Great work team!
Thanks!

[debovema]

First of all thanks for all the great work being done on 0.13 and thanks to the people in this thread giving diagnostic information.
We also have this bug while we're preparing the migration of our modules at our company. I don't want to pollute this issue but just let you know that this is really a show stopper on our side.

[mrobinet]

Also hitting a similar issue. I can provide additional crash logs if needed.

[danieldreier]

@ipmb I have reproduced this using your reproduction case, which I copied into my own repo (https://github.com/danieldreier/terraform-issue-reproductions/tree/master/25114).

Based on this, I think we have enough detail to work on this. Thank you @adamhorden @ipmb and everybody in this thread. I am so happy to have you all trying out this code, finding bugs, and figuring out reproduction cases. Thank you!

[amg1127]

Hi. I believe that the crash I am experiencing in my code is related to this bug report, because the panic message looks similar.

This Bash command sequence:

$ git clone -b terraform-v0.13beta1-test https://github.com/amg1127/MyCloudInfrastructure
$ cd MyCloudInfrastructure
$ terraform init
$ terraform plan

triggers the following Terraform crash:

panic: no expansion has been registered for module.prodInfrastructure

goroutine 560 [running]:
github.com/hashicorp/terraform/instances.(*expanderModule).moduleInstances(0xc00046b600, 0xc00009c400, 0x2, 0x2, 0xc0008bfb20, 0x0, 0x4, 0xc000bf0000, 0x93, 0x50000)
	/home/circleci/project/project/instances/expander.go:267 +0xb72
github.com/hashicorp/terraform/instances.(*Expander).ExpandModule(0xc00046b620, 0xc00009c400, 0x2, 0x2, 0x0, 0x0, 0x0)
	/home/circleci/project/project/instances/expander.go:116 +0x108
github.com/hashicorp/terraform/terraform.(*nodeExpandLocal).DynamicExpand(0xc00047d3e0, 0x23ae440, 0xc00062d110, 0x1, 0x1, 0x47d001)
	/home/circleci/project/project/terraform/node_local.go:70 +0x94
github.com/hashicorp/terraform/terraform.(*Graph).walk.func1(0x1cd4400, 0xc00047d3e0, 0x0, 0x0, 0x0)
	/home/circleci/project/project/terraform/graph.go:84 +0x4d8
github.com/hashicorp/terraform/dag.(*Walker).walkVertex(0xc00050c720, 0x1cd4400, 0xc00047d3e0, 0xc000797540)
	/home/circleci/project/project/dag/walk.go:387 +0x357
created by github.com/hashicorp/terraform/dag.(*Walker).Update
	/home/circleci/project/project/dag/walk.go:309 +0x11d7



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain 
sensitive information that must be redacted before it is safe to share 
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

The complete crash.log file is published here: https://gist.github.com/amg1127/f17c8ae7f6fb3dee6820df5edd6402ff

The Terraform version is:

$ ../terraform -v
Terraform v0.13.0-beta1
+ provider registry.terraform.io/hashicorp/null v2.1.2
+ provider registry.terraform.io/hashicorp/random v2.2.1
+ provider registry.terraform.io/terraform-providers/docker v2.7.1

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.