Problem with Terraform module dependencies (etcd, wireguard)

[PythonicNinja]

With current master and basic setup using:
hcloud - provider
cloudflare - dns

module.etcd.null_resource.etcd[2]: Provisioning with 'remote-exec'...
module.etcd.null_resource.etcd[2] (remote-exec): Connecting to remote host via SSH...
module.etcd.null_resource.etcd[2] (remote-exec):   Host: 159.69.2.36
module.etcd.null_resource.etcd[2] (remote-exec):   User: root
module.etcd.null_resource.etcd[2] (remote-exec):   Password: false
module.etcd.null_resource.etcd[2] (remote-exec):   Private key: false
module.etcd.null_resource.etcd[2] (remote-exec):   SSH Agent: true
module.etcd.null_resource.etcd[2] (remote-exec):   Checking Host Key: false
module.etcd.null_resource.etcd[1] (remote-exec): Connected!
module.etcd.null_resource.etcd[0] (remote-exec): Connected!
module.etcd.null_resource.etcd[2] (remote-exec): Connected!
module.etcd.null_resource.etcd[1] (remote-exec): disabled
module.etcd.null_resource.etcd[1] (remote-exec): Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /etc/systemd/system/etcd.service.
module.etcd.null_resource.etcd[0] (remote-exec): disabled
module.etcd.null_resource.etcd[0] (remote-exec): Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /etc/systemd/system/etcd.service.
module.etcd.null_resource.etcd[2] (remote-exec): disabled
module.etcd.null_resource.etcd[2] (remote-exec): Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /etc/systemd/system/etcd.service.

Error: Error applying plan:

3 error(s) occurred:

* module.etcd.null_resource.etcd[2]: error executing "/tmp/terraform_2090074829.sh": Process exited with status 1
* module.etcd.null_resource.etcd[0]: error executing "/tmp/terraform_1290618796.sh": Process exited with status 1
* module.etcd.null_resource.etcd[1]: error executing "/tmp/terraform_423106603.sh": Process exited with status 1

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

I logged into one of the instances and there seem to problem:

root@kube1 ~ # /tmp/terraform_1498877724.sh
enabled
Job for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details.
root@kube1 ~ # systemctl restart etcd.service
Job for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details.
root@kube1 ~ # systemctl status etcd.service
● etcd.service - etcd
   Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
   Active: activating (start) since Sun 2018-06-17 20:49:33 CEST; 5ms ago
 Main PID: 2719 ((etcd))
    Tasks: 0
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/etcd.service
           └─2719 (etcd)

Jun 17 20:49:33 kube1 systemd[1]: Starting etcd...
root@kube1 ~ # systemctl status etcd.service
● etcd.service - etcd
   Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Sun 2018-06-17 20:49:38 CEST; 2s ago
  Process: 2725 ExecStart=/opt/etcd/etcd --name kube1 --data-dir /var/lib/etcd --listen-client-urls http://10.0.1.1:2379,http://localhost:2379 --advertise-client-urls http://10.0.1.1:23
 Main PID: 2725 (code=exited, status=1/FAILURE)

Jun 17 20:49:38 kube1 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Jun 17 20:49:38 kube1 systemd[1]: Failed to start etcd.
Jun 17 20:49:38 kube1 systemd[1]: etcd.service: Unit entered failed state.
Jun 17 20:49:38 kube1 systemd[1]: etcd.service: Failed with result 'exit-code'.

I assume it's issue with networking between nodes.

[PythonicNinja]

Hello @pstadler!
Do you have any idea what might be the problem here?
Thanks!

[pstadler]

Terraform did it again. This is a long lasting issue with dependencies between modules. Every once in a while things stop working. With the current version etcd is getting created before the wireguard module finishes.

They promise to fix this in a future version. See hashicorp/terraform#18239

Unfortunately, I couln't find a workaround so far.

Theoretically you should be able to run terraform apply a second time to complete the setup.

[dumrauf]

@pstadler, there may be a hackaround for that so Terraform finishes in one go...

For the depends_on problem, the only viable workaround, which I've found to work, is to

1. output attributes of relevant resources created in the upstream module and
2. use the outputs as input variables in the downstream module/resource.

Here, the catch is that you

1. have to actually use the variables in the downstream module/resource or they will get "optimised" away
2. need to get the dependencies between resources in the upstream and downstream module/resource right or you may end up with race conditions (this may may also depend on how modules are implemented under the bonnet which I have no insight into)

It's not the best solution in the world but it does the job for now until this issue gets resolved. Please let me know if anyone has a better solution!

[pstadler]

It's really frustrating. I've been fighting with this since the beginning.

As you can see in main.tf, there are indeed output variables used as input across all modules. This has been working until some recent Terraform release and I can't really see how to hack around this problem.

[rm-rf-etc]

What was the last version of Terraform which did not have this issue?

[pstadler]

I'm quite sure it broke after 0.11.3 or 0.11.5.

[rm-rf-etc]

Confirmed, I don't see this issue in 0.11.3. Haven't tested yet with 0.11.5. Perhaps then this should be noted in the readme. The process for brew:

cd "$(brew --repo homebrew/core)" && git checkout 9cf4ffd17c615e143e8ba39a254ee2d116f1587c
HOMEBREW_NO_AUTO_UPDATE=1 brew install terraform

[pstadler]

Just upgraded Terraform to 0.11.8. Seems to be working again.

[lbialy]

I have encountered this on 0.11.10 😢

[endermAH]

I'm using Terraform 0.12.16 and have the same problem...

[hamdiakrimi]

0.11.11 and have the same problem :/

[pavelzagalsky-paloalto]

Still seeing this on Terraform v0.12.25