Split infrastructure into components

[danlester]

qhub deploy currently splits the infrastructure deployment into stages using Terraform resource targeting.

This is 'wrong' according to Terraform, and we should consider splitting it up into separate configurations instead and rolling them out in order.

If we had clear separate configs then this would also mean we can destroy in reverse order, and I think this might help with our timeout problems on tear down. (We could technically do this with target anyway, but only if all remaining targets are listed explicitly - our final deploy step just runs everything, including the initially-targeted items too.)

From HCL docs:

Once Terraform has selected one or more resource instances that you've directly targeted, it will also then extend the selection to include all other objects that those selections depend on either directly or indirectly.

This targeting capability is provided for exceptional circumstances, such as recovering from mistakes or working around Terraform limitations. It is not recommended to use -target for routine operations, since this can lead to undetected configuration drift and confusion about how the true state of resources relates to configuration.

Instead of using -target as a means to operate on isolated portions of very large configurations, prefer instead to break large configurations into several smaller configurations that can each be independently applied. Data sources can be used to access information about resources created in other configurations, allowing a complex system architecture to be broken down into more manageable parts that can be updated independently.

[dhirschfeld]

I found terraform support for large deployments pretty unworkable myself which is why I moved to using terragrunt - this use-case seems to be it's raison d'etre.

With terragrunt, dependencies between modules can be explicitly specified:
https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#dependency

[danlester]

Note I have attempted to improve qhub destroy using terraform targets in the meantime: #948

[costrouc]

Going to close this issue since it is implemented in #1003. We should definitely look into using terragrunt. Since right now the outputs from each module are being captured in python and being explicitly passed in via a var-files.