Making cdktf dynamic

[jsteinich]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

I'm starting to see two main groups of users:

1. Those that see and use cdktf as a terraform configuration language generator
2. Those that are looking to expose the full power of terraform through a programming language of their choice

For the 1st group, things mostly work as expected. There are a few bugs, not all hcl features are available, and there aren't yet higher level abstractions. This makes sense since cdktf quite literally generates the terraform configuration language json variant.

The 2nd group experiences quite a few issues since other than provided information, nothing is actually available at runtime. I think this is what someone without a CDK background or deeper knowledge of cdktf would expect. I also think that it is a completely reasonable expectation. That said, I would be very surprised if the terraform team actually had plans to control the terraform runtime via external languages (think replacing the jsii runtime with terraform).

Assuming an actual terraform runtime is out of the question (seems like it), then I think we need to look for ways of emulating it:

One possible avenue is analyzing the syntax tree of the user code and converting to terraform json. This would basically be a very smart generator (still static configuration). A for loop could get translated into a count or a for each, etc. I have no doubt we could get a few simple cases working, but further support could be a very large and brittle project, not to mention that it would quite likely break down in more complex scenarios.

Having access to the multitude of functions built into hcl would go a long ways. These are commonly what are used in hcl to go beyond simple references and make it feel more dynamic. This is also sort of tangential to my main points, but since I brought it up, we should figure out mimicking intrinsic functions from AWS CDK is the way to go, or if having an order of magnitude more functions (+ arithmetic operations) means that a different approach is needed.

One of the most common areas that have common up is actually how to access a list (or set, or map, or complex object). We have Tokens which partially make that work. They are a bit buggy, so that doesn't help anything, but they are also incomplete (and in some ways never can fully handle possible goals). There's also a special case method for accessing a particular index from a list. Generating full wrapper types that track access and convert to for each loops and count index loops in terraform might get us far enough here. Admittedly, something as simple as printing a value or accessing the length in an integer for loop would still not work. Side note, there's also a bug where computed types don't even generate all of their properties.

Another possible route is building custom code to simply call various sdks during run time (synth time) to make information actually available. This really feels like building out functionality that terraform already has though. I can imagine building an intermediate layer that is made up of just data sources and ran through terraform. However, even data sources aren't always that simple. They can have dependencies on resources being built and such so that weren't starting to venture into build terraform outside of terraform in order to call terraform.

I'd like to look towards AWS CDK for some guidance; however, terraform is by nature more dynamic than cloud formation, so we may need to forge our own path on a lot of this. A lot of thought and effort needs to go into solving these problems, but I'm confident the end result can be something very powerful and extremely useful.

This isn't meant to be a spec or a rant; just a brain dump that can hopefully serve as a starting point for some design conversations. I truly want this project to succeed. We just need to ask ourselves these hard questions to push the design forward.

References

• Retrieve Resolved Properties During Synthesis #423
• Iterating computed values #430
• Complex Computed Optional Types #25
• Add Computed List Accessor #288
• Built-in Function Compatibility #220
• Enable Attribute References for Nested Types #172
• Computed Lists and Maps #42
• ContainerPorts in Docker Provider Container getter returns string identifier rather than container ports  #434
• aws_acm_certificate.*.domain_validation_options throws Invalid index #424
• How to get the list of AZ's of a specific region #374
• Nested Computed Attributes for Block Types #29
• All Module Outputs Typed as String #287
• Token asAny / Intrinsic value doesn't work as expected #256
• Emulate Native Types like Lists and Dictionaries via Proxy Objects aws/jsii#1841
• Support for language specific iterators aws/jsii#1302
• Document array behavior aws/jsii#746
• Consider adding support for Set<T> aws/jsii#1

[skorfmann]

Well, first and foremost cdktf is a generator for HCL compatible JSON and relies on Terraform for applying the configuration. As you correctly stated, making it really dynamic would involve customizing the Terraform runtime.

Besides from this, a few bugs and other little issues, I think there are three major limitations which we have:

1. jsii limits Typescript capabilities to a commonly supported feature set across all target languages
2. Synth is a synchronous operation and makes it difficult to use most SDKs for data fetching (in TS / JS at least)
3. Complex operations (loops, functions, etc) aren't abstracted

I believe all points can be addressed, but need thorough thinking before implementation.

[cmclaughlin]

I'm happy with cdktf as a simple Python to HCL bridge... that's a major improvement for me and I'm very thankful for this project as it is.

However, interacting with Terraform dynamically would be amazing. When I first got started with cdktf, that is what I expected. But it didn't take me long to understand what the constraints are and how to work with them.

Previously I experimented with Pulumi, but didn't get very far with it - I couldn't get past a few bugs that I thought were serious. However, their backend design seems really good - I think they wrote some sort of RPC server written in Go that provides an interface to Terraform, so the various client programming languages (Python, Typscript, etc) can interact with Terraform directly at runtime.

[skorfmann]

I think they wrote some sort of RPC server written in Go that provides an interface to Terraform, so the various client programming languages (Python, Typscript, etc) can interact with Terraform directly at runtime.

Haven't looked at this in detail - but my understanding so far was, that they're using the Terraform providers via that rpc protocol and wrote their own engine.