Explore Terraform CDK

[aktech]

https://www.hashicorp.com/blog/cdk-for-terraform-enabling-python-and-typescript-support

[viniciusdc]

Some updates

CDK for Terraform generates Terraform configuration to enable provisioning with Terraform. The adaptor works with any existing provider and modules hosted in the Terraform Registry. The core Terraform workflow remains the same, with the ability to plan changes before applying.

• CDK support, programming languages such as Python and TypeScript

The CDK for Terraform project includes two packages:

• “cdktf-cli” - A CLI that allows users to run commands to initialize, import, and synthesize CDK for Terraform applications.
• “cdktf” - A library for defining Terraform resources using programming constructs

Terraform CDK generates a Terraform configuration in JSON, based on a main file, that can be applied with terraform apply or using the CDK for Terraform CLI with cdktf deploy.

To initialize a new project or specify a new template you can type: (providing the description and project names are option, but dispense the interactive mode)
cdktf init --template=<python/typescript> --local --project-name='cdk-example' --project-description=''

Several steps occur when running cdktf init command:

• Downloads a sample project with the template specified, defined in main.(ts/py).
• Configures Terraform Cloud organization and workspace if specified.
• Runs terraform init to download the appropriate Terraform providers and modules, or run cdkts get. It downloads the specified providers in cdktf.json if they exist.
• Generates language-specific objects for resources and modules under the .gen folder based on the provider and module schemas.
• After running cdktf init, the example directory contains the resources for use.

├── .gen
│   └── providers
│       └── ...
│        │   └── modules
├── .terraform
├── cdktf.json
├── help
├── main.(ts/py)
├── package.json
└── tsconfig.json (or pyconfig.json)

Here you can check a similar initial deploymet I did for testing: https://github.com/viniciusdc/terraform_cdk/tree/main/cdk-qhub-example

Based on the modifications specified in the main file and related sources files, the next stet is to synthesize it to Terraform configuration by running cdktf synth. The command generates Terraform JSON configuration files in the cdktf.out directory.

To deploy the generated configuration, we can use the Terraform CLI commands to provision infrastructure resources or the general Terraform syntax:

cd cdktf.out
terraform plan
terraform apply
terraform destroy

The general configuration and necessary docs I found so far include, how to use the provides and modules, handle outputs, variables.

The next steps are:

• Add new variables/modules definitions into main.py to generate the correct files;
• Get Google Provider configuration working in the test repo;
• Generate the correct workflow for QHub HPC based on it Terraform infrasctructure
• Based on the previous discussion, the deployment workflow will be compatible with the current one for Terraform.
  @tylerpotts @aktech

[costrouc]

I spent quite some time researching this issue. And I'd like to turn this into a discussion.

Some background on why this issue is so import to QHub. Qhub has several stages:

1. Render the terraform files via cookiecutter
2. Call terraform init/validate/plan/apply on the resulting directories
3. These terraform files then call terraform modules that we have developed within QHub
4. These modules try to provide a standard way to deploy kuberentes, registrties, ingress, and much much more on all cloud providers. There include cloud infrastructure, helm charts, and kubernetes resources.

We have been working hard to support many features within our terraform modules:

• gpu support on all cloud providers
• auto scaling kubernetes clusters
• spot/premptible instances
• custom environments
• optional extensions e.g. prefect
• tight coupling between dask/jupyterhub/jupyterhub-ssh/authentication/ingress

We have faced many issue with this design:

• onboarding new developers is hard. this is due to the many stages in Qhub shown above not to mention that cookiecutter is extremely sensitive to whitespace etc. and is not that flexible.
• terraform is not flexible enough hence why we have been using cookiecutter to enable digital ocean/gcp/aws via optional blocks in cookiecutter
• there are too many layers within this deployment model making it very difficult to expose ways for end users to tinker with their deployment

Doing enterprise deployments of QHub have really made this issue tough to deal with. All customers have different needs, use different clouds, prefer to use an existing vpc etc. Some of the issues related to this:

• [enhancement] Enable custom resource options in qhub-config #913
• [EPIC] Nebari Extension Mechanism #865
• Make load balancer configurable [enhancement] #638
• Support for annotations on ingress resource with Terraform #452

Point is there are a long list of issues that relate to the customizability and extensibility of QHub. Having the ability to write the resources in python has huge advantages and there are several projects that do this. These include:

• terraform-cdk
• pulumi
• terraformpy

I think the main argument around using something like this is that it makes the infrastructure files now dependent on a specific language.

I've looked at all three and let me briefly include my thoughts pros/cons:

• terraform-cdk is managed by hashicorp and has multilanguage support. But has a complex dependency stack which makes installation tricky. It would enable proper type checking on terraform modules. the fact that it is developed by hashicorp is a big boost to this project
• pulumi another project with multi language support which has a complex dependency stack. It can reuse any terrraform providers but would require a bit of code rewrites
• terraformpy is a small library with minimal dependencies but is not as actively developed as the others. However I don't think it needs additional features since it addresses what we need.

All of these could solve the problems that we have. I however have enjoyed terraformpy. Here is an example of how I envision it extending qhub. In the sort term we could eliminate the use of cookiecutter. Later in the future I believe we could stop using terraform modules all together.

See example https://github.com/costrouc/terraformpy-qhub-example

[costrouc]

Closing since we have a viable solution that does not require terrafirmcdk with #1003.