Using CDK to manage multiple Stacks generated separately ?

[warlockdn]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Working with CDK has been very helpful recently. But I have come across a use case and need to understand how would you approach it.

I have a simple configuration application which in-turn uses CDK to generate Terraform File works perfectly.
But here are some limitations I am not able to understand.

Let's suppose I have multiple Infra configuration for multiple projects. Now I have a simple config which I pass on to the CDK as parameters and based on the passed config params I would generate a stack with a unique-name. Then run deploy and the infra would provision and my rest of the things work out.

Every time there is a new project I would need to generate stack with a unique-name and provision new infra based on the config params using the underlying module that used CDK.

But the issue is every-time I would run synth all the stacks would delete the generated stack (not the actual infra) and run again.

What would be your ideal workflow here. How do I keep using the same module (CDK) to keep generating stacks based on incoming config params. And also make sure if there are any changes in the already generated stack it can take care of the diff and do the provisioning or deletion.

Would really like to know your opinions.

References

[jsteinich]

Are you using local state backend or a remote backend? If you are using local state and the terraform state file for the previous stack is being deleted that sounds like it may be a bug.

If just the synthesized json is being deleted, that sounds like a potential improvement area. The trade off with not deleting would be potentially having old stacks (that a user potentially wanted to remove/rename) pile up.

I think multiple stack support was added assuming stacks would be more static, but I'm not thinking of any strong reason why they couldn't be dynamically created.

In order to update existing stacks, you'll need to make sure to pass configuration that will generate the same stack name. If you are using a remote state backend, then it doesn't matter if any local files are present. The diff and deploy will still be possible.

[warlockdn]

Are you using local state backend or a remote backend?

Local for now. But I want to move to Terraform Cloud once I am able to run this POC

I am not sure what's the issue here. Whenever I do synth old generated stack gets deleted.

How can I stop this from happening. stacks and manifest.json is completely replaced every time I run synth.

How should I go about this. My idea is to generate json for multiple projects with multiple environments. And only source of truth needs to be the one where I am using the CDK.

[ansgarm]

Hi @warlockdn!
This is happening by design, we treat the source code (e.g. in TypeScript) as the single source of truth. If you want the cdk.tf.json file to be the source of truth, you could e.g. commit it to your Git repository to keep a history of changes to that file.

However for your use case I'd suggest an alternative. You could create multiple instances of the same stack for your different projects. Like in this example:

terraform-cdk/examples/typescript/aws-multiple-stacks/main.ts

Lines 31 to 33 in f43270e

	new MyStack(app, "multiple-stacks-dev", { environment: "dev" });
	new MyStack(app, "multiple-stacks-staging", { environment: "staging" });
	new MyStack(app, "multiple-stacks-production-us", {

(You can also do this for different projects – not just for different environments)

[warlockdn]

@ansgarm

The scenario that you are suggesting is by creating and running things at once. But my use case is where configuration don't come to different projects at once. Configurations can be generated at any given time for any project.

So for eg:
Project1 comes in and sends params: 1gb ram, 2 core cpu and source code
Project2 comes in at a separate time with: 2gb ram, 2 core and source code

Now by design cdktf would delete the last generated terraform. Now since all this would be a part of an automated process it becomes difficult to maintain git commits and run back and get the json and stuff. Hope you get my point.

Can this be taken as a use case and can be used to alter the cli to accomodate the change may be using some config change. May be a change in cdktf.json using deleteonsynth: boolean or something may be.

[warlockdn]

The idea behind using a CDK would be for many I suppose would to manage multiple infrastructure requirements. Now this way anyone can be truly an infrastructure as code (since we are generating infra based on received parameters). Now involving multiple infra would enable generating and maintaining much more complex and very exhaustive configuration detailing.

So instead of thinking other directories already generated as orphaned we could look at them as multiple projects. This way states and other things can be part of that single directory where the plan would take place. And that and only that directory becomes the source of truth for that particular project. (This is only while thinking things are local)

[ansgarm]

Ah, sounds more like you want to build some kind of self-service that offers an API where projects can request infrastructure, is that right?

I'm very hesitant with something like deleteonsynth: boolean because that could leave old artifacts inside the cdktf.out directory if you e.g. rename a stack or remove an Asset.

But that doesn't make your underlying use case invalid, so maybe we can find another way for the CDK for Terraform to support that (which might need some additional design work and deeper thoughts first before we incorporate it into the CLI itself).

So, to better understand your problem first, I'd like to dig a bit deeper (if that's okay with you).

You mentioned an example:

Project1 comes in and sends params: 1gb ram, 2 core cpu and source code
Project2 comes in at a separate time with: 2gb ram, 2 core and source code

How does the project send those params? And are they stored somewhere or just sent one-off?
How does the CDKTF itself run when it receives them? How is it invoked? Or is Terraform invoked directly instead?

Even if the use case is not supported natively by the CDKTF at the moment, I'm pretty sure that it could be achieved with a bit of scripting inside a CI that might run all of this. There are users of the CDKTF which synthesize the Terraform config without the cdktf-cli but by invoking the program defined in cdktf.json > app directly. You could do a similar thing where you take control of the cdk.tf.json output file and move it into a specific directory based on the project it was generated for and deploy from there using Terraform or Terraform Cloud.

[warlockdn]

@ansgarm you got my use case right.

So since cdk supports typescript I have build the whole thing as a submodule inside my main app which would takes in a config params containing the necessary stuff to generate a infra around it.

interface AppConfig {
  domain: string;
  type: 'backend' | 'ui';
  appName: string; // unique app name
}

Now this is passed to the appropriate app which looks like

constructor(scope: Construct, name: string, config: AppConfig) {
    super(scope, name);
}

Now by using exec or zx I am able to run synth I am able to create a stack

new AppStack(app, appName, config);
app.synth();

Just a very straight forward thing I am doing here. Now since this is a cdk it would generally allow a developer writing business logic and other validations to allow this thing to be deployed to any cloud provider right. That's pretty much my idea.

You could do a similar thing where you take control of the cdk.tf.json output file and move it into a specific directory based on the project it was generated for and deploy from there using Terraform or Terraform Cloud

Yes right I can go ahead and do that as well. I can depend on cdktf to do synth and move the stack to my own user directory or terraform cloud. Right now its a straight forward deal (For my POC) -> synth -> deploy

I understand your concern as well it could delete the old stack if the same name is passed again but same should or can even throw and error if in case the directory and the stack name exists already in the manifest.json right ?

What according to you I can try to make this work even more easier to achieve. I see a solution you are suggesting

1. synth a new stack
2. transfer the stack to a new directory all together
3. let terraform cli handle plan and deploy

Anything else should I be doing ?

[ansgarm]

Thank you for explaining your use case in more detail 👍 That helped a lot!

To me your proposed approach (using Terraform CLI to deploy) is reasonable for now. While we have thought about this use case / workflow before, we have not yet gathered sufficient data on it, so we cannot recommend a "single best way" to solve this, yet. But after gathering some more data on this workflow I'm pretty sure that we'll at some point be able to make a recommendation for how to best handle this.

On a related note: We plan to offer an API which exposes the underlying invocation of Terraform: #804
However, that won't be shipped shortly, so I'd recommend to invoke the Terraform CLI yourself for now.

Please feel free to post more feedback to this issue as you go along with your POC! That would be very valuable to us ❤️

[warlockdn]

@ansgarm thank you for the validation on this.

I have one case that I would like to mention. The current use case seems to be working when the infra is generated once. What happens when we modify the same config to update the infra with latest changes. No my plan is to use terraform cloud for remote state. How all this would pan out when using terraform cloud.

I will try and post small poc on this. But what are your views on this. Do you think the use case can be taken as an initial idea and be discussed with core team and check if it means any sense to be taken for further dev.

Thanks.

[ansgarm]

Do you think the use case can be taken as an initial idea and be discussed with core team and check if it means any sense to be taken for further dev.

Definitely, this workflow will be something we will think about 👍

[warlockdn]

I would definitely be looking forward for this. Thank you

[jsteinich]

What happens when we modify the same config to update the infra with latest changes. No my plan is to use terraform cloud for remote state. How all this would pan out when using terraform cloud.

Until you ran again with parameters that match the original stack name, nothing would happen. Once that stack is known to be needed, it will update it.

[skorfmann]

What happens when we modify the same config to update the infra with latest changes. No my plan is to use terraform cloud for remote state. How all this would pan out when using terraform cloud.

Until you ran again with parameters that match the original stack name, nothing would happen. Once that stack is known to be needed, it will update it.

Yes, as long the input config stays the same, the output will be identical for the given input. So, in theory your described use-case should be working already. You'd essentially generate one stack at a time, with changed configurations.

Regarding the TF Cloud backend, I’d try to use an API client for TF cloud (e.g. https://github.com/mijailr/terraform-cloud) to make sure a workspace exists and pass this as config into the stack. Generally, that’s something which is one of the use-cases we have in mind for #682