What is the best way to run import with terragrunt?

[flmmartins]

Hello All,

I really like terragrunt =) and one of the reasons I like it is the plan-all and the dependency declaration in terraform.tfvars.

Unfortunately not all things can be created by Terraform and sometimes we need to import them... and the carry on with our terragrunt execution.

My flow would be like:
terragrunt apply-all
terragrunt import
terragrunt apply-all

What is the best way to accomplish this?

One way we could do it is by not using the -all command and build a shell that enters in each folder and does the plans & imports but that implies in loosing the -all functionality.

What would be the best way to do this with terragrunt?

[yorinasub17]

Can you elaborate a bit on your use case? E.g what would be an example of a module that is applied in the first apply-all, an example of a resource that needs to be imported, and an example of what would be applied in the second apply-all? I think this would help in recommending the right structure + approach to solve your problem.

On a side note, we generally discourage usage of the xxx-all variants on a day to day basis for various reasons. See #720 (comment) for some of the reasons (that response is specific to plan-all, but applies generally).

[flmmartins]

Okey.. it seems good arguments but I have another question: if I DON'T use -all then what would happen if I ran a terraform.tfvars that has a dependency with another module? Will it request for me to run that module first?

The only reason I'm rellying in -all variants is due to dependency = [] block. Is there a way to ensure this dependency execution without -all variant?

[yorinasub17]

Ah, in that case, no the dependency blocks are only run in the -all variants, so if you need to rely on them, the -all variants help with that. I still don't think you need to rely on this for day to day operations (see below if you are interested in why), but that is not really the point of this thread, because your question is still valid for use cases of the -all variants of the command, such as the initial setup of the infrastructure.

So given that, can you provide some more context on the infrastructure itself? What is a module that needs to be applied first, and then another module that needs to be imported before the third module can be applied?

I am thinking that you might be able to leverage the before hooks to optimistically import infrastructure in the module that needs to do it before the call to apply happens. But it is a bit hard to recommend options without having a concrete use case to work off of.

---

To elaborate a bit more on my point about daily operations and -all:

In most cases, you should only need to rely on the dependency ordering when you are initially setting up your infrastructure. Once your infrastructure is stood up, the cases where you need to touch your entire infrastructure should be rare. We have found that it is usually much safer to implement and update your changes one module at a time.

Here is a toy example to explain the thought process: let's suppose you were deploying a VPC and a database into it. You might structure your infrastructure folder as follows:

live
├── sql
│   └── terraform.tfvars
└── vpc
    └── terraform.tfvars

You would then specify the vpc folder as a dependency to the sql folder, so that when you do a terragrunt apply-all, it would deploy the vpc before deploying the sql folder. When you are initially deploying the infrastructure, you would use terragrunt apply-all so that all the infrastructure gets stood up, in the correct order.

Now let's consider what you might change in the infrastructure. What would be changes that you might make? Which modules would be affected by those changes?

Chances are, you would rarely ever need to touch the VPC module to make most of your changes. For example, you might want to deploy an app that uses the database. For such a change, you would add a new module, and most likely the existing VPC and SQL modules will suffice.

Or maybe you want to modify the port that the db uses. In this case, you need to change the DB to update the port it is listening on, and the app. In a naive deployment with downtime, you can make the requisite changes and apply them using apply-all. There is nothing wrong with this if you can afford the downtime and the dependency block will help.

But what if you wanted to do it with limited downtime? In that case, you will first deploy a replica of the DB listening on a different port. You will want to make sure it is deployed correctly and tested before touching the app, so you would make the changes to the sql module first. Then, once you verify the replica is working correctly, you would deploy a canary/blue-green deployment of your app that uses the replica on the new port. Once you confirm the replica works, you can failover the DB to the new one and slowly transition traffic to the new version of your app.

In the latter scenario, you would be making your changes one module at a time instead of simultaneously to the DB and the app, despite the change affecting both modules. Because of this, you would most likely not use the -all variant, implicitly addressing the dependency as part of the rollout process.

So really, the usage of -all depends on your infrastructure needs and your deployment process, but for vast majority of production deployments, you don't really need to touch all your infrastructure all the time.

I also go back to this question when I find a use case reaching for -all all the time: should they be different modules? If I am always using plan and apply on all the infrastructure that needs to be targeted in -all, should I really have this layer of indirection and have them separate modules? Or should they actually be managed as one module and avoid all the potential complexities and gotchas of plan-all?

[flmmartins]

So in our case we have 2 main situations where we use the dependency declaration.

1. I use a lot of null_resources for example: a kubernetes commands that creates a load_balancer. And I can only run the rest of the infra once this command is created. Even if that wasn't the case separating by modules help the code to be organized so you know you need to run load_balancer and then install the application... but unfortunately you cannot link this two with terragrunt interpolation... so you need to rely in the dependency for that.

2. Dependency between modules (check issue) that chase me in my sleep. Terraform does not support it and I have a lot of modules that depends on each other....

So before I used to do something like:

module "create-role-and-attach-policy" {
...
}

module "id-doc-encryption" {
    owner = ${create-role-and-attach-policy.arn}
}

That seems fine when you look at it but the role needs a specific policy before the module can actually be run. Sometimes the module runs when ARN is created but not necessarily when user has permission for that actual owner-link. Therefore AWS causes an error....

Now I do it like this:

Example:
├── kms_role -> here I create roles and attach policie
│   └── terraform.tfvars
├── app_storage
│   └── terraform.tfvars -> depends on kms_role

In app_storage:

data "aws_iam_role" "kms_role" {
  name = "${var.kms_role}"
}

module "id-doc-encryption" {
  source = "../kms_encryption_key"
  ....
  key_owner = "${data.aws_iam_role.kms_role.arn}"
}

Since modules don't handle dependencies (check the issue) with depends_on I use dependency via terragrunt to explicit declare it.

But you are right -all variants are used mostly first time but also I want to make something consistent because sometimes people in my team change something in a module that affect another and running the -all variantes gives FULL perspective. I was also glad with -all cause I didn't needed to do a shell script for it.

A shell script with the execution order can be confusing for people that change terraform but are not very savy with it... For a person that is learning terraform... terraform in itself can be challenging.. you put a shell script with execution order and then things are more complex so we are all very glad with -all variant (except that it doesn't support landscape but this is another issue)

Why do I want to import then?

Well due to reason number 1. Sometimes in Kubernetes runs a command that creates something in AWS so I would like to run that module and then run on import on top of the stuff things that were created by that command but currently using the -all variant it doesn't support that.

[yorinasub17]

Sometimes in Kubernetes runs a command that creates something in AWS so I would like to run that module and then run on import on top of the stuff things that were created by that command but currently using the -all variant it doesn't support that.

Is there a reason data sources don't work for your use case?

If you are using Kubernetes to manage AWS resources (e.g Service with type LoadBalancer to create an ELB, or Ingress resource to create an ALB using the aws-alb-ingress-controller), you will most likely want to use data sources instead of importing them as resources, since terraform is not managing those resources. Data sources are designed to lookup existing resources not managed by terraform, which seems like it is the right use case here.

As far as message passing across modules go, you can leverage the terraform_remote_state data source to access outputs from another module. In a terragrunt folder structure, the state file is defined clearly via the file path, so it is relatively easy to pick out the right state file to pass through to terraform_remote_state.

The idea would be to have the kube command module create the resource using null_resource, have another command that waits for the resource to exist, and then another command that outputs the resource ID from k8s using an external data source.

You can then use terraform_remote_state in your dependent module get the resource ID output from the kube module and use that as an input to the corresponding resource's data source (e.g for load balancers, it would be aws_lb).

At least, this would be how I would handle it. It is a bit hard for me to say if you can adapt this pattern into your modules without seeing your code, but I hope this leaves some inspiration? Let me know if this doesn't work for you!

---

Dependency between modules (check issue) that chase me in my sleep. Terraform does not support it and I have a lot of modules that depends on each other....

Module dependencies are something we struggle with too, but there is a workaround that we have found works pretty well. See this comment on a related issue to the one you pointed at: hashicorp/terraform#1178 (comment)

The key idea is to use null_resource as a gate in your modules: you have a null_resource that renders all the elements in a given input list and have all the resources in the module depend on that null_resource. For the output side, you can do something similar and create a null_resource that is only created when all resources in the module are created, and output its ID. You can then feed that null_resource id to the next module.

[flmmartins]

If you are using Kubernetes to manage AWS resources (e.g Service with type LoadBalancer to create an ELB, or Ingress resource to create an ALB using the aws-alb-ingress-controller), you will most likely want to use data sources instead of importing them as resources, since terraform is not managing those resources. Data sources are designed to lookup existing resources not managed by terraform, which seems like it is the right use case here.

Yes, currently I'm fetching the asset with data but I would like to import it. Of course... I could use data and then output everything and then "simulate" a import. That I could do. I suppose that answers my original question. I just wished there was an easier way.

You can then use terraform_remote_state in your dependent module get the resource ID output from the kube module and use that as an input to the corresponding resource's data source (e.g for load balancers, it would be aws_lb).

At least, this would be how I would handle it. It is a bit hard for me to say if you can adapt this pattern into your modules without seeing your code, but I hope this leaves some inspiration? Let me know if this doesn't work for you!

Yes. I tried using import from state but if you are creating infra from scratch then terraform plan will output an error if the state still doesn't exist. Meaning I cannot even plan, let alone apply my infra from scratch. That's a problem so one more reason for me to like terragrunt dependency and all variants

Dependency between modules (check issue) that chase me in my sleep. Terraform does not support it and I have a lot of modules that depends on each other....

Module dependencies are something we struggle with too, but there is a workaround that we have found works pretty well. See this comment on a related issue to the one you pointed at: hashicorp/terraform#1178 (comment)

The key idea is to use null_resource as a gate in your modules: you have a null_resource that renders all the elements in a given input list and have all the resources in the module depend on that null_resource. For the output side, you can do something similar and create a null_resource that is only created when all resources in the module are created, and output its ID. You can then feed that null_resource id to the next module.

Hummm for me this looks quite ugly but guess I could try. Thanks for the tip!

Thanks for all the replies!
It's nice to have debates like this.. I'm quite new to Terraform myself and I'm the only devops where I work so this issues like this are helping me a lot! =)

[flmmartins]

In conclusion I think I will keep running -all variant... but slowly move to an opt-in non-all approach maybe relying in some shell. I prefer doing that then using the terraform hack for module depedency issue... I found this to be super ugly.

For the imports I'll continue using data and outputing instead of the import.

[flmmartins]

I closed this by mistake.. reopening...

[yorinasub17]

Meaning I cannot even plan, let alone apply my infra from scratch.

This is the annoying part about the xxx-all variants. It is hard to do a read only plan across the modules because terraform is not aware of the future state from up stream modules. We have discussed before about trying to make this better, but the nuances are complicated. See #262.

That said, apply-all should work because terragrunt will apply in the right order, so by the time it gets to the dependent module, it should already exist.

I just wished there was an easier way.

One thought. If you really want to implement the import feature, you can see if you can use a before hook to do the import. This might be tricky because you only want to run the import if it isn't already in state, but that might be the closest.

Hummm for me this looks quite ugly but guess I could try. Thanks for the tip!

I agree. I really hope they implement this in the core.

[yorinasub17]

Thanks for all the replies!
It's nice to have debates like this.. I'm quite new to Terraform myself and I'm the only devops where I work so this issues like this are helping me a lot! =)

Thanks for the kind words! Glad that you are finding it useful!

[ofbeaton]

For completeness, here is how I ended up solving it with hooks, in a operating system agnostic way (devs run terragrunt manually on windows, CICD runs terragrunt on linux) using a python script:

MIT licensed use this however you want.

in my env/bootstrap/terragrunt.hcl file

terraform {
  # terragrunt will bootstrap for us and create these resources
  # but we want to further modify them, so we need to import them
  after_hook "import_s3_state_bucket" {
    commands = ["init"]
    execute = ["py", "${get_parent_terragrunt_dir()}/../bin/import-resource-hook", "aws_s3_bucket.state_bucket", local.bucket_state_fullname]
  }

  after_hook "import_dynamodb_tf_lock_state" {
    commands = ["init"]
    execute = ["py", "${get_parent_terragrunt_dir()}/../bin/import-resource-hook", "aws_dynamodb_table.tf_lock_state", local.dynamodb_endpoint_fullname]
  }

  source = "${get_parent_terragrunt_dir()}/../infrastructure/bootstrap"
}

then in my bin/import-resource-hook file:

#!/usr/bin/env python3
import sys, argparse, subprocess

parser = argparse.ArgumentParser()
parser.add_argument("type")
parser.add_argument("name")
args = parser.parse_args()

tf_import = subprocess.run(["terraform", "import", args.type, args.name], text=True)
sys.exit(0)

Please note I am colocating my .tf files in infrastructure/ with my .hcl files in env/ within a single repo. If you're open sourcing or re-using your tf files across multiple projects, you'll want to put them in a separate repo per the terragrunt recommendation to do so. You can still use this solution just your paths will vary.

Since if the resource exists the import will fail, and the bad error code will make terragrunt bail out, I have to overwrite the bad error code with a exit 0. Note that python is only used to make a exit 0 os agnostic, if you are only on linux || true or somesuch will likely work just fine for you after the import command. On windows & exit 0 usually would work but I couldn't get this to work with terragrunt. Like related to the same reason we don't have access to commands like pwd, cd, etc so that syntax does not work. So I needed a wrapper script for windows either way. I chose python3 since I could eventually grow the wrapper script to perhaps only return 0 if the error output indicated the resource already existed, otherwise pass along the error. The import seems to fail cleanly if it already exists, so ends up being a NOOP.

As for why I wanted to do this? I have specific security requirements to implement on my state bucket, so while I found that it is fantastic that Terragrunt creates the state resources if they don't exist, and great that they allow me to specify tags, it is a slippery slope, I eventually want to manage the resource in terraform itself for full configurability. For example Terraform also recommends you turn on versioning, and many recommend you prevent_destroy on the state bucket. So this is best of both worlds and lets me do whatever I want after the "basic" bucket is created.

[yorinasub17]

Closing as original question has been answered thoroughly. The answers and examples here should be captured in a knowledge base though, so added knowledge-base tag to indicate this should be moved.