Support multiple isolated deployments per one TF root

[radeksimko]

There's currently assumption that each TF root/codebase will be deployed once.

In simple terms 1 codebase = 1 terraform.tfvars = 1 .terraform/ = 1 terraform.tfstate = 1 terraform remote config = 1 Atlas config.

I think that Terraform should support multiple isolated deployments of the same code. I can see use cases for isolated environments for testing purposes. Many people maintain separate/isolated infrastructure for testing things before going live.

It could work like RVM or Virtualenv:

user@local $ terraform env dev
user@local (dev) $ terraform [apply|destroy|plan] # all related to dev ENV

user@local $ terraform env live
user@local (live) $ terraform [apply|destroy|plan] # all related to live ENV

which is bringing another question - potential differences between these environments (i.e. deploying the same infrastructure in its less scalable & less fault-tolerant version for testing purposes).

I like that Terraform doesn't support conditionals as it's making it a simple & straightforward DSL exactly as it should be, but how to solve this problem using the DSL approach?

[radeksimko]

btw. I know I can always use options like -state, -var-file etc., but I more looking for more systematic approach, that may eventually leverage these existing features.

[radeksimko]

Just thinking that there could be a very simple, minimal and IMO nice solution... the remote config could just use the provided ENV name as a suffix in Atlas state name or elsewhere, but it really depends on what the common use-case for this is => do people have separate Atlas accounts for each environment like in AWS?

terraform apply -env=dev # or `TF_ENV=dev terraform apply`

connect to the right AWS account

provider "aws" {
  credentials_file_profile = "radeksimko-${terraform.env}"
}

use terraform.tfvars for common or default variables:

asg_size = 3
instance_size = t2.medium

and overwrite it in terraform.dev.tfvars:

asg_size = 1
instance_size = t2.small

or in terraform.live.tfvars:

asg_size = 9
instance_size = t2.large

so expect a pattern-based filenames like terraform.${ENV}.tfvars and automatically use that ENV name as a suffix for terraform.${ENV}.tfstate and give it a top priority when loading *.tfvars. Module data and similar stuff normally residing in .terraform would be kept in .terraform.${ENV}/.

Whoever doesn't want to use these environments will just keep using the same naming convention with no harm.

What do you think?

[ketzacoatl]

+1

[johnrengelman]

I've been thinking about this as well recently, but I was thinking I could make this work now by making my entire environment a module and then still having a separate tf file for each environment that simply pulls that module in.

[ketzacoatl]

In some cases, the environment is just an identifier that provides namespace in the real world, over the same formula. Right now I have a couple of projects with separate terraform paths for dev/production as separate deployments, and this is an anti-pattern (which is ok for now, because dev and production are significantly different). In all cases, I'm nearly losing my mind with the need to keep track of the TF state files. I realize this can be alleviated with consul and Atlas. Either way.. Ideally, multiple environments from the same TF root is just about helping the user manage vars and state

[JeanMertz]

Is there any update regarding managing different environments @mitchellh? We're also currently using different tfstate files, but also require different tfvar files.

Would be nice for a builtin clean solution to manage multiple environments with the same terraform configurations (but optionally different variables).

[nadnerb]

What are the current workarounds people are using for this? I feel like remote configuration is not currently feasible when deploying to multiple regions. We can store terraform state in github however this does not seem as ideal as S3 or Atlas.

[nadnerb]

It seems like a few workarounds have been documented above in #2824. I will check those out.

[nathanielks]

@radeksimko so, I hate being that guy who can't actually contribute code, BUT, here's an idea. When working with terraform remote config, would it be possible for the -state flag to specify a local file to read/write from? I attempted to use it that way and it continues to default to .terraform/terraform.tfstate.

[nathanielks]

@radeksimko I was able to work around that, so no worries there!

[pll]

+1!

I'm currently deploying the same code multiple times using different -var-file and -state files to keep management of different environments separate. This works fine UNTIL I want to share the states to other groups with S3.

[createdbysk]

+1

Same problem as mentioned by pII.

[2rs2ts]

+1 – especially with respect to remote state.

[danshao]

I'd like it so that I could reference other variables within variables.

For example, in my variables.tf file:

# choose (alpha|beta|prod)
variable "env_name" {
  default = "alpha"
}

variable "cidr_block" {
  alpha = {
    main = "10.115.0.0/16"
    public_1 = "10.115.10.0/24"
    public_2 = "10.115.11.0/24"
    web_1 = "10.115.20.0/24"
    web_2 = "10.115.21.0/24"
    rds_1 = "10.115.30.0/24"
    rds_2 = "10.115.31.0/24"
  }
  beta = {
    main = "10.114.0.0/16"
    public_1 = "10.114.10.0/24"
    public_2 = "10.114.11.0/24"
    web_1 = "10.114.20.0/24"
    web_2 = "10.114.21.0/24"
    rds_1 = "10.114.30.0/24"
    rds_2 = "10.114.31.0/24"   
  }
  prod = {
    main = "10.113.0.0/16"
    public_1 = "10.113.10.0/24"
    public_2 = "10.113.11.0/24"
    web_1 = "10.113.20.0/24"
    web_2 = "10.113.21.0/24"
    rds_1 = "10.113.30.0/24"
    rds_2 = "10.113.31.0/24"   
  }
}

.. I'd like to be able to switch environments by only changing var.env_name without touching my main terraform file. In my main terraform main.tf, I'd like to do this:

resource "aws_vpc" "main" {
  cidr_block = "${var.${var.env_name}.cidr_block.main}"
  enable_dns_hostnames = true # For private hosted zones
  tags {
    Name = "${var.proj_name}-${var.env_name}"
  }
  lifecycle {
    create_before_destroy = true
  }
}

[pll]

@2rs2ts & @danshao - I've worked around these flaws in terraform in a couple of different ways. My basic approach can be seen here:

https://github.com/pll/terraform_infrastructure

I'm frequently on #terraform-tool on freenode if you have questions.

[jonapich]

I am also trying to get my head around this problem. We have multiple AWS accounts to manage, and each account might include one or several environments. We want to be able to update each environment individually, and I'd even say, parts of an environment.

I must maximize flexibility and reduce repetitions, so I have created configurable modules that are shared between my environment definitions. My terraform folder looks a bit like this:

./terraform
./terraform/dev_account
./terraform/dev_account/dev.tf
./terraform/dev_account/qa.tf
./terraform/dev_account/networking.tf
./terraform/prod_account
./terraform/modules

This is how I can apply the dev and prod plans separately. Unfortunately, it doesn't allow me to apply the dev and qa plans separately. I thought about adding more folders, but then it makes it more difficult to interpolate to shared elements. It's not a solution at all, but it's the method that seems to work best. And i'm not even talking about that moment when a major change is initiated in dev, that we must apply, but that should not be applied in QA (then we must shelve our changes until dev is merged in QA, it's a pita).

I like the idea of naming files according to environments, and it's pretty much what we've been doing in our non-terraform deployments so far. We have a default infrastructure definition that contains the whole thing, which is then overriden by environment-specific configurations. I know it's technically possible to do this in terraform, for instance by using terraform init to isolate into a separate directory, but it's kinda clunky and error-prone.

Ideally, I'd like my hierarchy to look a bit like this:

./terraform
./terraform/modules
./terraform/default.tf
./terraform/default.dev.tf
./terraform/default.qa.tf
./terraform/default.prod.tf
./terraform/networking.tf
./terraform/networking.dev.tf
./terraform/networking.prod.tf

And ideally, I see multiple tfstate files, or the ability to separate my definition in logical entities (a bit like modules) and be able to run just the entities i'm interested in updating.

This is probably more complicated than it has to be, and sorry for the long post, I guess I had to vent a bit ;) congrats if you made it through!

[nathanielks]

@muikrad if you're interested, my script should solve that quite nicely: https://gist.github.com/nathanielks/5bd4de708e831bbc170f. As long as you use S3 to back your remote state, it'll work great 👍. Been using it in production the last few months.

[jonapich]

Thanks @nathanielks - you're right, this should do the trick! I wanted to avoid moving files around, but this is a clean way to put it.

[nathanielks]

@muikrad I've symlink'd the script to tfenv to make it a little simpler and more readable 😄

[tobiasmcnulty]

I just got bitten by this as well. It's a huge pain because you don't really see that it's happening until TerraForm's created some convoluted mess of resources on the account you just switched to.

All that's really missing is a way to specify the path to the local cached copy of the tfstate file. If that were possible...the rest would be easy!

[bkc1]

Here's my attempt at isolated multi-env,multi-region terraform deployments using env/region-specific remote state files: https://github.com/bkc1/terraform-multi-env

[aholbreich]

Is that something that was solved by version 0.9 State Environments (https://www.terraform.io/docs/state/environments.html)? Should this maybe be closed now?

[mitchellh]

Yep! Thanks for reminding us. :)

[gonzaloserrano]

I think https://www.terraform.io/docs/state/environments.html is not linked in the State section in the website.

[gonzaloserrano]

Sorry for the noise, since it's been deprecated in favor of workspaces makes sense.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.