backend (s3/azure): Source config from environment variables

[omar]

Allows customizing the remote state targets through environment variables.
Solves #17288, #12067, #13022, #12270, and many other requests to interpolate backend configuration. See discussion that prompted this here.

Common usage pattern is to separate Terraform state into S3 bucket/Azure Storage Account for production while all lower environments (dev/QA) can reside in the same location. This gives developers the freedom and sandbox they require to build using Terraform and gives operations folks comfort in separating dev and prod.

Because S3 Buckets and Azure Storage Account must be unique across all AWS/Azure accounts, there will always be name conflicts. Therefore, a different name must be used for the different environments. It's much easier (and better supported) in CI/CD systems to inject the bucket/storage account name via environment variables than in command line switches.

More use cases outlined in the links above prompted me to also include the key as an environment variable. For example, the author of #12270 is using a variable in key to create a generic backend config for use by other applications that get stored in a centralized S3 bucket per region.

[omar]

Changed the wording of how this is sourced to match the other values in this document.

[LinguineCode]

Is it really this simple? I'm not even a Go developer, but I can see what's going on here plain as day.

This PR, assuming it works, solves one of the biggest problems I personally have with managing customer's infrastructure using Terraform. I know I'm not the only one, just look at all those linked threads. I can't believe this was all it took. I'm really looking forward to hearing opinions from a Hashicorp person and hoping it gets released fast

I've never built terraform from source before, but I may have to, just so I can try this

Thank you @omar

[omar]

Yeah, this wasn't too bad to implement given the Terraform code is well structured. Please do test it 😄!

One thing to note, the AWS provider will throw an error if you try to use it with Terraform v0.12 which is what the master branch is building. To test it with AWS provider, I had to apply the changes from the v0.11.9 tag.

I know that the Terraform team is gearing up to do a v0.12 release, but I'd be interested if it would be okay to get this in the next v0.11.* release as I'd like to be able to take advantage of this feature without having to switch the CLI to v0.12.

[omar]

Any update on this? Would like to know if this is something the team is willing to consider.

[apparentlymart]

Hi @omar! Sorry for the delay in looking at this. The team at HashiCorp is currently 100% focused on the remaining steps for the Terraform v0.12 release, so we've been unable to give as much attention as we'd like to other things.

With that in mind, I've not yet had a chance to look closely at this but I wanted to share some design context for the current implementation of backends that may be helpful:

Although it's not directly represented anywhere in the code, conceptually we think of two different categories of inputs to the backends, which answer different questions:

• Location information tells us about where the state will be stored and where we can find any other resources (like locks) needed for safe backend operation. These should always be set in the configuration because it's important that they be constant for all users running Terraform, and so terraform init can detect when they change and offer the option of a safe migration to the new location.
• Credentials tell us who will be making the requests to the backend. The primary way to set these is via environment variables or external credentials files because that is more about the context Terraform is running in than specifically where the data is stored. Having a new user apply a Terraform configuration is not grounds for migrating the data and doesn't require any special safety procedures, aside from ensuring the user has the necessary access.

(indeed, for some historical/pragmatic reasons the boundary here is blurry in the sense that some credentials can be provided in configuration, though doing so of course causes Terraform to think of them like location information and force a state migration when they change.)

We consider it very important that the location information can change only during terraform init. It might be set directly in the configuration, or it might be set using the -backend-config arguments, but the important thing is that once terraform init completes successfully the fully-assembled configuration is saved into .terraform/terraform.tfstate and that is the source of record for all other Terraform commands. This allows Terraform to reduce the risk of inadvertently forking state, applying without locks active, or various other potentially-dangerous situations.

Supporting environment variables for these location-related settings would be acceptable only if the environment variables are checked only during terraform init, and ignored during other operations. I have some concerns that users would find that confusing though: the credentials-related environment variables are separate for each command, so we'd need to find a way to be very clear about which variables are init-only and which ones are per-command.

We previously implemented the possibility of setting an environment variable called TF_CLI_ARGS_init which is interpreted as extra arguments to add to the command line of terraform init, which can then be used with the -backend-config argument to get a similar result:

export TF_CLI_ARGS_init='-backend-config="bucket=s3-bucket-name"'

The tradeoff there was that we hoped that the environment variable name makes it clear that this variable is only for the terraform init command, thus mitigating the potential confusion I described above. As far as I can tell, aside from some admittedly-more-awkward syntax there doesn't seem to be anything possible with these new environment variables that isn't already possible with TF_CLI_ARGS_init. If you are aware of some limitations I'm not thinking of, please let me know!

[omar]

@apparentlymart, I appreciate the thorough response. I had no idea about the TF_CLI_ARGS_init and you're right, we could use that instead of the environment variables.

Thanks. I'll close this out.

[LinguineCode]

I didn't know about TF_CLI_ARGS_init either. Gonna have some fun with that today. Thanks again @omar and Hashicorp team

[SantoshKumarArjunan]

Very thorough response. Really helped my day. thanks.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.