Remote config should be stored separately from terraform state

[daveadams]

I've been experimenting with using Consul to store and share Terraform state, but I'm a bit flummoxed by the fact that the local user's remote configuration gets stored in the Terraform state JSON and synced up to Consul.

For example, if I run these commands:

$ terraform remote config \
    -backend=consul \
    -backend-config=address=consul.endpoint:443 \
    -backend-config=path=tf/test/test1 \
    -backend-config=access_token=<secret-value> \
    -backend-config=scheme=https
$ terraform remote push

Then I get this file in .terraform/terraform.tfstate and in Consul at kv/tf/test/test1:

{
    "version": 1,
    "serial": 0,
    "remote": {
        "type": "consul",
        "config": {
            "access_token": "<secret-value>",
            "address": "consul.endpoint:443",
            "path": "tf/test/test1",
            "scheme": "https"
        }
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {}
        }
    ]
}

I can't think of any reason to store the remote storage credentials in the remote storage itself. In fact there are a number of problems with this setup, including:

• Revealing secret credentials to other tokens with read access to the TF state path is a big security risk.
• If multiple users are collaborating on shared state, a terraform remote pull overwrites the remote config of the user with whatever the last-uploaded state was, which may well contain another person's Consul access token, as well as a potentially incorrect address, path, and scheme setting, since those may differ between users as well.

In summary, it would be best to keep remote configuration in a different file from the tfstate cache. Perhaps follow the pattern of git and allow the user to specify multiple named remotes in global and project-local config files. Perhaps a .terraform/config HCL file that's both human-editable and tool-manageable.

I'm happy to help work on some such solution if it seems reasonable, though I'm a relative newbie to Go.

[daveadams]

I see in the remote config docs that the S3 backend also has this issue if you specify the access_key and secret_key values with terraform remote config, and the documentation specifically recommends against doing so, in favor of using environment variables.

That's a good recommendation and practice. Could the Consul backend also support some environment variables, say, CONSUL_TOKEN etc, to avoid specifying secrets in the remote config?

[apparentlymart]

Agreed that having an environment variable for this would be a good idea. It'd be nice to support the same enviroment variable on the `consul_keys resource so users can just set it once in the environment and then forget about it.

It's interesting that the access_token argument is on the resource rather than the provider. I wonder if this was done under the assumption that different resources would use different tokens? It's strange to me that the Consul provider has this flexibility but we don't offer anything like this for the other providers.

The Consul cluster I've been working with doesn't actually have ACLs at all yet since we're relying on network security while we bootstrap, but I'd expect that when we do eventually enable ACLs we'd just have a single, somewhat-privileged credential that is always used by our deployment tools, just like we do for AWS, Rundeck, etc. The idea that we might use multiple different credentials within a single Terraform apply seems weird to me, but maybe other people have different workflows where this makes sense? Just wondering, since of course a single global environment variable is not compatible with using multiple different credentials for different resources...

(We write our configs with no credentials at all and assume the credentials will always come from the environment; this allows us to apply the same configuration multiple times in different environments, where different credentials are needed, and the orchestration tools just set up the environment appropriately depending on what target environment is selected.)

[daveadams]

Great thoughts, @apparentlymart. We're still experimenting with what level of ACL control we're going to have, but I'm mostly just concerned about possibilities for confusion, overwriting, and revealing a secret even if we decide to have only a set of RO and RW credentials.

However, I have some great news. I was digging around in the source code and found that the Consul API library (which Terraform uses to make the API calls) does allow for environment variable control. See the API config constructor in the Consul source. Specifically:

• CONSUL_HTTP_TOKEN
• CONSUL_HTTP_ADDR
• CONSUL_HTTP_AUTH
• CONSUL_HTTP_SSL
• CONSUL_HTTP_SSL_VERIFY

So I think this solves the immediate problem. I still would like the option to have multiple remote configs available globally and per-project. But I think most of us are writing wrapper scripts to handle the environment in any case, so this will go a long way towards making things more flexible and secure.

I was already working on a PR to update some documentation that's wrong/outdated, so I'll add this to the remote config page before I submit it.

[daveadams]

Related documentation update PR submitted and merged.

[vancluever]

I'm going to go one step further - I'd love to see Terraform keep remote config info (minus credentials) in the terraform file, or something similar, so that remote state can be calculated off of an parameter like so:

remote "consul" {
  address = "demo.consul.io:80",
  path = "key/path/tf/${var.environment}"
}

This is how I was hoping it was set up :\

[apparentlymart]

@vancluever there is actually already a separate issue #1964 for that sort of thing, BTW.

[vancluever]

Thanks @apparentlymart! I'll check it out.

[phinze]

Hey folks! Can this be closed as a duplicate of #1964 or is there still something separate to track in this thread?

[vancluever]

Hey @phinze, yeah I'm good, I'd imagine that any further discussions of this kind are kind of moot until TF state config is reworked somehow.

[phinze]

Sounds good, @vancluever - I'm hoping it's something we can address soon!

@daveadams as OP I'll wait until you chime in to close this. 👍

[daveadams]

I think this issue is OK to close. Thanks @phinze!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.