Terraform fails to recognize JSON generated from HCL

[renier]

I don't get an error from plan when loading a JSON .tf file, but it doesn't find anything to do vs. when loading the corresponding HCL .tf version. The JSON was generated from HCL using the hcl library in this way:

renier@taurus test $ cat test.hcl
provider "softlayer" {}

resource "softlayer_ssh_key" "my_key" {
  label = "Renier Key Add Test"
  public_key = "${file("~/.ssh/id2_rsa.pub")}"
}

renier@taurus test $ hcl2json test.hcl test.tf

renier@taurus test $ cat test.tf
{
    "provider": [
        {
            "softlayer": [
                {}
            ]
        }
    ],
    "resource": [
        {
            "softlayer_ssh_key": [
                {
                    "my_key": [
                        {
                            "label": "Renier Key Add Test",
                            "public_key": "${file(\"~/.ssh/id2_rsa.pub\")}"
                        }
                    ]
                }
            ]
        }
    ]
}

renier@taurus test $ ls *.tf
-rw-r--r--  1 renier  staff   480B Oct 11 16:49 test.tf

renier@taurus test $ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.


No changes. Infrastructure is up-to-date. This means that Terraform
could not detect any differences between your configuration and
the real physical resources that exist. As a result, Terraform
doesn't need to do anything.

renier@taurus test $ cp test.hcl test.tf

renier@taurus test $ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.


The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

+ softlayer_ssh_key.my_key
    fingerprint: "<computed>"
    label:       "Renier Key Add Test"
    public_key:  "ssh-rsa ********\n"


Plan: 1 to add, 0 to change, 0 to destroy.

If I manually edit the json, to strip out the array brackets, it works. However, shouldn't terraform be able to understand the JSON with the array brackets?

[jbardin]

Hi @renier,

Thanks for the report. What version of Terraform did you try this with?

There are still some ambiguities in converting HCL->JSON->HCL which we haven't codified yet, but I think is a good test of what should be covered.

[renier]

Hi @jbardin I'm on commit f2c95cd of terraform.

[jbardin]

@renier, Thanks.

Yes, this is still an issue. Interestingly, this isn't one of the failure cases for HCL itself, as this simple structure will round-trip outside of Terraform.

[renier]

@jbardin was not able to parse that last bit "structure will round-trip outside of Terraform"

You may also be interested in this issue, hashicorp/hcl#141

[jbardin]

@renier, I meant that the example you provided will decode "generically", marshal to JSON, then decode again into a matching data structure within the hcl package. It's some Terraform specific behavior which is causing this particular mismatch.

Yes, hcl#141 is related. Because the structure of HCL doesn't necessarily map directly to the data structures used, the addition of tags can complicate the issue.

[teamterraform]

Hi @renier! Sorry for the long silence here. We're reviewing some older issues now that Terraform 0.12 is out, and found this one.

In general it's not possible to map between HCL native syntax and HCL JSON without application-specific knowledge, because HCL's decoder uses information in the application's schema in order to resolve ambiguities in the JSON representation. For example, a JSON object can represent either an HCL block or a map value, so the only way the decoder can know is to refer to the relevant schema to see what the application is expecting.

For that reason, there is no way for a tool like hcl2json to work robustly. For such a tool to exist, it would need to be implemented inside Terraform itself and incorporate Terraform-specific knowledge (including provider schemas) in order to produce an equivalent JSON result.

Although Terraform 0.12 has ensured that everything can be expressed both in HCL native and in JSON, it doesn't do that by translating one format to another but instead by having two different decoder implementations. There is currently no implementation of translation between the two syntaxes, and we don't have any short-term plans to create such a thing because Terraform itself can already accept both syntaxes, so it's better to just let Terraform directly parse what you've written rather than translating it as a preprocessing step.

There's detailed information about the JSON variant of the Terraform language (as of Terraform 0.12) in the documentation section JSON Configuration Syntax. It's written in a way that describes the JSON equivalent of each construct you might see in native syntax, and via that guide it should be possible to find the correct JSON representation of anything that can be represented in native syntax.

Since having Terraform parse JSON translated by a separate tool is not a use-case we intend to support, we're going to close this now. While it would be technically possible to have a native-to-JSON translation tool built in to Terraform itself, we won't be able to prioritize anything like that for the foreseeable future because we are occupied with other work.

Thanks for reporting this, and hopefully the above helps to understand why this didn't behave as you originally expected.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.