add PoC cmd converting HCL to JSON

[josephholsten]

This adds an hcl2json command which reads HCL from STDIN and writes indented JSON to STDOUT.

At the moment it is not friendly at all, and probably is only useful for people who are testing HCL code changes. Before merging, it should be a friendly tool to allow people who ❤️ HCL to use it in environments where importing this package is unwieldy.

Some things it should probably do:

• hcl2json, hcl2json -h should print usage
• hcl2json foo.hcl should read from foo.hcl
• hcl2json - should read from STDIN
• refactor to extract a CLI struct for testing
• add tests for the above functionality
• build through gox?

Fixes #32

[josephholsten]

ok, this meets my minimum usage reqs, what does hashicorp think?

[sethvargo]

A test or two would certainly be nice - one for each of your checkboxes would be really great, just so we don't accidentally break the API in the future.

[josephholsten]

I was just thinking about that. Do you like consul_template/cli_test.go's pattern for this?

[sethvargo]

@josephholsten ish. I'd really like to see cucumber-style tests when i run this, this happens in Go, but that's not happening anytime soon 😄. I think something like (super psuedo code here):

func TestCLI_parsesHCLtoJSON(t *testing.T) {
  cli, err := &CLI{}
  if err != nil {
    t.Fatal(err)
  }

  tmpfile, err := ioutil.Writefile("...", "some HCL here", 0644)
  if err != nil {
    t.Fatal(err)
  }

  cli.Run("hcl2json", tmpfile.Name())
}

The hard part becomes - how do you get the output 😄. In CT, we actually create a CLI struct that has an outStream and errStream. In tests, we set that to a buffer that we can read from. You may need to take a similar approach here.

[eropple]

Hey--

So I tried to use hcl2json to reverse-engineer the Terraform HCL format in order to work on Terraframe. The sample on the file provisioner page looks like this when fed through hcl2json:

{
  "resource": [
    {
      "aws_instance": [
        {
          "web": [
            {
              "provisioner": [
                {
                  "file": [
                    {
                      "destination": "/etc/myapp.conf",
                      "source": "conf/myapp.conf"
                    }
                  ]
                },
                {
                  "file": [
                    {
                      "destination": "/etc",
                      "source": "conf/configs.d"
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

I have known-good TF input of similar structure that looks like this:

{
  "resource": {
    "aws_instance": {
      "web": {
        "tags": {
          "Name": "web"
        }
      }
    }
  }
}

I still haven't figured out how provisioner is supposed to look, but if you look at resource and aws_instance, you can see hcl2json appears to be adding a bunch of extra arrays to the tree where they're not supposed to be. I don't know if this is TF magic or a bug in hcl2json, but I was directed here by @mitchellh so I figured I'd make sure everything was cool. :)

[josephholsten]

This is TF magic. Try running JSON through hcl2json to see how HCL parses into data. That output is what I'd expect to see.

[eropple]

Gotcha, thanks.

[virtuald]

+1, add some tests. This is a useful tool.

[virtuald]

Interesting, it's worth noting that in at least one case, the output of hcl2json does not match the expected input from the fixtures. In particular, the terraform_heroku example does not match. This is probably similar to @eropple's concern.

As maintainer of pyhcl, I can attest that it has similar problems due to the ambiguities in hcl. It would be nice to get this clarified.

[virtuald]

hcl2json's output of terraform_heroku.hcl:

{
  "config_vars": [
    {
      "FOO": "bar"
    }
  ],
  "name": "terraform-test-app"
}

pyhcl's hcltool output (which is the same as fixtures/terraform_heroku.json):

{
    "config_vars": {
        "FOO": "bar"
    },
    "name": "terraform-test-app"
}

[josephholsten]

@virtuald I've just refactored this to enable tests. Now I get to knock the rust off my go testing.

[josephholsten]

all right, I've added some tests for the four use cases. This should be ready for code review and possibly merge.

[solarce]

Just wanted to add my two ¢ here that it would be great to see this merged and also have the ambiguity around handling of multiple keys with the same name cleared up, e.g. virtuald/pyhcl#1, as this is going to be the common case for any complex Terraform configuration files.

All it takes is two aws_security_group or aws_instance resource definitions in the same configuration file and you've got this "issue", as right now pyhcl's implementation will collapse them.

See https://gist.github.com/solarce/39d634fe5f6b3ae494c6 for a quick example

[virtuald]

Yes, having this merged would be good -- but I would feel much better about this if the tests did tests against all of the existing fixtures -- just like I do with pyhcl.

[virtuald]

Still waiting on something like this.

[josephholsten]

@sethvargo what's remaining for this? Anything I can do to help?

@virtuald would you be satisfied with tests against everything in test-fixtures/? I'm not entirely sure why that's necessary as part of this tool, would it make more sense to just do these tests in decoder_test.go?

[virtuald]

@josephholsten Well, I'm not a maintainer of this repo, so technically my opinion doesn't count...

As far as the fixtures go, it seems like it would make sense that for each of the fixtures, you compare the output of hcl2json to see if it actually matches the json file in the fixture directory as a sanity check.

[jefferai]

There's another example of this tool adding arrays where they shouldn't be in hashicorp/vault#582

[jbaxleyiii]

I would also love to see this documented somewhere. I'm looking to write a node parser for HCL but am unsure of the mappings right now. Currently there is a rough parser but it doesn't support nested objects. https://github.com/NewSpring/node-hcl

[kvz]

Same problem for me where infra.tf with

variable "TSD_AWS_ACCESS_KEY" {
  description = "TSD_AWS_ACCESS_KEY"
}

Results in infra.tf.json with

{
  "variable": [
    {
      "TSD_AWS_ACCESS_KEY": [
        {
          "description": "TSD_AWS_ACCESS_KEY"
        }
      ]
    }
  ]
}

That Terraform trips over with a root.Variable: not an object type for map (ValueTypeList)

Then I tried with @virtuald's hcltool which returned JSON that Terraform said was valid:

{
    "variable": {
        "TSD_AWS_ACCESS_KEY": {
            "description": "TSD_AWS_ACCESS_KEY"
        }
    }
}

On a side note, the redundant descriptions were added due to hcl2json tripping over this notation:

variable "TSD_AWS_ACCESS_KEY" {}

[virtuald]

It would be great to get this finished off.

[scalp42]

To be clear we can't convert the json created to hcl correct?

The pain point for us is the templating of *.tf files (meaning ERBs for example) and I'm not exactly sure how other folks are doing it (besides storing vars in Consul backend for example).

[josephholsten]

@scalp42 yes, at the moment this codebase is simply showing the internal data structures as JSON, not converting to the JSON that would generate those data structures.

[snakescott]

Any ETA on hcl2json? I'm trying to get started with terraform, and want to both (a) write idiomatic config in hcl (.tf, not .tfjson) and (b) programmatically validate my config with python. hcl2json seems like the missing piece to get that workflow running.

Thanks!

[kvz]

Hey there, you can use the HCL printer (and terraform fmt) to do this now. @Acconut and I took the liberty of wrapping a few of HashiCorp's libraries and release json2hcl (and hcl2json via -reverse) binaries for this. In the offchance this is still useful to somebody: https://github.com/kvz/json2hcl

[josephholsten]

@kvz woo! I'll give that a look soon. Closing in the assumption that we can move any issues over there.

[kevinburkenotion]

Sorry - what is the "HCL printer" ?

[Acconut]

HCL printer (see https://godoc.org/github.com/hashicorp/hcl/hcl/printer) is Go package which allows you to transform the in-memory representation of a HCL structure into the corresponding textual format. It's similar to Go's json.Marshal function but for HCL.