Feature request: HCL file format support

[scizzorz]

The HashiCorp configuration language is pretty well-known, especially in projects that are using other supported HashiCorp products like Consul and Vault. I'm not an active Go author, but it appears that the linked repository is a ready-for-use, first-party HCL parser that could be leveraged to implement parsing and serializing (a la data.ToJSON etc).

I've been searching for a good templating tool to replace some weak in-house tooling and stumbled across gomplate that seems to meet all of my team's needs perfectly; however, a large amount of our project configuration is already written in HCL, so it would be spectacular to be able to drop in gomplate with a few minor changes to our template files.

Thanks!

[hairyhenderson]

Thanks for this suggestion @scizzorz!

From that repo:

This is major version 1 of HCL, which is now in maintenance mode only. We strongly recommend that new projects use HCL 2 instead.

Which version of HCL would you need to be supported? I'm not entirely certain from looking at that repo, but I think an HCL2 parser could support HCL1 as well... At worst, I could attempt to parse with HCL2 and then fall back to HCL1 if that fails.

I'm in favour of this feature, and I feel like it could be relatively simple.

A few issues to figure out, however:

• Which MIME Type to assign? There doesn't seem to be any prior art for something like application/hcl, but maybe that's the best choice...
• Which file extensions to auto-detect? .tf comes to mind as HCL is used for Terraform configs, but it's also used by Consul, Vault, Packer, and others where the file extension is usually .conf or similar. Would it make sense to detect .hcl as well? What other extensions would it make sense to detect?

[scizzorz]

Glad this seems reasonable!

I would follow their guidance and aim for HCL 2. As far as my needs go, all of my project configs are using modern versions of Terraform (0.12.x) which is HCL 2, so that also works out nicely :)

I have no clue what MIME type to assign. I hadn't even considered that - it might be helpful to ask a maintainer of the HCL repo? Or you could just elect to use application/hcl - I really have no clue!

I would certainly pick .tf and .hcl as those are pretty common, but there's some other oddities like .tfvars that also pop up. I'll try to dig around my projects tomorrow morning and see if there's any other common extensions we have laying around.

[hairyhenderson]

Ok, I've opened hashicorp/hcl#375 to start the discussion around the MIME type.

Good point about .tfvars - I'd forgotten about that one!

[hairyhenderson]

Oh also, it's a different issue, but worth linking the two: #774 was logged a while ago around supporting terraform output - that one's more around parsing output values from a Terraform state file, but it's at least tangentially-related.

[hairyhenderson]

@scizzorz FYI I've had some very useful feedback in hashicorp/hcl#375 - the upshot is that for HCL 2, it's not a simple matter of parsing a file like you would with JSON. Instead, there's a schema that needs to also be provided so that the parser can know how to map the various blocks.

I think it'd be fairly straightforward to support the Terraform 0.12 HCL schema, but it's somewhat more complex to support HCL in general.

Assuming that gomplate can support parsing your terraform configs, how do you expect to use them? Can you think of some examples? Are you thinking about only reading variable definitions, or also resource blocks and such?

As a side note, since Terraform does support JSON syntax, and I've used this in the past to provide a JSON file that can then be used by Terraform and as a gomplate datasource for separate purposes. It's not the cleanest, but it works in a pinch.

[scizzorz]

Oh that is quite interesting. We're currently using pyhcl in our homegrown template system, which I believe parses the HCL using the dynamic attributes model mentioned in the other thread, so I hadn't fully grasped that HCL 2 expected a schema as well. For my particular use case, that is completely sufficient. Because of Terraform's evaluation order, it's not able to perform variable substitutions in certain places where we would really like to have it perform substitutions (remote backend configs in particular), so if we use only native Terraform tooling we end up double specifying a lot of arguments, and even hard coding some of them into the code. This causes a lot of maintenance pain, so we developed a very minimal preprocessor to read a generic HCL file and apply it as arguments to a set of template files, producing ready-to-use Terraform code. That said, this is probably not the only way a gomplate user might want to read HCL files. I think users could get a lot of value out of also being able to provide a HCL2 schema when loading a file, so I think it would be great to explore those approaches as well. Perhaps if the data source received one argument it could be treated as the dynamic attributes model and if it receives two arguments it can interpret the second as a schema? I haven't dug too deeply into gomplate yet so I'm not sure if that's feasible. In any case, I really appreciate the effort you've already put into this! I'm really looking forward to being able to use gomplate in my system!

[hairyhenderson]

Hmm... If https://github.com/virtuald/pyhcl is the library you're referencing, it looks like it's HCL1-only, and has some changes to allow it to also parse Terraform 0.12 files.

Are you able to post some examples of your HCL files that you input into pyhcl, and how you reference them in your terraform configs? I'm trying to wrap my head around exactly what these inputs are expected to look like 🙂

[scizzorz]

Oh my - it does appear to be HCL1-only, doesn't it? Quite an interesting turn of events for me, to be honest!

I can't post real examples, but the gist is roughly a config like this:

web_app = {
  environment = "dev"
  component_a = {
    foo = "bar"
    boo = "far"
  }
  component_b = {
    flux = "qar"
    qux = "flar"
  }
  component_c = null
}

And then a template of some form:

{{if web_app}}
module "web_app_base" {
   source = "..."
   env = {{ web app.environment | data.ToHCL }}
}

{{if web_app.component_a}}
module "comp_a" {
  source = "..."
  base = module.web_app_base
  foo = {{ web_app.component_a.foo | data.ToHCL }}
  bar = {{ web_app.component_a.boo | data.ToHCL }}
}
{{end}}

{{if web_app.component_b}}
...
{{end}}

{{if web_app.component_c}}
...
{{end}}

{{end}}

As mentioned before, this is to work around a few of the limitations that Terraform has when processing its files - in particular, things like remote state can't be configured using the same variables that individual resources are configured with, and modules can't be conditionally disabled with the same count = 0 trick that other resources can be disabled with. By using the template preprocessor, I can produce "pure" Terraform that's already parameterized correctly and can selectively disable modules / components through the template's conditionals.

[hairyhenderson]

@scizzorz Ok, thanks for the examples!

It looks to me like your HCL file is parseable without a schema, purely with the dynamic attributes model. This is good, and so I think basic HCL2 support with just dynamic attributes support is the right first step.

I'm currently working on a few last issues before I release gomplate v3.7, but I think I can get to this for the release after that. Or if you're willing to work on a PR, that'd be great 😉

[hairyhenderson]

@scizzorz Just FYI, this may interest you - it looks like Terraform 0.13 will support count for modules: hashicorp/terraform#25016

[scizzorz]

Absolutely that does interest me! I'm still thinking I'll have to use gomplate as a preprocessor for some other logic going on, but that's a good step forwards. They must have just published that news because I remember looking at the 0.13 in-progress change notes and being disappointed that count wasn't supported for modules.

[github-actions]

This issue is stale because it has been open for 60 days with no activity. Remove stale label or comment or this will be automatically closed in a few days.

[hairyhenderson]

Is there still interest in HCL support in gomplate? Given it's been almost 3 full years of silence I'm assuming not...

[scizzorz]

Yes - I've since moved on with my career and suspect my previous team has either moved on or given up on this initiative! Thanks for double checking!

…

On Wed, Apr 19, 2023, 8:15 AM Dave Henderson ***@***.***> wrote: Is there still interest in HCL support in gomplate? Given it's been almost 3 full years of silence I'm assuming not... — Reply to this email directly, view it on GitHub <#839 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAT4UZTXMLX3RXJPHOR27YDXB7JNBANCNFSM4NAFSLKA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[hairyhenderson]

Thanks for following up @scizzorz! I'll close this now. If anyone feels strongly that HCL support is important for gomplate, please file a new issue! 🙂