meta: Product-agnostic HCL syntax highlighting

[radeksimko]

Problem Statement

Most editors, plugins, extensions (including this Terraform VS Code extension), or websites wanting displaying files with some flavour of HCL (such as Terraform language) need a way of highlighting these files.

The fundamental problem of highlighting involves answering two (related) questions:

• which files to highlight
  • e.g. highlight *.tf as Terraform configuration
• how to highlight the file that should be highlighted
  • e.g. sequence of bytes between 0 and 10 on line 3 is an attribute name

Often times implementations start by supporting a single flavour of HCL, such as Terraform and many end there, or just decide to treat different HCL flavours as equivalent, e.g. highlight Nomad job spec with the same set of rules as Terraform configuration.

This creates a few challenges as described below.

Differences between Terraform versions

Terraform language contains a number of components which change between versions. For example function names, or block types. Product-specific highlighting often hard-codes such components, which makes it version-specific.

For example alltrue or anytrue functions were added in Terraform 0.14.0, but are highlighted differently (in this VS Code extension) than other functions introduced earlier (such as lower or max):

Other HCL-based languages

Reusing the same implementation which is tailored for one product (Terraform) results in inaccurate and incomplete
highlighting.

While the underlying parser is the same and components available within HCL2 are the same, the highlighting differs in small subtle ways. This results in inconsistent and suboptimal experience across HashiCorp products.

For example Packer template highlighted with Terraform VS Code extension:

or Nomad job spec

User Impact

All users of VS Code Terraform extension would benefit from more accurate highlighting of Terraform configurations.

At the time of writing there's close to 18 000 public repositories on GitHub which are detected as majority HCL (with close to 1M files across them). All of these files in these repositories could be highlighted more accurately.

Visitors of all HashiCorp .io product sites contain snippets of HCL configurations which would benefit from more accurate highlighting.

Expected User Experience

• Users of the VS Code Terraform extension are provided with files highlighted consistently and correctly for any Terraform 0.12+ compatible configuration.
• Users displaying Terraform configuration on GitHub see it highlighted consistently and correctly for any Terraform 0.12+ compatible configuration
• Visitors of terraform.io and other HashiCorp .io websites see snippets of HCL2 code highlighted accurately

Proposal

• create new repository, e.g. hashicorp/hcl-grammar to host the HCL grammar
• implement a generator which generates TextMate JSON grammar (initially) for product-agnostic HCL
• commit generated grammar to the repository
• create tests to cover the generated grammar, e.g. via https://github.com/PanAeon/vscode-tmgrammar-test
• use this grammar in the Terraform extension to highlight *.tf files (while assuming that any richer highlighting experience can be provided via LS)
• switch GitHub's linguist to this grammar for *.tf, *.hcl and other known extensions

Blocking Questions (Optional)

• Is generating syntax grammars feasible? A prototype would likely need to be created to answer this question.

[simonkeng]

Hello, just curious about the status of this? Asking for a friend who's learning Packer and would like *.pkr.hcl and *.pkrvars.hcl syntax highlighting 😅 cheers!

[radeksimko]

@simonkeng It's not on our near-term roadmap currently, but I would recommend you to upvote this issue and subscribe to get notified when we start working on this.

[simonkeng]

Okay got it. Thanks for the info @radeksimko

[damien]

This has already been called out in other tickets, but I wanted to mention here: Having this would make using tools like Terragrunt much easier.

[jpogran]

I can't explain more at the moment, but this is something we're actively working on and will have more information in the coming weeks.

[damien]

@jpogran Very exciting! I understand if you can't speak much to this, but would this work be related to anything on existing product roadmap items you share either publicly or with Hashicorp clients?

[Shocktrooper]

I second the modification to allow something like terragrunt to be used effectively. In the most recent updates the language server is undoing a format I need to make the get_env function to work as a standalone call

[radeksimko]

In the most recent updates the language server is undoing a format I need to make the get_env function to work as a standalone call

Can you explain further on what do you mean by this? Generally the Terraform language server never supported Terragrunt. If it ever worked, it was likely by accident.

More details and context is yet to come, but you can try out https://marketplace.visualstudio.com/items?itemName=HashiCorp.HCL which should highlight Terragrunt config files without issues, since these files follow *.hcl naming convention.

As for IntelliSense, we do not intend to provide that as part of the Terraform extension nor Terraform LS, since Terragrunt has its own schema (own keywords in its config files). We'd be however happy for anyone to create a dedicated Terragrunt extension and/or language server, which claims terragrunt.hcl. That allows all extensions to co-exist and each to keep only the necessary scope. A Terragrunt extension can leverage the HCL extension for highlighting (i.e. let it be installed automatically with Terragrunt extension).

As for building out the language server, we maintain https://github.com/hashicorp/hcl-lang which has still experimental (unstable) API, but is designed to be used to build any HCL2-based language server, such as Terragrunt one.

[radeksimko]

We now maintain a dedicated repository for the HCL grammar which also includes plenty of tests.

That HCL grammar is in turn used by a recently updated HCL extension - so you can use it for highlighting any *.hcl files, such as Packer, Terragrunt or tflint config files.

Users displaying Terraform configuration on GitHub see it highlighted consistently and correctly for any Terraform 0.12+ compatible configuration

This is tracked under hashicorp/syntax#35

Visitors of terraform.io and other HashiCorp .io websites see snippets of HCL2 code highlighted accurately

This is tracked under hashicorp/syntax#39

---

I have explained our intentions with regards to Terragrunt in more detail under #239 (comment)

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.