Add support for Terraform v1.6+

[wata727]

Introduction

Terraform v1.6 added the following features related to TFLint.

• Unknown value refinements
  • Refined Unknown Values, allowing some operations with unknown values to produce known results hashicorp/terraform#33234
  • This concept has already been introduced in HCL and cty and has also been applied to TFLint, but Terraform's built-in functions have been expanded on this concept in v1.6. If a plugin checks refinements, you may not be able to get full refinements without this change.
• Testing framework
  • [Testing Framework] Add test file HCL configuration and parser functionality hashicorp/terraform#33325
  • Introduce separate testing scope for reference validation hashicorp/terraform#33339
  • Introduce 'run' keyword for referencing outputs from earlier run blocks hashicorp/terraform#33683
  • These updates are required for linting .tftest.hcl and tftest.json files.

Proposal

Update the embedded Terraform fork to v1.6.

However, this change will be affected by the licensing change to BUSL. As a result, TFLint will be bound to MPL and BUSL. See #1826
Fortunately, this update does not immediately impact (refinement is an edge case) Terraform v1.6 users, so we can postpone this decision further. At this point, I'm on the side of accepting the license change. OpenToFu could be a solution, but maintaining full compatibility without any references to Terraform's implementation would be difficult. As long as TFLint exists as a "Terraform" linter, this option may not be practical.

References

• https://github.com/hashicorp/terraform/releases/tag/v1.6.0
• https://github.com/hashicorp/terraform/tree/v1.6.0/internal/addrs
• https://github.com/hashicorp/terraform/tree/v1.6.0/internal/configs
• https://github.com/hashicorp/terraform/tree/v1.6.0/internal/lang
• https://github.com/hashicorp/terraform/tree/v1.6.0/internal/terraform
• Terraform v1.5 #1778

[wyardley]

I'm not super ideologically motivated w/r/t license, but for 1.6, I think OpenTofu is claiming to retain compatibility, so maybe at least see if using the fork would work without much change / hassle?

[bendrucker]

CLI compatibility isn't the issue here. It's a question of the Go packages and their public (exported) API. Those are explicitly not part of the Terraform compatibility promise.

Legal speculation ahead. I am not a lawyer. This is not legal advice.

US courts have held that APIs are generally excluded from copyright protections. Only implementation (source code) is protected.

This is straightforward for closed-source products, e.g., storage offerings with an S3-compatible API. And decidedly not for "source available" licenses like BUSL. Directly copying Terraform's internals (code) for a competing product violates the license.

The only guaranteed protection would come from being fully blinded to the Terraform source code and relying entirely on reverse engineering. We have no way of knowing the threshold of legality except to say that it is somewhere on the spectrum from wrote source copying to clean room reverse engineering.

End legal speculation.

It seems likely that OpenTofu's Go package API will diverge from Terraform. It will have to solve the same problems and so it's likely TFLint could port to those packages/interfaces. But these packages are inlined out of convenience not absolute necessity. If we wanted to spend time reverse engineering those internals instead of copying we could have done that. Porting to OpenTofu's internals is probably somewhat less work than implementing what we need from scratch, but it's still a) fairly expensive upfront b) creating an ongoing dependency on OpenTofu.

[wata727]

Terraform v1.7 has been released.
https://github.com/hashicorp/terraform/releases/tag/v1.7.0

Only one change in the nonsensitive function is affected.
hashicorp/terraform#33856

[wata727]

Terraform v1.8 has been released.
https://github.com/hashicorp/terraform/releases/tag/v1.8.0
https://www.hashicorp.com/blog/terraform-1-8-improves-extensibility-with-provider-defined-functions

The relevant changes are:

• Provider-defined functions
  • enable use of provider functions in the Terraform language hashicorp/terraform#34394
  • Introduced namespaced functions like provider::aws::arn_parse. Also, existing functions can be called like core::*.
  • Since provider-defined functions cannot statically retrieve all function definitions, TFLint requires some tricks.　
    • Perhaps when evaluating an expression, if it contains a provider-defined function, we will need to dynamically generate a pseudo-function that returns cty.DynamicVal. Specifically, we will modify EvalContext so that it can receive a list of function calls.

    • tflint/terraform/lang/eval.go

      Line 80 in e449b61

      func (s *Scope) EvalContext(refs []*addrs.Reference) (*hcl.EvalContext, hcl.Diagnostics) {

    • Question: How to evaluate unknown functions without errors? hashicorp/hcl#642 will help you with the trick of extracting function calls from hcl.Expression.
• Terraform-specific encoding functions
  • provider/terraform: Terraform-specific encoding functions tfvarsencode, tfvarsdecode, and exprencode hashicorp/terraform#34718
  • TFLint should support provider::terraform::tfvarsencode, provider::terraform::tfvarsdecode, and provider::terraform::exprencode functions.
• New issensitive function
  • add issensitive function hashicorp/terraform#34619

There are some other interesting changes as well:

• hashicorp/terraform@545980d
• hashicorp/terraform@f9998ff

These seem like they're worth digging into, but I haven't seen enough of them yet. However, the above should be the only changes that have a direct impact.

The changes in v1.8 have a big impact on the syntax, so I plan to follow them in TFLint. Perhaps the next version will update the embedded fork to v1.8.

[wyardley]

Just to confirm, if the embedded fork is updated, will this project change its license as well?

[wata727]

Most of TFLint's source code will continue to be licensed under MPL 2.0, but release binaries will now be distributed under MPL 2.0 + BUSL.