cty Refinements: Unknown values with tighter range

[apparentlymart]

My upstream project cty, which HCL uses as its type system, has a new feature in its v1.13.0 release where it can often refine the range of an unknown result from an operation based either on innate characteristics of that operation (e.g. many operations never return a null value) or based on already-refined input values.

There's more information on how exactly refinements work, and the scope of the initial design, in the cty documentation on Refinements.

Along with updating to the new cty which includes that feature, and updating some of the existing tests to expect the newly-refined unknown results, this also makes HCL itself aware of refinements in some specific cases where my research indicates that it'll make it more likely for applications like Terraform to raise errors sooner (e.g. during validation or planning rather than during apply) and to more often produce known results in count and for_each arguments, which Terraform currently requires to be known during planning.

The specific cases I implemented here are:

• In an expression like "https://${hostname}", if hostname is unknown then we still know that the final string will be non-null and will start with https://, which in particular means that result != null and result != "" can both produce known true instead of an unknown bool result.
• In an expression like unknown ? a : b we can propagate some refinements that are shared by both a and b into the result even though we don't know which match arm to choose yet. This isn't super useful on its own but composes with other rules (both here in HCL and down in cty) to increase the likelihood of producing a wholly-known or partially-known result.
• When processing optional attributes in the typeexpr extension -- which Terraform uses for its input variable type constraint mechanism -- if either the given value or the default value are refined as non-null then we can be sure that the value of that attribute won't be null. Again, this is not super useful on its own but knowing that a value definitely isn't null gives more opportunities for replacing unknown values with wholly-known or partially-known results.
• When resolving the splat operator [*] we can propagate refinements about the length of the input into the output, as long as we already know the type of the input and so can be sure about whether we're working in the list mode or in the special "null or not" mode.

There's also one extra thing which does not have immediate benefit but may be useful downstream in Terraform:

• hcldec has a new "spec" that wraps another spec and applies refinements to its value. Terraform uses hcldec to interpret the content of resource blocks (and similar) in terms of the provider schema, and so if Terraform later allows providers to mention certain static refinements as part of the provider schema then Terraform could potentially check and apply those automatically while decoding the configuration.

  In particular, legacy SDK providers are guaranteed to never return a null value for id, and so a hypothetical change to refine that attribute as never being null would make it easier to use the id of a not-yet-created resource as part of a conditional or other similar expression downstream without getting an unknown result.

These changes are all consistent with cty's compatibility rule that Unknown Values can become "more known", which HCL effectively passes on to its own callers as a result of using cty as its type system. Therefore applications adopting a new version of HCL that includes these changes should expect to see more tightly-refined results in their tests involving unknown values, and so should expect to update those tests to match as part of upgrading HCL, as long as the new results are actually correctly-refined for the expected final result.

(If this change is accepted, I have a similar sort of change ready to submit to Terraform itself which would make it refine even more cases, again building on the foundations from both cty and HCL.)

[apparentlymart]

The following Terraform issues would either be resolved completely or partially improved by merging this and then adopting it in Terraform:

• In an unknown choice of 1 of 2 lists, if the lists are the same length, the list length should be known terraform#33237
• Support partial rendering of templatefile instead of marking the whole value "known after apply" terraform#32651
• Allow "count" for non-null check on resource attributes that are always present terraform#26755 (not completed by this change, but this change provides some building-blocks for it)

[apparentlymart]

I ended up submitting separately the Terraform-specific parts of adopting refined values: hashicorp/terraform#33234

...but once this PR is merged and released I'll make a new PR in Terraform to adopt the new version of HCL too.

[jbardin]

should we bump this to the latest patch?

[apparentlymart]

These changes I made only require 1.13.0, so I don't see a strong reason to force it since any calling application can select a newer version if it wants to. This is only here to represent that this new version of HCL literally won't work if you use v1.12 or earlier. 🤷‍♂️

[apparentlymart]

(We could still do it, but I'm inclined to make that a separate change since this particular change doesn't need it.)