Custom expression decoding in function calls and hcldec specs

ext/customdecode/README.md

@@ -0,0 +1,209 @@
+# HCL Custom Static Decoding Extension
+
+This HCL extension provides a mechanism for defining arguments in an HCL-based
+language whose values are derived using custom decoding rules against the
+HCL expression syntax, overriding the usual behavior of normal expression
+evaluation.
+
+"Arguments", for the purpose of this extension, currently includes the
+following two contexts:
+
+* For applications using `hcldec` for dynamic decoding, a `hcldec.AttrSpec`
+  or `hcldec.BlockAttrsSpec` can be given a special type constraint that
+  opts in to custom decoding behavior for the attribute(s) that are selected
+  by that specification.
+
+* When working with the HCL native expression syntax, a function given in
+  the `hcl.EvalContext` during evaluation can have parameters with special
+  type constraints that opt in to custom decoding behavior for the argument
+  expression associated with that parameter in any call.
+
+The above use-cases are rather abstract, so we'll consider a motivating
+real-world example: sometimes we (language designers) need to allow users
+to specify type constraints directly in the language itself, such as in
+[Terraform's Input Variables](https://www.terraform.io/docs/configuration/variables.html).
+Terraform's `variable` blocks include an argument called `type` which takes
+a type constraint given using HCL expression building-blocks as defined by
+[the HCL `typeexpr` extension](../typeexpr/README.md).
+
+A "type constraint expression" of that sort is not an expression intended to
+be evaluated in the usual way. Instead, the physical expression is
+deconstructed using [the static analysis operations](../../spec.md#static-analysis)
+to produce a `cty.Type` as the result, rather than a `cty.Value`.
+
+The purpose of this Custom Static Decoding Extension, then, is to provide a
+bridge to allow that sort of custom decoding to be used via mechanisms that
+normally deal in `cty.Value`, such as `hcldec` and native syntax function
+calls as listed above.
+
+(Note: [`gohcl`](https://pkg.go.dev/github.com/hashicorp/hcl/v2/gohcl) has
+its own mechanism to support this use case, exploiting the fact that it is
+working directly with "normal" Go types. Decoding into a struct field of
+type `hcl.Expression` obtains the expression directly without evaluating it
+first. The Custom Static Decoding Extension is not necessary for that `gohcl`
+technique. You can also implement custom decoding by working directly with
+the lowest-level HCL API, which separates extraction of and evaluation of
+expressions into two steps.)
+
+## Custom Decoding Types
+
+This extension relies on a convention implemented in terms of
+[_Capsule Types_ in the underlying `cty` type system](https://github.com/zclconf/go-cty/blob/master/docs/types.md#capsule-types). `cty` allows a capsule type to carry arbitrary
+extension metadata values as an aid to creating higher-level abstractions like
+this extension.
+
+A custom argument decoding mode, then, is implemented by creating a new `cty`
+capsule type that implements the `ExtensionData` custom operation to return
+a decoding function when requested. For example:
+
+```go
+var keywordType cty.Type
+keywordType = cty.CapsuleWithOps("keyword", reflect.TypeOf(""), &cty.CapsuleOps{
+    ExtensionData: func(key interface{}) interface{} {
+        switch key {
+        case customdecode.CustomExpressionDecoder:
+            return customdecode.CustomExpressionDecoderFunc(
+                func(expr hcl.Expression, ctx *hcl.EvalContext) (cty.Value, hcl.Diagnostics) {
+                    var diags hcl.Diagnostics
+                    kw := hcl.ExprAsKeyword(expr)
+                    if kw == "" {
+                        diags = append(diags, &hcl.Diagnostic{
+                            Severity: hcl.DiagError,
+                            Summary:  "Invalid keyword",
+                            Detail:   "A keyword is required",
+                            Subject:  expr.Range().Ptr(),
+                        })
+                        return cty.UnkownVal(keywordType), diags
+                    }
+                    return cty.CapsuleVal(keywordType, &kw)
+                },
+            )
+        default:
+            return nil
+        }
+    },
+})
+```
+
+The boilerplate here is a bit fussy, but the important part for our purposes
+is the `case customdecode.CustomExpressionDecoder:` clause, which uses
+a custom extension key type defined in this package to recognize when a
+component implementing this extension is checking to see if a target type
+has a custom decode implementation.
+
+In the above case we've defined a type that decodes expressions as static
+keywords, so a keyword like `foo` would decode as an encapsulated `"foo"`
+string, while any other sort of expression like `"baz"` or `1 + 1` would
+return an error.
+
+We could then use `keywordType` as a type constraint either for a function
+parameter or a `hcldec` attribute specification, which would require the
+argument for that function parameter or the expression for the matching
+attributes to be a static keyword, rather than an arbitrary expression.
+For example, in a `hcldec.AttrSpec`:
+
+```go
+keywordSpec := &hcldec.AttrSpec{
+    Name: "keyword",
+    Type: keywordType,
+}
+```
+
+The above would accept input like the following and would set its result to
+a `cty.Value` of `keywordType`, after decoding:
+
+```hcl
+keyword = foo
+```
+
+## The Expression and Expression Closure `cty` types
+
+Building on the above, this package also includes two capsule types that use
+the above mechanism to allow calling applications to capture expressions
+directly and thus defer analysis to a later step, after initial decoding.
+
+The `customdecode.ExpressionType` type encapsulates an `hcl.Expression` alone,
+for situations like our type constraint expression example above where it's
+the static structure of the expression we want to inspect, and thus any
+variables and functions defined in the evaluation context are irrelevant.
+
+The `customdecode.ExpressionClosureType` type encapsulates a
+`*customdecode.ExpressionClosure` value, which binds the given expression to
+the `hcl.EvalContext` it was asked to evaluate against and thus allows the
+receiver of that result to later perform normal evaluation of the expression
+with all the same variables and functions that would've been available to it
+naturally.
+
+Both of these types can be used as type constraints either for `hcldec`
+attribute specifications or for function arguments. Here's an example of
+`ExpressionClosureType` to implement a function that can evaluate
+an expression with some additional variables defined locally, which we'll
+call the `with(...)` function:
+
+```go
+var WithFunc = function.New(&function.Spec{
+    Params: []function.Parameter{
+        {
+            Name: "variables",
+            Type: cty.DynamicPseudoType,
+        },
+        {
+            Name: "expression",
+            Type: customdecode.ExpressionClosureType,
+        },
+    },
+    Type: func(args []cty.Value) (cty.Type, error) {
+        varsVal := args[0]
+        exprVal := args[1]
+        if !varsVal.Type().IsObjectType() {
+            return cty.NilVal, function.NewArgErrorf(0, "must be an object defining local variables")
+        }
+        if !varsVal.IsKnown() {
+            // We can't predict our result type until the variables object
+            // is known.
+            return cty.DynamicPseudoType, nil
+        }
+        vars := varsVal.AsValueMap()
+        closure := customdecode.ExpressionClosureFromVal(exprVal)
+        result, err := evalWithLocals(vars, closure)
+        if err != nil {
+            return cty.NilVal, err
+        }
+        return result.Type(), nil
+    },
+    Impl: func(args []cty.Value, retType cty.Type) (cty.Value, error) {
+        varsVal := args[0]
+        exprVal := args[1]
+        vars := varsVal.AsValueMap()
+        closure := customdecode.ExpressionClosureFromVal(exprVal)
+        return evalWithLocals(vars, closure)
+    },
+})
+
+func evalWithLocals(locals map[string]cty.Value, closure *customdecode.ExpressionClosure) (cty.Value, error) {
+    childCtx := closure.EvalContext.NewChild()
+    childCtx.Variables = locals
+    val, diags := closure.Expression.Value(childCtx)
+    if diags.HasErrors() {
+        return cty.NilVal, function.NewArgErrorf(1, "couldn't evaluate expression: %s", diags.Error())
+    }
+    return val, nil
+}
+```
+
+If the above function were placed into an `hcl.EvalContext` as `with`, it
+could be used in a native syntax call to that function as follows:
+
+```hcl
+  foo = with({name = "Cory"}, "${greeting}, ${name}!")
+```
+
+The above assumes a variable in the main context called `greeting`, to which
+the `with` function adds `name` before evaluating the expression given in
+its second argument. This makes that second argument context-sensitive -- it
+would behave differently if the user wrote the same thing somewhere else -- so
+this capability should be used with care to make sure it doesn't cause confusion
+for the end-users of your language.
+
+There are some other examples of this capability to evaluate expressions in
+unusual ways in the `tryfunc` directory that is a sibling of this one.

ext/customdecode/customdecode.go

@@ -0,0 +1,56 @@
+// Package customdecode contains a HCL extension that allows, in certain
+// contexts, expression evaluation to be overridden by custom static analysis.
+//
+// This mechanism is only supported in certain specific contexts where
+// expressions are decoded with a specific target type in mind. For more
+// information, see the documentation on CustomExpressionDecoder.
+package customdecode
+
+import (
+	"github.com/hashicorp/hcl/v2"
+	"github.com/zclconf/go-cty/cty"
+)
+
+type customDecoderImpl int
+
+// CustomExpressionDecoder is a value intended to be used as a cty capsule
+// type ExtensionData key for capsule types whose values are to be obtained
+// by static analysis of an expression rather than normal evaluation of that
+// expression.
+//
+// When a cooperating capsule type is asked for ExtensionData with this key,
+// it must return a non-nil CustomExpressionDecoderFunc value.
+//
+// This mechanism is not universally supported; instead, it's handled in a few
+// specific places where expressions are evaluated with the intent of producing
+// a cty.Value of a type given by the calling application.
+//
+// Specifically, this currently works for type constraints given in
+// hcldec.AttrSpec and hcldec.BlockAttrsSpec, and it works for arguments to
+// function calls in the HCL native syntax. HCL extensions implemented outside
+// of the main HCL module may also implement this; consult their own
+// documentation for details.
+const CustomExpressionDecoder = customDecoderImpl(1)
+
+// CustomExpressionDecoderFunc is the type of value that must be returned by
+// a capsule type handling the key CustomExpressionDecoder in its ExtensionData
+// implementation.
+//
+// If no error diagnostics are returned, the result value MUST be of the
+// capsule type that the decoder function was derived from. If the returned
+// error diagnostics prevent producing a value at all, return cty.NilVal.
+type CustomExpressionDecoderFunc func(expr hcl.Expression, ctx *hcl.EvalContext) (cty.Value, hcl.Diagnostics)
+
+// CustomExpressionDecoderForType takes any cty type and returns its
+// custom expression decoder implementation if it has one. If it is not a
+// capsule type or it does not implement a custom expression decoder, this
+// function returns nil.
+func CustomExpressionDecoderForType(ty cty.Type) CustomExpressionDecoderFunc {
+	if !ty.IsCapsuleType() {
+		return nil
+	}
+	if fn, ok := ty.CapsuleExtensionData(CustomExpressionDecoder).(CustomExpressionDecoderFunc); ok {
+		return fn
+	}
+	return nil
+}