config: Refined Unknown Values

This introduces the "refinements" concept from upstream cty, which allows Terraform to track some additional information about unknown values that constrains their possible range even though we don't yet know the final value.

docs/plugin-protocol/object-wire-format.md

@@ -76,8 +76,7 @@ in the table below, regardless of type:
 * An unknown value (that is, a placeholder for a value that will be decided
   only during the apply operation) is represented as a
   [MessagePack extension](https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types)
-  value whose type identifier is zero and whose value is unspecified and
-  meaningless.
+  value, described in more detail below.
 
 | `type` Pattern | MessagePack Representation |
 |---|---|
@@ -91,6 +90,64 @@ in the table below, regardless of type:
 | `["tuple",TYPES]` | A MessagePack array with one element per element described by the `TYPES` array. The element values are constructed by applying these same mapping rules to the corresponding element of `TYPES`. |
 | `"dynamic"` | A MessagePack array with exactly two elements. The first element is a MessagePack binary value containing a JSON-serialized type constraint in the same format described in this table. The second element is the result of applying these same mapping rules to the value with the type given in the first element. This special type constraint represents values whose types will be decided only at runtime. |
 
+Unknown values have two possible representations, both using
+[MessagePack extension](https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types)
+values.
+
+The older encoding is for unrefined unknown values and uses an extension
+code of zero, with the extension value payload completely ignored.
+
+Newer Terraform versions can produce "refined" unknown values which carry some
+additional information that constrains the possible range of the final value/
+Refined unknown values have extension code 12 and then the extension object's
+payload is a MessagePack-encoded map using integer keys to represent different
+kinds of refinement:
+
+* `1` represents "nullness", and the value of that key will be a boolean
+  value that is true if the value is definitely null or false if it is
+  definitely not null. If this key isn't present at all then the value may or
+  may not be null. It's not actually useful to encode that an unknown value
+  is null; use a known null value instead in that case, because there is only
+  one null value of each type.
+* `2` represents string prefix, and the value is a string that the final
+  value is known to begin with. This is valid only for unknown values of string
+  type.
+* `3` and `4` represent the lower and upper bounds respectively of a number
+  value, and the value of both is a two-element msgpack array whose
+  first element is a valid encoding of a number (as in the table above)
+  and whose second element is a boolean value that is true for an inclusive
+  bound and false for an exclusive bound. This is valid only for unknown values
+  of number type.
+* `5` and `6` represent the lower and upper bounds respectively of the length
+  of a collection value. The value of both is an integer representing an
+  inclusive bound. This is valid only for unknown values of the three kinds of
+  collection types: list, set, and map.
+
+Unknown value refinements are an optional way to reduce the range of possible
+values for situations where that makes it possible to produce a known result
+for unknown inputs or where it allows detecting an error during the planning
+phase that would otherwise be detected only during the apply phase. It's always
+safe to ignore refinements and just treat an unknown value as wholly unknown,
+but considering refinements may allow a more precise answer. A provider that
+produces refined values in its planned new state (from `PlanResourceChange`)
+_must_ honor those refinements in the final state (from `ApplyResourceChange`).
+
+Unmarshalling code should ignore refinement map keys that they don't know about,
+because future versions of the protocol might define additional refinements.
+
+When encoding an unknown value without any refinements, always use the older
+format with extension code zero instead of using extension code 12 with an
+empty refinement map. Any refined unknown value _must_ have at least one
+refinement map entry. This rule ensures backward compatibility with older
+implementations that predate the value refinements concept.
+
+A server implementation of the protocol should treat _any_ MessagePack extension
+code as representing an unknown value, but should ignore the payload of that
+extension value entirely unless the extension code is 12 to indicate that
+the body represents refinements. Future versions of this protocol may define
+specific formats for other extension codes, but they will always represent
+unknown values.
+
 ### `Schema.NestedBlock` Mapping Rules for MessagePack
 
 The MessagePack serialization of a collection of blocks of a particular type

go.mod

@@ -76,7 +76,7 @@ require (
 	github.com/tombuildsstuff/giovanni v0.15.1
 	github.com/xanzy/ssh-agent v0.3.1
 	github.com/xlab/treeprint v0.0.0-20161029104018-1d6e34225557
-	github.com/zclconf/go-cty v1.12.2
+	github.com/zclconf/go-cty v1.13.2
 	github.com/zclconf/go-cty-debug v0.0.0-20191215020915-b22d67c1ba0b
 	github.com/zclconf/go-cty-yaml v1.0.3
 	golang.org/x/crypto v0.1.0
@@ -172,8 +172,8 @@ require (
 	github.com/spf13/pflag v1.0.5 // indirect
 	github.com/stretchr/objx v0.5.0 // indirect
 	github.com/ulikunitz/xz v0.5.10 // indirect
-	github.com/vmihailenco/msgpack/v4 v4.3.12 // indirect
-	github.com/vmihailenco/tagparser v0.1.1 // indirect
+	github.com/vmihailenco/msgpack/v5 v5.3.5 // indirect
+	github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
 	go.opencensus.io v0.23.0 // indirect
 	golang.org/x/exp/typeparams v0.0.0-20221208152030-732eee02a75a // indirect
 	golang.org/x/time v0.3.0 // indirect

go.sum

@@ -778,10 +778,10 @@ github.com/tombuildsstuff/giovanni v0.15.1/go.mod h1:0TZugJPEtqzPlMpuJHYfXY6Dq2u
 github.com/ulikunitz/xz v0.5.10 h1:t92gobL9l3HE202wg3rlk19F6X+JOxl9BBrCCMYEYd8=
 github.com/ulikunitz/xz v0.5.10/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
 github.com/vmihailenco/msgpack v3.3.3+incompatible/go.mod h1:fy3FlTQTDXWkZ7Bh6AcGMlsjHatGryHQYUTf1ShIgkk=
-github.com/vmihailenco/msgpack/v4 v4.3.12 h1:07s4sz9IReOgdikxLTKNbBdqDMLsjPKXwvCazn8G65U=
-github.com/vmihailenco/msgpack/v4 v4.3.12/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4=
-github.com/vmihailenco/tagparser v0.1.1 h1:quXMXlA39OCbd2wAdTsGDlK9RkOk6Wuw+x37wVyIuWY=
-github.com/vmihailenco/tagparser v0.1.1/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI=
+github.com/vmihailenco/msgpack/v5 v5.3.5 h1:5gO0H1iULLWGhs2H5tbAHIZTV8/cYafcFOr9znI5mJU=
+github.com/vmihailenco/msgpack/v5 v5.3.5/go.mod h1:7xyJ9e+0+9SaZT0Wt1RGleJXzli6Q/V5KbhBonMG9jc=
+github.com/vmihailenco/tagparser/v2 v2.0.0 h1:y09buUbR+b5aycVFQs/g70pqKVZNBmxwAhO7/IwNM9g=
+github.com/vmihailenco/tagparser/v2 v2.0.0/go.mod h1:Wri+At7QHww0WTrCBeu4J6bNtoV6mEfg5OIWRZA9qds=
 github.com/xanzy/ssh-agent v0.3.1 h1:AmzO1SSWxw73zxFZPRwaMN1MohDw8UyHnmuxyceTEGo=
 github.com/xanzy/ssh-agent v0.3.1/go.mod h1:QIE4lCeL7nkC25x+yA3LBIYfwCc1TFziCtG7cBAac6w=
 github.com/xlab/treeprint v0.0.0-20161029104018-1d6e34225557 h1:Jpn2j6wHkC9wJv5iMfJhKqrZJx3TahFx+7sbZ7zQdxs=
@@ -795,8 +795,8 @@ github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1
 github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
 github.com/zclconf/go-cty v1.1.0/go.mod h1:xnAOWiHeOqg2nWS62VtQ7pbOu17FtxJNW8RLEih+O3s=
 github.com/zclconf/go-cty v1.2.0/go.mod h1:hOPWgoHbaTUnI5k4D2ld+GRpFJSCe6bCM7m1q/N4PQ8=
-github.com/zclconf/go-cty v1.12.2 h1:h4VH6eKXHTw60DiEJEVjh6pqVPDcoe3DuAkH/Ejs+4g=
-github.com/zclconf/go-cty v1.12.2/go.mod h1:s9IfD1LK5ccNMSWCVFCE2rJfHiZgi7JijgeWIMfhLvA=
+github.com/zclconf/go-cty v1.13.2 h1:4GvrUxe/QUDYuJKAav4EYqdM47/kZa672LwmXFmEKT0=
+github.com/zclconf/go-cty v1.13.2/go.mod h1:YKQzy/7pZ7iq2jNFzy5go57xdxdWoLLpaEp4u238AE0=
 github.com/zclconf/go-cty-debug v0.0.0-20191215020915-b22d67c1ba0b h1:FosyBZYxY34Wul7O/MSKey3txpPYyCqVO5ZyceuQJEI=
 github.com/zclconf/go-cty-debug v0.0.0-20191215020915-b22d67c1ba0b/go.mod h1:ZRKQfBXbGkpdV6QMzT3rU1kSTAnfu1dO8dPKjYprgj8=
 github.com/zclconf/go-cty-yaml v1.0.3 h1:og/eOQ7lvA/WWhHGFETVWNduJM7Rjsv2RRpx1sdFMLc=

internal/builtin/providers/terraform/resource_data.go

@@ -75,7 +75,7 @@ func planDataStoreResourceChange(req providers.PlanResourceChangeRequest) (resp
 	case req.PriorState.IsNull():
 		// Create
 		// Set the id value to unknown.
-		planned["id"] = cty.UnknownVal(cty.String)
+		planned["id"] = cty.UnknownVal(cty.String).RefineNotNull()
 
 		// Output type must always match the input, even when it's null.
 		if input.IsNull() {
@@ -90,7 +90,7 @@ func planDataStoreResourceChange(req providers.PlanResourceChangeRequest) (resp
 	case !req.PriorState.GetAttr("triggers_replace").RawEquals(trigger):
 		// trigger changed, so we need to replace the entire instance
 		resp.RequiresReplace = append(resp.RequiresReplace, cty.GetAttrPath("triggers_replace"))
-		planned["id"] = cty.UnknownVal(cty.String)
+		planned["id"] = cty.UnknownVal(cty.String).RefineNotNull()
 
 		// We need to check the input for the replacement instance to compute a
 		// new output.

internal/builtin/providers/terraform/resource_data_test.go

@@ -124,7 +124,7 @@ func TestManagedDataPlan(t *testing.T) {
 				"input":            cty.NullVal(cty.DynamicPseudoType),
 				"output":           cty.NullVal(cty.DynamicPseudoType),
 				"triggers_replace": cty.NullVal(cty.DynamicPseudoType),
-				"id":               cty.UnknownVal(cty.String),
+				"id":               cty.UnknownVal(cty.String).RefineNotNull(),
 			}),
 		},
 
@@ -140,7 +140,7 @@ func TestManagedDataPlan(t *testing.T) {
 				"input":            cty.NullVal(cty.String),
 				"output":           cty.NullVal(cty.String),
 				"triggers_replace": cty.NullVal(cty.DynamicPseudoType),
-				"id":               cty.UnknownVal(cty.String),
+				"id":               cty.UnknownVal(cty.String).RefineNotNull(),
 			}),
 		},
 
@@ -156,7 +156,7 @@ func TestManagedDataPlan(t *testing.T) {
 				"input":            cty.StringVal("input"),
 				"output":           cty.UnknownVal(cty.String),
 				"triggers_replace": cty.NullVal(cty.DynamicPseudoType),
-				"id":               cty.UnknownVal(cty.String),
+				"id":               cty.UnknownVal(cty.String).RefineNotNull(),
 			}),
 		},
 
@@ -198,7 +198,7 @@ func TestManagedDataPlan(t *testing.T) {
 				"input":            cty.StringVal("input"),
 				"output":           cty.UnknownVal(cty.String),
 				"triggers_replace": cty.StringVal("new-value"),
-				"id":               cty.UnknownVal(cty.String),
+				"id":               cty.UnknownVal(cty.String).RefineNotNull(),
 			}),
 		},
 
@@ -225,7 +225,7 @@ func TestManagedDataPlan(t *testing.T) {
 				"triggers_replace": cty.MapVal(map[string]cty.Value{
 					"key": cty.StringVal("new value"),
 				}),
-				"id": cty.UnknownVal(cty.String),
+				"id": cty.UnknownVal(cty.String).RefineNotNull(),
 			}),
 		},
 	} {

internal/command/views/json/diagnostic.go

@@ -318,7 +318,27 @@ func NewDiagnostic(diag tfdiags.Diagnostic, sources map[string][]byte) *Diagnost
 							// unknown value even when it isn't.
 							if ty := val.Type(); ty != cty.DynamicPseudoType {
 								if includeUnknown {
-									value.Statement = fmt.Sprintf("is a %s, known only after apply", ty.FriendlyName())
+									switch {
+									case ty.IsCollectionType():
+										valRng := val.Range()
+										minLen := valRng.LengthLowerBound()
+										maxLen := valRng.LengthUpperBound()
+										const maxLimit = 1024 // (upper limit is just an arbitrary value to avoid showing distracting large numbers in the UI)
+										switch {
+										case minLen == maxLen:
+											value.Statement = fmt.Sprintf("is a %s of length %d, known only after apply", ty.FriendlyName(), minLen)
+										case minLen != 0 && maxLen <= maxLimit:
+											value.Statement = fmt.Sprintf("is a %s with between %d and %d elements, known only after apply", ty.FriendlyName(), minLen, maxLen)
+										case minLen != 0:
+											value.Statement = fmt.Sprintf("is a %s with at least %d elements, known only after apply", ty.FriendlyName(), minLen)
+										case maxLen <= maxLimit:
+											value.Statement = fmt.Sprintf("is a %s with up to %d elements, known only after apply", ty.FriendlyName(), maxLen)
+										default:
+											value.Statement = fmt.Sprintf("is a %s, known only after apply", ty.FriendlyName())
+										}
+									default:
+										value.Statement = fmt.Sprintf("is a %s, known only after apply", ty.FriendlyName())
+									}
 								} else {
 									value.Statement = fmt.Sprintf("is a %s", ty.FriendlyName())
 								}

internal/lang/funcs/cidr.go

@@ -27,7 +27,8 @@ var CidrHostFunc = function.New(&function.Spec{
 			Type: cty.Number,
 		},
 	},
-	Type: function.StaticReturnType(cty.String),
+	Type:         function.StaticReturnType(cty.String),
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		var hostNum *big.Int
 		if err := gocty.FromCtyValue(args[1], &hostNum); err != nil {
@@ -56,7 +57,8 @@ var CidrNetmaskFunc = function.New(&function.Spec{
 			Type: cty.String,
 		},
 	},
-	Type: function.StaticReturnType(cty.String),
+	Type:         function.StaticReturnType(cty.String),
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		_, network, err := ipaddr.ParseCIDR(args[0].AsString())
 		if err != nil {
@@ -88,7 +90,8 @@ var CidrSubnetFunc = function.New(&function.Spec{
 			Type: cty.Number,
 		},
 	},
-	Type: function.StaticReturnType(cty.String),
+	Type:         function.StaticReturnType(cty.String),
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		var newbits int
 		if err := gocty.FromCtyValue(args[1], &newbits); err != nil {
@@ -126,7 +129,8 @@ var CidrSubnetsFunc = function.New(&function.Spec{
 		Name: "newbits",
 		Type: cty.Number,
 	},
-	Type: function.StaticReturnType(cty.List(cty.String)),
+	Type:         function.StaticReturnType(cty.List(cty.String)),
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		_, network, err := ipaddr.ParseCIDR(args[0].AsString())
 		if err != nil {

internal/lang/funcs/collection.go

@@ -35,6 +35,7 @@ var LengthFunc = function.New(&function.Spec{
 			return cty.Number, errors.New("argument must be a string, a collection type, or a structural type")
 		}
 	},
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (cty.Value, error) {
 		coll := args[0]
 		collTy := args[0].Type()
@@ -71,7 +72,8 @@ var AllTrueFunc = function.New(&function.Spec{
 			Type: cty.List(cty.Bool),
 		},
 	},
-	Type: function.StaticReturnType(cty.Bool),
+	Type:         function.StaticReturnType(cty.Bool),
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		result := cty.True
 		for it := args[0].ElementIterator(); it.Next(); {
@@ -100,7 +102,8 @@ var AnyTrueFunc = function.New(&function.Spec{
 			Type: cty.List(cty.Bool),
 		},
 	},
-	Type: function.StaticReturnType(cty.Bool),
+	Type:         function.StaticReturnType(cty.Bool),
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		result := cty.False
 		var hasUnknown bool
@@ -149,6 +152,7 @@ var CoalesceFunc = function.New(&function.Spec{
 		}
 		return retType, nil
 	},
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		for _, argVal := range args {
 			// We already know this will succeed because of the checks in our Type func above
@@ -181,7 +185,8 @@ var IndexFunc = function.New(&function.Spec{
 			Type: cty.DynamicPseudoType,
 		},
 	},
-	Type: function.StaticReturnType(cty.Number),
+	Type:         function.StaticReturnType(cty.Number),
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		if !(args[0].Type().IsListType() || args[0].Type().IsTupleType()) {
 			return cty.NilVal, errors.New("argument must be a list or tuple")
@@ -346,6 +351,7 @@ var MatchkeysFunc = function.New(&function.Spec{
 		// the return type is based on args[0] (values)
 		return args[0].Type(), nil
 	},
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		if !args[0].IsKnown() {
 			return cty.UnknownVal(cty.List(retType.ElementType())), nil
@@ -489,7 +495,8 @@ var SumFunc = function.New(&function.Spec{
 			Type: cty.DynamicPseudoType,
 		},
 	},
-	Type: function.StaticReturnType(cty.Number),
+	Type:         function.StaticReturnType(cty.Number),
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 
 		if !args[0].CanIterateElements() {
@@ -558,7 +565,8 @@ var TransposeFunc = function.New(&function.Spec{
 			Type: cty.Map(cty.List(cty.String)),
 		},
 	},
-	Type: function.StaticReturnType(cty.Map(cty.List(cty.String))),
+	Type:         function.StaticReturnType(cty.Map(cty.List(cty.String))),
+	RefineResult: refineNotNull,
 	Impl: func(args []cty.Value, retType cty.Type) (ret cty.Value, err error) {
 		inputMap := args[0]
 		if !inputMap.IsWhollyKnown() {