internal/core/adt: implement new evaluator

This does not yet implement disjunctions.

Much of the specific code from the old evaluator is shared with
the new evaluator. It is mostly the top-level control that is
reimplemented.

The new evaluator is based on the new scheduler and the new
closedness implementation. We will describe the function of each
new file below, with more details in the file itself.

conjuncts.go:
   This file take adt.Expr and adt.Value and either stores
   the corresponding value, if present, in a Vertex directly
   or otherwise schedules a task in the scheduler for later
   evaluation.

tasks.go:
   This file implements the tasks corresponding to the various
   adt.Expr expressions needed to evaluate them. These are
   scheduled by the code in conjuncts.go

states.go:
   This file adds more of the low-level state logic around the
   various properties a CUE node can achieve. These state map
   to the low-level scheduler, which is CUE agnostic.

unify.go:
   This file contains the high-level evaluation code that kicks
   off the evaluation process, including the initialization of
   a scheduler for each node. The analogous file for the old
   evaluator is eval.go.

Tests:
Some of the test results are different between the old and new
evaluator. In this case, the txtar files will contain both
an out/evalalpha file for the new evaluator, the original
out/alpha file for the old evaluator, and a diff file for the
diffed results.

If the changes were an improvement, this is described in the
diff/explanation file. Otherwise, there is a diff/todo/pX, where
X is the priority of the fix, with an explanation of what still
needs to be done.

Issue #2884

Signed-off-by: Marcel van Lohuizen <mpvl@gmail.com>
Change-Id: Ia0c5ad71cfdc7777b29f0aafd768da199844b710

cue/testdata/basicrewrite/010_lists.txtar

@@ -84,6 +84,98 @@ Retain: 4
 Unifications: 27
 Conjuncts:    46
 Disjuncts:    26
+-- out/evalalpha --
+Errors:
+e: conflicting values 4 and [] (mismatched types int and list):
+    ./in.cue:4:8
+    ./in.cue:4:13
+e2: undefined field: d:
+    ./in.cue:5:12
+e3: invalid index -1 (index must be non-negative):
+    ./in.cue:6:8
+    ./in.cue:6:12
+e4.3: invalid value 8 (out of bound <=5):
+    ./in.cue:7:24
+    ./in.cue:7:41
+e5.3: invalid value 8 (out of bound <=5):
+    ./in.cue:8:39
+    ./in.cue:8:18
+
+Result:
+(_|_){
+  // [eval]
+  list: (#list){
+    0: (int){ 1 }
+    1: (int){ 2 }
+    2: (int){ 3 }
+  }
+  index: (int){ 2 }
+  unify: (#list){
+    0: (int){ 1 }
+    1: (int){ 2 }
+    2: (int){ 3 }
+  }
+  e: (_|_){
+    // [eval] e: conflicting values 4 and [] (mismatched types int and list):
+    //     ./in.cue:4:8
+    //     ./in.cue:4:13
+  }
+  e2: (_|_){
+    // [eval] e2: undefined field: d:
+    //     ./in.cue:5:12
+  }
+  e3: (_|_){
+    // [eval] e3: invalid index -1 (index must be non-negative):
+    //     ./in.cue:6:8
+    //     ./in.cue:6:12
+  }
+  e4: (_|_){
+    // [eval]
+    0: (int){ 1 }
+    1: (int){ 2 }
+    2: (int){ 4 }
+    3: (_|_){
+      // [eval] e4.3: invalid value 8 (out of bound <=5):
+      //     ./in.cue:7:24
+      //     ./in.cue:7:41
+    }
+  }
+  e5: (_|_){
+    // [eval]
+    0: (int){ 1 }
+    1: (int){ 2 }
+    2: (int){ 4 }
+    3: (_|_){
+      // [eval] e5.3: invalid value 8 (out of bound <=5):
+      //     ./in.cue:8:39
+      //     ./in.cue:8:18
+    }
+  }
+}
+-- diff/-out/evalalpha<==>+out/eval --
+diff old new
+--- old
++++ new
+@@ -2,7 +2,7 @@
+ e: conflicting values 4 and [] (mismatched types int and list):
+     ./in.cue:4:8
+     ./in.cue:4:13
+-e2: invalid list index d (type string):
++e2: undefined field: d:
+     ./in.cue:5:12
+ e3: invalid index -1 (index must be non-negative):
+     ./in.cue:6:8
+@@ -34,7 +34,7 @@
+     //     ./in.cue:4:13
+   }
+   e2: (_|_){
+-    // [eval] e2: invalid list index d (type string):
++    // [eval] e2: undefined field: d:
+     //     ./in.cue:5:12
+   }
+   e3: (_|_){
+-- diff/todo/p2 --
+error: more precise message
 -- out/eval --
 Errors:
 e: conflicting values 4 and [] (mismatched types int and list):

cue/testdata/basicrewrite/012_selecting.txtar

@@ -68,6 +68,74 @@ Retain: 17
 Unifications: 27
 Conjuncts:    35
 Disjuncts:    25
+-- out/evalalpha --
+Errors:
+e: index out of range [4] with length 0:
+    ./in.cue:4:16
+h: undefined field: b:
+    ./in.cue:7:13
+
+Result:
+(_|_){
+  // [eval]
+  obj: (struct){
+    a: (int){ 1 }
+    b: (int){ 2 }
+  }
+  index: (int){ 2 }
+  mulidx: (int){ 3 }
+  e: (_|_){
+    // [eval] e: index out of range [4] with length 0:
+    //     ./in.cue:4:16
+  }
+  f: (_|_){
+    // [incomplete] f: undefined field: b:
+    //     ./in.cue:5:16
+  }
+  g: (_|_){
+    // [incomplete] g: undefined field: b:
+    //     ./in.cue:6:16
+  }
+  h: (_|_){
+    // [eval] h: undefined field: b:
+    //     ./in.cue:7:13
+  }
+}
+-- diff/-out/evalalpha<==>+out/eval --
+diff old new
+--- old
++++ new
+@@ -1,7 +1,7 @@
+ Errors:
+-e: invalid struct selector 4 (type int):
++e: index out of range [4] with length 0:
+     ./in.cue:4:16
+-h: invalid list index b (type string):
++h: undefined field: b:
+     ./in.cue:7:13
+
+ Result:
+@@ -14,7 +14,7 @@
+   index: (int){ 2 }
+   mulidx: (int){ 3 }
+   e: (_|_){
+-    // [eval] e: invalid struct selector 4 (type int):
++    // [eval] e: index out of range [4] with length 0:
+     //     ./in.cue:4:16
+   }
+   f: (_|_){
+@@ -26,7 +26,7 @@
+     //     ./in.cue:6:16
+   }
+   h: (_|_){
+-    // [eval] h: invalid list index b (type string):
++    // [eval] h: undefined field: b:
+     //     ./in.cue:7:13
+   }
+ }
+-- diff/explanation --
+Better error message for undefined field in new.
+Other error got worse, but this is covered elsewhere.
 -- out/eval --
 Errors:
 e: invalid struct selector 4 (type int):

cue/testdata/basicrewrite/013_obj_unify.txtar

@@ -83,6 +83,70 @@ Retain: 0
 Unifications: 15
 Conjuncts:    32
 Disjuncts:    15
+-- out/evalalpha --
+Errors:
+e: conflicting values 1 and {a:3} (mismatched types int and struct):
+    ./in.cue:7:5
+    ./in.cue:8:4
+e: cannot combine regular field "a" with 1:
+    ./in.cue:8:8
+    ./in.cue:7:5
+
+Result:
+(_|_){
+  // [eval]
+  o1: (struct){
+    a: (int){ 1 }
+    b: (int){ 2 }
+  }
+  o2: (struct){
+    a: (int){ 1 }
+    b: (int){ 2 }
+  }
+  o3: (struct){
+    a: (int){ 1 }
+    b: (int){ 2 }
+  }
+  o4: (struct){
+    a: (int){ 1 }
+    b: (int){ 2 }
+  }
+  e: (_|_){
+    // [eval] e: conflicting values 1 and {a:3} (mismatched types int and struct):
+    //     ./in.cue:7:5
+    //     ./in.cue:8:4
+    // e: cannot combine regular field "a" with 1:
+    //     ./in.cue:8:8
+    //     ./in.cue:7:5
+    a: (int){ 3 }
+  }
+}
+-- diff/-out/evalalpha<==>+out/eval --
+diff old new
+--- old
++++ new
+@@ -2,6 +2,9 @@
+ e: conflicting values 1 and {a:3} (mismatched types int and struct):
+     ./in.cue:7:5
+     ./in.cue:8:4
++e: cannot combine regular field "a" with 1:
++    ./in.cue:8:8
++    ./in.cue:7:5
+
+ Result:
+ (_|_){
+@@ -26,6 +29,9 @@
+     // [eval] e: conflicting values 1 and {a:3} (mismatched types int and struct):
+     //     ./in.cue:7:5
+     //     ./in.cue:8:4
++    // e: cannot combine regular field "a" with 1:
++    //     ./in.cue:8:8
++    //     ./in.cue:7:5
+     a: (int){ 3 }
+   }
+ }
+-- diff/explanation --
+An extra error messages that is correct, but not required.
 -- out/eval --
 Errors:
 e: conflicting values 1 and {a:3} (mismatched types int and struct):

cue/testdata/basicrewrite/018_self-reference_cycles.txtar

@@ -33,6 +33,44 @@ Retain: 11
 Unifications: 6
 Conjuncts:    26
 Disjuncts:    7
+-- out/evalalpha --
+(struct){
+  a: (_|_){
+    // [incomplete] b: non-concrete value _ in operand to +:
+    //     ./in.cue:2:4
+    //     ./in.cue:1:4
+  }
+  b: (_|_){
+    // [incomplete] b: non-concrete value _ in operand to +:
+    //     ./in.cue:2:4
+    //     ./in.cue:1:4
+  }
+  c: (#list){
+    0: (_){ _ }
+    1: (_){ _ }
+  }
+}
+-- diff/-out/evalalpha<==>+out/eval --
+diff old new
+--- old
++++ new
+@@ -1,10 +1,12 @@
+ (struct){
+   a: (_|_){
+-    // [cycle] cycle error:
++    // [incomplete] b: non-concrete value _ in operand to +:
++    //     ./in.cue:2:4
+     //     ./in.cue:1:4
+   }
+   b: (_|_){
+-    // [cycle] cycle error:
++    // [incomplete] b: non-concrete value _ in operand to +:
++    //     ./in.cue:2:4
+     //     ./in.cue:1:4
+   }
+   c: (#list){
+-- diff/explanation --
+More useful error messages.
 -- out/eval --
 (struct){
   a: (_|_){