Support 'outputs' in tests and steps

Allow users to define 'outputs' at the step and test levels, which is
a map (like 'env') that is interpolated last, after the 'run' command is
executed and checked by 'expect' conditions. Having 'outputs'
facilitates cleaner reuse of computed values and comparison of values
across containers without resorting to writing temp files and/or using
':host'.

'outputs' are always interpolated, even if the test/step has failed, to
allow users to inspect/react to issues (ex: using stderr) in subsequent
tests/steps. 'outputs' would only be skipped if the step 'if' is false.

Besides the addition of the new 'outputs' key, we allow adding 'id' to
steps and add two new contexts, 'tests' and 'steps', which represent the
previously completed tests and steps respectively.

The 'tests' context and the 'depends' key are analogous to 'needs' in
GitHub Actions; however, we currently do not check that a user
expression that references a test with 'tests' also includes that test
in their 'depends' graph. If a user references a non-existent test or
non-existent output, they get null (not an exception).

Due to our "in place" interpolation of tests and steps, we need to clear
uninterpolated 'outputs' in the case the test/step was skipped, so that
future tests/steps don't observe uninterpolated strings as 'outputs'.
Future refactoring of how tests/steps are interpolated will hopefully
make this unnecessary.

docs/reference/latest/expressions.md

@@ -79,6 +79,25 @@ The `step` context references the step itself and contains the following:
 | `step.stdout` | string | standard output of the command, only available after command is run |
 | `step.stderr` | string | standard error of the command, only available after command is run |
 
+### steps
+
+The `steps` context is a mapping of preceeding steps, indexed by `id`, in the
+current test. `steps` contains all the information from the `step` plus the
+following:
+
+| name | type | description |
+| ---- | ---- | ----------- |
+| `steps.<step-id>.outputs` | object | resolved `outputs` mappings |
+
+### tests
+
+The `tests` context is a mapping of preceeding tests, indexed by `id`, which
+includes the following information:
+
+| name | type | description |
+| ---- | ---- | ----------- |
+| `tests.<test-id>.outputs` | object | resolved `outputs` mappings |
+
 ## Functions and Methods
 
 ### Status

docs/reference/latest/input.md

@@ -22,9 +22,11 @@ Any type clarifications can be found in the glossary at the bottom.
 | `env` | map(str, istr) | set environment variables for all steps in the test, shadows suite `env` | `{}` |
 | `depends` | str or list(str) | test(s) by id that should be run before this test | `[]` |
 | `name` | istr | a human-readable test name | test id |
+| `outputs` | map(str, istr) | accessible to future tests via `tests` context; interpolated after steps | `{}` |
 | `steps` | list(step) | steps to be run | `[]` |
 
 > NOTE: `env` values are interpolated first and available to all other keys.
+> NOTE: `outputs` will always be interpolated, even if the test has failed.
 
 ## Step
 
@@ -33,9 +35,11 @@ Any type clarifications can be found in the glossary at the bottom.
 | `env` | map(str, istr) | set environment variables for command and expressions in the step, shadows suite and test `env` | `{}` |
 | `exec` | istr | location to execute `run` command, either a Docker Compose service name or `:host` | **required** |
 | `expect` | expr or list(expr) | additional success conditions, evaluated after `run` command, all of which must return a truthy value | `[]` |
+| `id` | str | identifier for the step, referenced in `steps` context | |
 | `if` | expr | execute the step, when result is truthy; otherwise, skip | `success()` |
 | `index` | int | references the index of the container to execute `run` commmand | `1` |
 | `name` | istr | a human-readable step name | step index |
+| `outputs` | map(str, istr) | accessible to future steps via `steps` context; interpolated after `run` | `{}` |
 | `repeat` | map(str, any) | presence indicates a step should be retried if the `run` or any `expect` condition fails | `null` |
 | `repeat.interval` | str | time to wait between retries in Docker Compose healthcheck format, ex: `1m20s` | `1s` |
 | `repeat.retries` | int | indicates number of retry attempts; retry indefinitely, if omitted | `null` |
@@ -44,6 +48,8 @@ Any type clarifications can be found in the glossary at the bottom.
 
 > NOTE: `if` is evaluated before `env`. When `if` is truthy, `env`
         values are interpolated and available to all other keys.
+> NOTE: Unless step is skipped entirely using `if`, `outputs` will
+        be always interpolated, even if the step has failed.
 
 ## Glossary
 

docs/reference/latest/results-file.md

@@ -41,6 +41,7 @@ Where steps contain the following keys:
 
 | name | type | description |
 | ---- | ---- | ----------- |
+| `id` | string | the step id |
 | `name` | string | the step name |
 | `outcome` | string | the overall outcome of the test, one of `passed`, `failed`, or `skipped` |
 | `error` | error | the primary/first error that occurred, if the step failed for any reason |

examples/00-intro.yaml

@@ -103,3 +103,38 @@ tests:
 
       - exec: node1
         run: rm -f repeat-test-file
+
+  outputs:
+    name: Outputs test
+    outputs:
+      TEST_OUTPUT: ${{ steps['step-1'].outputs.A_VALUE }}
+    steps:
+      - id: step-1
+        exec: node1
+        run: |
+          echo '{ "a": 13 }'
+        outputs:
+          A_VALUE: ${{ fromJSON(step.stdout).a }}
+
+      - exec: node1
+        run: echo -n '13 is the same as ${{ steps['step-1'].outputs.A_VALUE }}'
+        expect:
+          # Can use outputs directly in expressions or interpolate them
+          - steps['step-1'].outputs.A_VALUE == '13'
+          - step.stdout == '13 is the same as 13'
+          # Return null for non-existent steps and keys
+          - null == steps['step-1'].outputs.NONEXISTENT_VALUE
+          - null == steps['nonexistent-step-id']
+
+  check-previous-outputs:
+    # repeat is an arbitrary example of a succesful test without 'outputs' defined
+    depends: [ outputs, repeat ]
+    name: Check previous tests' outputs
+    steps:
+      - exec: node1
+        run: echo -n 'Earlier job output was ${{ tests['outputs'].outputs.TEST_OUTPUT }}'
+        expect:
+          - step.stdout == 'Earlier job output was 13'
+          - tests['outputs'].outputs.TEST_OUTPUT == '13'
+          # tests always return an outputs mapping, even if they do not define them
+          - tests['repeat'].outputs == {}

examples/01-fails.yaml

@@ -1,5 +1,9 @@
 name: Example Suite - Failures
 
+# This file demonstrates negative cases. Tests IDs prefixed 'fail-'
+# are expected to always fail. Test IDs prefixed 'maybe-' are expected
+# to pass (when using `--continue-on-error`).
+
 tests:
 
   fail-exit-code:
@@ -65,3 +69,44 @@ tests:
       - exec: node1
         run: /bin/true
         expect: throw("Intentional Failure")
+
+  fail-with-outputs:
+    name: Failing test that defines outputs
+    outputs:
+      TWO: 2
+      THREE: ${{ steps['failed-step'].outputs.THREE }}
+      ERROR: ${{ throw("Intentional Failure") }}
+    steps:
+      - id: failed-step
+        exec: node1
+        run: /bin/false
+        outputs:
+          THREE: ${{ 3 }}
+
+  maybe-check-empty-failure-outputs:
+    depends: fail-with-outputs
+    name: Assert failed tests still interpolate outputs
+    steps:
+      - exec: node1
+        run: /bin/true
+        expect:
+          # Successfully interpolated outputs are returned, even for failed tests
+          - tests['fail-with-outputs'].outputs.TWO == '2'
+          - tests['fail-with-outputs'].outputs.THREE == '3'
+          - tests['fail-with-outputs'].outputs.ERROR == null
+
+  fail-outputs-in-test-error:
+    name: Tests fail if test outputs error
+    outputs:
+      ERROR: ${{ throw("Intentional Failure") }}
+    steps:
+      - exec: node1
+        run: /bin/true
+
+  fail-outputs-in-step-error:
+    name: Tests fail if step outputs error
+    steps:
+      - exec: node1
+        run: /bin/true
+        outputs:
+          ERROR: ${{ throw("Intentional Failure") }}

schemas/input.yaml

@@ -4,6 +4,10 @@ $defs:
     type: object
     propertyNames: { type: string }
     additionalProperties: { type: string, expression: InterpolatedText }
+  outputs:
+    type: object
+    propertyNames: { type: string }
+    additionalProperties: { type: string, expression: InterpolatedText }
 
 # suite
 type: object
@@ -23,6 +27,7 @@ properties:
       properties:
         env:  { "$ref": "#/$defs/env" }
         name: { type: string, expression: InterpolatedText }
+        outputs: { "$ref": "#/$defs/outputs" }
         depends:
           oneOf:
             - type: string
@@ -45,8 +50,10 @@ properties:
                   - { type: string, expression: Expression }
                   - { type: array, items: { type: string, expression: Expression } }
               name: { type: string, expression: InterpolatedText }
+              id: { type: string }
               if: { type: string, expression: Expression, default: "success()" }
               index: { type: integer }
+              outputs: { "$ref": "#/$defs/outputs" }
               run:
                 oneOf:
                   - { type: string, expression: InterpolatedText }

schemas/results-file.yaml

@@ -47,6 +47,7 @@ properties:
             additionalProperties: false
             required: [ name, outcome, start, stop ]
             properties:
+              id: { type: string }
               name: { type: string }
               outcome: { "$ref": "#/$defs/outcome" }
               error: { "$ref": "#/$defs/error" }

src/dctest/core.cljs

@@ -7,7 +7,8 @@
             [clojure.string :as S]
             [clojure.walk :refer [postwalk]]
             [dctest.expressions :as expr]
-            [dctest.outcome :refer [failure? passed? pending? pending-> short-outcome
+            [dctest.outcome :refer [failure? passed? pending? skipped?
+                                    pending-> short-outcome
                                     fail! pass! skip!]]
             [dctest.util :as util :refer [obj->str js->map log indent indent-print-table-str]]
             [promesa.core :as P]
@@ -210,7 +211,7 @@ Options:
   (P/let [{:keys [interval retries]} (:repeat step)
           run-attempt (fn []
                         (P/let [step (run-exec context step)
-                                context (assoc context :step step)
+                                context (assoc context :step (select-keys step [:stdout :stderr]))
                                 step (pending-> step
                                                 (->> (run-expectations context)))]
                           step))]
@@ -219,6 +220,25 @@ Options:
                        {:delay-ms interval
                         :check-fn #(not (failure? %))})))
 
+(defn interpolate-outputs
+  "Interpolates test/step outputs using context. If any output errors,
+  mark as failed and record error(s)."
+  [context test-or-step]
+  (let [interp-val-or-err (update-vals (:outputs test-or-step)
+                                       (fn [o]
+                                         (try
+                                           {:value (expr/interpolate-text context o)}
+                                           (catch js/Error err
+                                             {:error {:message (str "Exception thrown: " (.-message err))}}))))
+        outputs (update-vals interp-val-or-err :value)
+        errors (vec (keep :error (vals interp-val-or-err)))
+
+        test-or-step (assoc test-or-step :outputs outputs)]
+
+    (if (not-empty errors)
+      (apply fail! test-or-step errors)
+      test-or-step)))
+
 (defn skip-if-necessary
   "Evaluates the step 'if' expression. Marks step as skipped, if 'if'
   evaluates to falsy; otherwise, returns step unmodified."
@@ -235,6 +255,7 @@ Options:
    - evaluating 'if'
    - interpolating keys ('env', 'name', ...)
    - executing 'run' and 'expect' conditions
+   - interpolating 'outputs'
 
   Short-circuits if skipped ('if' is falsy) or any aspect fails (including
   any errors during interpolation of keys). On failure, fail the step
@@ -252,11 +273,17 @@ Options:
                           (update :name #(interpolate-any context %))
                           (update :exec #(interpolate-any context %))
                           (update :run #(interpolate-any context %))
-                          (->> (execute-step-retries context))
-                          pass!)
+                          (->> (execute-step-retries context)))
+          context (assoc context :step (select-keys step [:stdout :stderr]))
+
+          step (if (skipped? step)
+                 (assoc step :outputs {})
+                 (interpolate-outputs context step))
+          step (pending-> step pass!)
+
           stop (js/Date.now)
           step (assoc step :start start :stop stop)]
-    (select-keys step [:outcome :name :start :stop :error :additionalErrors])))
+    (select-keys step [:outcome :id :name :start :stop :outputs :error :additionalErrors])))
 
 (defn execute-steps
   "Runs all steps for a test. Returns test with completed steps.
@@ -272,6 +299,9 @@ Options:
                          (assoc-in context [:state :failed] true)
                          context)
               step (execute-step context step)
+              context (if (:id step)
+                        (assoc-in context [:steps (:id step)] step)
+                        context)
               test (update test :steps conj step)
               test (if (failure? step)
                      (fail! test)
@@ -285,6 +315,7 @@ Options:
 
    - interpolating keys ('env', 'name', ...)
    - running all 'steps'
+   - interpolating 'outputs'
 
   Fail the test if any interpolated key throws an error or any step fails.
   If any key cannot be interpolated successfully, do not run steps, but do
@@ -299,16 +330,25 @@ Options:
 
           test (pending-> test
                           (update :name #(interpolate-any context %))
-                          (->> (execute-steps context))
-                          pass!)
+                          (->> (execute-steps context)))
+          steps-by-id (into {}
+                            (for [s (:steps test) :when (:id s)]
+                              [(:id s) s]))
+          context (assoc context :steps steps-by-id)
+
+          test (interpolate-outputs context test)
+          test (pending-> test pass!)
+
+          ;; Clean/remove to conform to results-file format
+          test (assoc test :steps (mapv #(dissoc % :outputs) (:steps test)))
 
           stop (js/Date.now)
           test (assoc test :start start :stop stop)
 
           duration-in-sec (js/Math.floor (/ (- stop start) 1000))
           _ (log opts "    " (short-outcome test) (:name test) (str "(" duration-in-sec "s)"))]
 
-    (select-keys test [:id :name :outcome :start :stop :steps :error :additionalErrors])))
+    (select-keys test [:id :name :outcome :start :stop :steps :outputs :error :additionalErrors])))
 
 (defn filter-tests [graph filter-str]
   (let [raw-list (if (= "*" filter-str)
@@ -350,6 +390,7 @@ Options:
       suite
       (P/let [[test & tests] tests
               test (run-test context suite test)
+              context (assoc-in context [:tests (:id test)] test)
               suite (update suite :tests conj test)
               suite (if (failure? test)
                       (fail! suite)
@@ -379,11 +420,16 @@ Options:
           suite (pending-> suite
                            (update :name #(interpolate-any context %))
                            (update :tests #(resolve-test-order % (:test-filter opts))))
+
           _ (log opts)
           _ (log opts "  " (:name suite))
           suite (pending-> suite
                            (->> (run-tests context))
-                           pass!)]
+                           pass!)
+
+          ;; Clean up for results-file format
+          suite (update suite :tests (fn [tests]
+                                      (mapv #(dissoc % :outputs) tests)))]
 
     (select-keys suite [:outcome :name :tests])))
 
@@ -502,12 +548,14 @@ Options:
                      (update :expect #(if (string? %) [%] %))
                      (update :index #(or % 1))
                      (update :name #(or % (str "steps[" index "]")))
+                     (update :outputs update-keys name)
                      (update-in [:repeat :interval] parse-interval)))
         ->test (fn [test id]
                  (-> (merge {:name id} test)
                      (assoc :outcome :pending)
                      (update :env update-keys name)
-                     (update :steps #(vec (map-indexed ->step %)))))
+                     (update :steps #(vec (map-indexed ->step %)))
+                     (update :outputs update-keys name)))
         ->suite (fn [suite path]
                   (-> (merge {:name path} suite)
                       (assoc :outcome :pending)

src/dctest/expressions.cljs

@@ -14,7 +14,7 @@
 (declare read-ast)
 
 (def supported-contexts
-  #{"env" "process" "step"})
+  #{"env" "process" "step" "steps" "tests"})
 
 (def stdlib
   ;; {name {:arity number :fn (fn [context & args] ..)}}

src/dctest/outcome.cljs

@@ -9,6 +9,9 @@
 (defn passed? [{:keys [outcome]}]
   (= :passed outcome))
 
+(defn skipped? [{:keys [outcome]}]
+  (= :skipped outcome))
+
 (defn pending? [{:keys [outcome]}]
   (= :pending outcome))
 

test/runexamples

@@ -66,11 +66,11 @@ eopt="--environ-file ${repo_root}/examples/03-env-file"
 
 up
 
-check 5  0 "${copt}        " 00-intro.yaml
-check 6  7 "${copt}        " 00-intro.yaml 01-fails.yaml
-check 5  1 "               " 00-intro.yaml 01-fails.yaml
-check 12 0 "${copt}        " 02-deps.yaml
-check 4  0 "${copt} ${eopt}" 03-env.yaml
-check 2  2 "${copt}        " 03-env.yaml
+check 7  0  "${copt}        " 00-intro.yaml
+check 9  10 "${copt}        " 00-intro.yaml 01-fails.yaml
+check 7  1  "               " 00-intro.yaml 01-fails.yaml
+check 12 0  "${copt}        " 02-deps.yaml
+check 4  0  "${copt} ${eopt}" 03-env.yaml
+check 2  2  "${copt}        " 03-env.yaml
 
 down