test-example: behave differently based on test type

* success: builds and passes the tests.
* fail: builds, but fails the tests.
* error: doesn't build.

Closes #397

Author: petertseng

README.md

@@ -127,6 +127,14 @@ The first condition is always enforced and cannot be circumvented.
 ### Example solution
 The example solution could be inspiration for other language implementors. It doesn't need to be perfect or very elegant. But it should be efficient enough for the test suite to finish in only a few seconds.
 
+Examples are named `<type>-<name>`.
+There are three possible types of examples:
+
+* success: The example is expected to pass the tests.
+  There should be at least one of these per exercise.
+* fail: The example is expected to build, but fail the tests.
+* error: The example is expected to fail to build.
+
 ### Test suite
 The test suite should be derived from the respective `x-common/exercises/<exercise-name>/canonical-data.json` and comply to some formatting and coding standards (to get an idea you may look at some of the existing tests).
 

bin/test-example

@@ -35,9 +35,39 @@ examplecache="${cachedir}/.foldercache/${exercisename}/${examplename}/.stack-wor
 mkdir -p "$examplecache"
 ln -f -s "$examplecache"
 
-echo "testing ${exampledir}"
-# SET_RESOLVER passed by .travis.yml - sets --resolver if not current.
-stack test ${SET_RESOLVER} `# Select the correct resolver. `\
-           --install-ghc   `# Download GHC if not in cache.`\
-           --no-terminal   `# Terminal detection is broken.`\
-           --pedantic      `# Enable -Wall and -Werror.    `
+exampletype=$(echo "$examplename" | cut -d- -f1)
+
+runstack () {
+    # SET_RESOLVER passed by .travis.yml - sets --resolver if not current.
+    stack "$@" ${SET_RESOLVER} `# Select the correct resolver. `\
+               --install-ghc   `# Download GHC if not in cache.`\
+               --no-terminal   `# Terminal detection is broken.`\
+               --pedantic      `# Enable -Wall and -Werror.    `
+}
+
+if [ "$exampletype" = "success" ]; then
+    echo "testing ${exampledir} - should succeed"
+    # Implicit exit value: this is last thing in this path,
+    # so the script's exit value = stack's exit value.
+    # If statements are reordered, please preserve this property.
+    runstack "test"
+elif [ "$exampletype" = "fail" ]; then
+    echo "testing ${exampledir} - should build, but fail tests"
+    if ! runstack "build"; then
+        echo "${exampledir} build failed unexpectedly"
+        exit 1
+    fi
+    if runstack "test"; then
+        echo "${exampledir} test succeeded unexpectedly"
+        exit 1
+    fi
+elif [ "$exampletype" = "error" ]; then
+    echo "testing ${exampledir} - should fail to build"
+    if runstack "test" "--no-run-tests"; then
+        echo "${exampledir} build succeeded unexpectedly"
+        exit 1
+    fi
+else
+    echo "unknown example type: ${exampledir}"
+    exit 1
+fi