Support importing WDL unit tests from spec

.gitignore

@@ -18,4 +18,6 @@ tmp/
 _version_1.0.wdl
 _version_1.1.wdl
 _version_draft-2.wdl
-datasets/
+datasets/
+unit_tests/
+wdl-1.1-spec/

LICENSE

@@ -204,7 +204,7 @@
    limitations under the License.
 
 All files in this repository excluding files under the integration/gatk directory are under the Apache license.
-All files under the integration/gatk directory are under the BSD 3-Clause license from OpenWDL, copied below.
+All files under the integration/gatk directory and the setup_unit_tests.py script are under the BSD 3-Clause license from OpenWDL, copied below.
 
 BSD 3-Clause License
 

Makefile

@@ -14,6 +14,7 @@ build: cromwell womtool
 
 clean-build: clean
 	rm -rf build
+
 clean:
 	rm -f results*.json
 	rm -rf miniwdl-logs
@@ -27,6 +28,14 @@ clean:
 clean-csv:
 	rm csv_output_*
 
+clean-unit: clean-unit-setup
+	rm -rf wdl-1.1-spec
+	rm -rf unit_tests
+
+clean-unit-setup:
+	rm -rf /mnt/outputs
+	rm -rf /mnt/tmp
+
 cromwell:
 	mkdir -p build
 	if [ ! -f build/cromwell.jar ]; then \

README.md

@@ -20,6 +20,9 @@ Runners this has been evaluated against:
  - [miniwdl](https://github.com/chanzuckerberg/miniwdl)
  - [Cromwell](https://github.com/broadinstitute/cromwell)
 
+## WDL Specification Unit Tests
+For information about running specification tests from the WDL spec, calling a separate script is required. [See `README_UNIT.md`](README_UNIT.md)
+
 ## Note
 
 The tests in this directory are incomplete and are in the process of being added to.  If you'd like 

README_UNIT.md

@@ -0,0 +1,62 @@
+# Unit tests
+The WDL spec includes an extractable set of specification tests as per [their specification](https://github.com/openwdl/wdl-tests).
+This test suite allows for those tests to be extracted and tested against a variety of runners.
+
+## Notes
+- Specification tests are only available for [WDL versions 1.1.1 and above](https://github.com/openwdl/wdl-tests/blob/58ff36209586ed69c9a64d3e0b151a343f12a4eb/README.md?plain=1#L7).
+    > Starting with WDL 1.1.1, nearly all the examples in the WDL specification are also test cases that conform to the WDL markdown test specification.
+- If running locally, it is **not** recommended to run test IDs `hisat2` and `gatk_haplotype_caller` (test indices 66 and 67). These tests will take a lot of network bandwidth and require a large amount of resources.
+- The test ID `relative_and_absolute` (test index 52) needs to be run with root as it will read a root owned file
+- Cromwell is not supported as Cromwell cannot run WDL 1.1+
+- MiniWDL cannot run all tests, at least until [this issue is resolved](https://github.com/chanzuckerberg/miniwdl/issues/693)
+
+## Usage
+### Extracting Tests
+First, use the script `setup_unit_tests.py` to pull the WDL specification and to extract the tests.
+Calling the script with no arguments `python setup_unit_tests.py` is recommended, but passing in arguments will not drastically change the behavior of the script.
+```commandline
+usage: setup_unit_tests.py [options]
+
+Extracts the unit tests from the spec to be runnable by this test suite;test files will be outputted into the `unit_tests` directory and a new conformance file will be created.To run the
+unit tests, see [insert other script here]
+
+options:
+  -h, --help            show this help message and exit
+  --version {1.1,1.2}, -v {1.1,1.2}
+                        Version of the spec to grab unit tests out of. Only WDL 1.1 and above has support for built in unit tests.
+  --output-type {yaml,json}
+                        Specify conformance file output type.
+  --force-pull          Default behavior does not pull the spec repo if it was already pulled. Specify this argument to force a refresh.
+  --extra-metadata EXTRA_METADATA, -e EXTRA_METADATA
+                        Include extra metadata when extracting the unit tests into the config. Since we have our own method of testing file outputs, this is mostly used for file
+                        hashes/regexes.
+  --script SCRIPT       Bash script to run alongside. This is to set up the environment for the test suite, ex mount points and files tests may depend on.(Probably need to run with root).
+  --repo REPO           Repository to pull from.
+```
+
+By default, WDL version 1.1 is used, extra metadata from `unit_tests_metadata.yaml` is used, and `yaml` is the default output type.
+Other arguments by default are none.
+
+The script will pull the github repository for the specific version of WDL at `wdl-spec-[version]`. The unit tests and configuration file will be extracted into the `unit_tests/` folder. The configuration file will be at `unit_tests/test_config.yaml`, which is used for the `run_unit.py` script.
+
+Before running the test suite, the shell script `unit_tests_script.sh` may need to be executed. This shell script creates several directories that the specification tests will require to exist.
+To remove these directories and anything else that the script may create, call `make clean-unit-setup`.
+
+### Running Tests
+THen, use the script `run_unit.py` to actually run the tests according to the configuration file. If the default setup was performed,
+the script can be used as if using `run.py`.
+
+For example: `python run_unit.py --runner toil-wdl-runner --version 1.1 -n 105-109 --quiet --threads=4 --progress`
+
+There are extra arguments specific to `run_unit.py` that may be useful.
+```commandline
+  --config CONFIG, -c CONFIG
+                        Specify the path of the conformance config file.
+  --reset               Specify whether to run the setup script again.
+  --force-pull          Specify whether to use the cached SPEC or to force a pull.The setup script will be run as well.
+```
+The config argument by default is `unit_tests/test_config.yaml`, but can be overridden by passing in the path to another config file.
+Resetting the script will extract the tests from the specification file again with the default behavior, essentially calling `setup_unit_tests.py` with no arguments.
+Forcing a pull will pull the WDL repository, overwriting the existing WDL repository and specification file, before extracting the tests again. `--reset` must be included for `--force-pull` to work.
+
+To remove the WDL repository and the unit tests folder, call `make clean-unit`.

conformance.yaml

@@ -210,8 +210,8 @@
     dir: tests/basic_fail
     wdl: basic_fail.wdl
     json: basic_fail.json
-  outputs:
-    fail: {}
+  fail: true
+  outputs: {}
 - description: |
     Basic ceil() test
   tags: ["ceil", "standard_library"]
@@ -802,8 +802,8 @@
     dir: tests/range_as_input
     wdl: range_as_input.wdl
     json: range_invalid.json
-  outputs:
-    fail: {}
+  fail: true
+  outputs: {}
 - description: |
     Legacy test for range_as_input, range=0
   tags: ["range", "legacy"]
@@ -865,8 +865,8 @@
     dir: tests/length_as_input_with_map
     wdl: length_as_input_with_map.wdl
     json: length.json
-  outputs:
-    fail: {}
+  fail: true
+  outputs: {}
 - description: |
     Legacy test for length_as_input, fail
   tags: ["length", "legacy", "fail"]
@@ -876,8 +876,8 @@
     dir: tests/length_as_input
     wdl: length_as_input.wdl
     json: length_invalid.json
-  outputs:
-    fail: {}
+  fail: true
+  outputs: {}
 - description: |
     Legacy test for zip_as_input
   tags: ["zip", "legacy"]
@@ -1133,8 +1133,8 @@
     dir: tests/array_coerce
     wdl: array_coerce.wdl
     json: empty.json
-  outputs:
-    fail: {}
+  fail: true
+  outputs: {}
 - description: |
     Test ENCODE
   versions: ["draft-2", "1.0"]