Support importing WDL unit tests from spec #40
Conversation
…consistent with the wdl-test spec
There was a problem hiding this comment.
This looks like it should mostly work, but there are some parts that seem easy to break and a lot of things I found confusing. If we're just going to sort of cheat on parts of the parsing we should note that with appropriate TODOs, or if we're doing something actually safe that looks like it won't generalize we should cite evidence somewhere that it won't need to.
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
…formance-tests into support-unit-tests
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
…formance-tests into support-unit-tests
There was a problem hiding this comment.
It looks like some things have been fixed, and I understand some of what is going on better, but it looks like some things I flagged in the last review haven't been dealt with yet. Also I think I now understand why you're trying to use absolute paths under /mnt and I think you shouldn't be doing it.
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
…formance-tests into support-unit-tests
Co-authored-by: Adam Novak <anovak@soe.ucsc.edu>
…ts into support-unit-tests
| if repeat is not None: | ||
| response["repeat"] = repeat | ||
| # Turn failing tests to warnings if they violate a test dependency | ||
| response.update(test_dependencies(dependencies=test.get("dependencies"), current_result=response)) |
There was a problem hiding this comment.
If we're passing response in, and making this function return a modification, and then applying the modification, I think it gets kind of confusing.
We could make the function take the old response and return a new modified response copy. Or we could make it modify the response in place. Either of those would be simpler, and maybe easier to explain in the docstring.
The WDL spec unit tests should mostly be runnable (some may still have typos as I'm unable to fully check them due to miniwdl being partially broken).
The yaml representation for the original conformance tests have also changed a bit; failures are represented a little differently.
The dependency solution may be a bit iffy though, and one of the tests relies on having root access.