Develop a comprehensive suite test battery

[cylc]

I've already started this, since coming up with a way to verify full test runs recently, but an open issue will be useful in case anyone has ideas for further improvement. The basic idea is this: The "run" command has new --reference-log and --reference-test options, and the suite definition supports a new [cylc] [[reference test]] section. A reference log run generates a log file in the suite definition directory that contains extra triggering information, namely who satisfied who's prerequisites at run time. The user manually verifies the correctness of this reference run. Subsequent reference test runs use a suite shutdown event handler to compare the new triggering behaviour with that of the reference run, disregarding time ordering (this means the test does not depend on different cycle interleaving behaviour, for example, due to use of a different runahead limit or different loading on the external queueing environment) . Additionally, the test fails if any task fails that is not expected to fail, or if the suite times out (this catches any problem that prevents the suite from finishing normally and shutting itself down). Finally, a new "cylc admin test-battery" command runs, by default, any tests found under the tests/ directory of the cylc installation. It can be given an optional directory argument to target some other tree or sub-tree of test suites, or a single test suite. The full test battery is now run automatically by the new-release script, which aborts if any test fails.

[cylc]

In addition to the standard tests, which should eventually use almost every feature of cylc - thereby preventing accidental introduction of new bugs in the future, any real suite can be used as its own reference test for the ultimate test that a new cylc release will not break some critical feature. Reference tests can be run in live mode, or much more quickly in simulation or dummy mode. In dummy mode only task command scripting is dummied out, so a dummy mode reference test is very nearly a complete test of cylc for a given suite.

[hjoliver]

I'm closing this - the test battery is fully functional - and posting an ongoing issue to record things that the test battery does not test yet, but should: #249