3695.abort if any task fails

[wxtim]

CLOSES: #3695

Largely straightforward. There is a design decision to move the config item back to it's Cylc 7 location [cylc][abort if any task fails] and to allow the use of a custom script. As far as I know parsec doesn't have a mechanism for dealing with an item deprecated in favour of a command line option.

There are a number of tests which rely on this config item which may or may not be ok (the deprecation warning may break them). I am addressing these tests, but the logic of the PR is ready for review.

[kinow]

+1 on @oliver-sanders comments. No other comments after reading code & tests 👍

Tested with this workflow from the docs:

[scheduling]
    [[dependencies]]
        graph = "hello"
[runtime]
    [[hello]]
        script = """
sleep 10
if [[ $CYLC_TASK_TRY_NUMBER < 3 ]]; then
    echo "Hello ... aborting!"
    exit 1
else
    echo "Hello World!"
fi"""

Running with cylc run suite:

2020-08-25T15:16:34+12:00 INFO - [hello.1] status=submitted: (received)started at 2020-08-25T15:16:33+12:00  for job(01) flow(o)
2020-08-25T15:16:34+12:00 INFO - [hello.1] -health check settings: execution timeout=None
2020-08-25T15:16:43+12:00 INFO - [client-command] put_messages kinow@ranma:cylc-message
2020-08-25T15:16:44+12:00 CRITICAL - [hello.1] status=running: (received)failed/EXIT at 2020-08-25T15:16:43+12:00  for job(01) flow(o)
2020-08-25T15:16:44+12:00 CRITICAL - [hello.1] -job(01) failed
2020-08-25T15:16:45+12:00 WARNING - suite stalled

And running with cylc run --abort-if-any-task-fails suite:

2020-08-25T15:16:12+12:00 INFO - [hello.1] -health check settings: execution timeout=None
2020-08-25T15:16:21+12:00 INFO - [client-command] put_messages kinow@ranma:cylc-message
2020-08-25T15:16:22+12:00 CRITICAL - [hello.1] status=running: (received)failed/EXIT at 2020-08-25T15:16:21+12:00  for job(01) flow(F)
2020-08-25T15:16:22+12:00 CRITICAL - [hello.1] -job(01) failed
2020-08-25T15:16:22+12:00 ERROR - Suite shutting down - AUTOMATIC(ON-TASK-FAILURE)
2020-08-25T15:16:22+12:00 INFO - DONE

Deprecation warning present if I add the option to my flow.cylc.

2020-08-25T15:20:10+12:00 WARNING - deprecated items were automatically upgraded in 'suite definition':
2020-08-25T15:20:10+12:00 WARNING -  * (8.0.0) [cylc][abort if any task fails] - DELETED (OBSOLETE)
2020-08-25T15:20:11+12:00 INFO - Cold Start 1

[wxtim]

Can I poke @oliver-sanders for a re-review

[oliver-sanders]

Looks ok.

Just thought of something that needs doing, will need to automatically add --abort-if-any-task-fails to the restart command when we form it in suite_auto_restart.

cylc-flow/cylc/flow/scheduler.py

Line 1548 in b397d97

cmd = ['cylc', 'restart', quote(self.suite)]

[oliver-sanders]

Edited description to close issue, waiting for tests to pass.

[hjoliver]

Tests passed.