job exit script

[hjoliver]

(Coded on a long-haul flight, when I couldn't sleep).

Add support for a new exit-script runtime setting, for scripting that is executed at the very last before a job exits , whether normal exit, or fail, or kill (except SIGKILL, obviously) [successfully].

Similar to the existing err-script, which executes on job failure.

Use case example: automatic deletion heavily used job temp directories, e.g. for jobs that don't use the work directories provided by cylc. (This is evidently a thing for direct ports from SMS at one of our main sites).

Tests added (expect a few test failures based on a new config item being added without adjusting the config tests - TODO)

I won't bother adding documentation until or if agreed that this change is good (or at least, not unreasonable). [documented]

One thing I'm not sure of: perhaps the exit script should only run on successful exit? Or do we need both behaviours as an option? [success only]

[hjoliver]

(I suppose it would be sensible to have this run only for successful exit, along side the err script for job fail).

[matthewrmshin]

Minor comment. Looks OK otherwise. Note an old config test needs adding a bunch of new exit_script = to pass.

[hjoliver]

Old config test fixed.

[trwhitcomb]

One area this is useful (for me at least) is in the handling of the extra log files entry. Right now, those log files just reside in the job log directory. This is great, as it gives me a way to view extra log files through the GUI/CLI - however, the way things are currently set up, I need log files in the share directory, not in a job-specific log directory. I had thought about adding a post-script that copied/linked the log files to the approrpriate locations for GUI viewing later, but ran into a challenge that it wouldn't work if the job wasn't successful. This might help with some of that.

[hjoliver]

Changing this to execute exit-script on job success only.

[hjoliver]

Now run on job success only; documented; ready for review.

Note I've left my use of eval above - but feel free to convince me to replace it with more complex code 😁

[hjoliver]

Assigned to next-release as this is a pretty simple change (and needed at the aforementioned site).

[matthewrmshin]

Change looks OK. Feel free to agree/ignore my eval comment.

[kinow]

Code looks good, docs too. Works as expected on my environment (and Today-I-Learned --set=VAR=VALUE 🌟 ).

# after: cylc run --no-detach --debug --verbose --set=EXIT=true exit-script

kinow@ranma:~/cylc-run/exit-script$ cat log/job/1/foo/01/job.out 
Suite    : exit-script
Task Job : 1/foo/01 (try 1)
User@Host: kinow@ranma

HELLO
2018-11-21T22:57:31+13:00 INFO - started
kinow@ranma:~/cylc-run/exit-script$ cat log/job/1/foo/01/job.err 
Sending DEBUG MODE xtrace to job.xtrace
2018-11-21T22:57:32+13:00 DEBUG - Loading site/user global config files
2018-11-21T22:57:32+13:00 DEBUG - Reading file /home/kinow/Development/python/workspace/cylc/etc/global.rc
2018-11-21T22:57:32+13:00 DEBUG - deprecated items were automatically upgraded in 'site config':
2018-11-21T22:57:32+13:00 DEBUG -  * (6.11.0) [state dump rolling archive length] - DELETED (OBSOLETE)
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [communication][base port] -> [suite servers][run ports] - Format as range list
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [communication][maximum number of ports] - DELETED (OBSOLETE)
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [suite host scanning] -> [suite servers] - value unchanged
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [suite servers][hosts] -> [suite servers][scan hosts] - value unchanged
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [suite logging][roll over at start-up] - DELETED (OBSOLETE)
2018-11-21T22:57:32+13:00 DEBUG - Reading file /home/kinow/.cylc/global.rc
2018-11-21T22:57:32+13:00 CRITICAL - failed/EXIT
2018-11-21T22:57:32+13:00 DEBUG - Loading site/user global config files
2018-11-21T22:57:32+13:00 DEBUG - Reading file /home/kinow/Development/python/workspace/cylc/etc/global.rc
2018-11-21T22:57:32+13:00 DEBUG - deprecated items were automatically upgraded in 'site config':
2018-11-21T22:57:32+13:00 DEBUG -  * (6.11.0) [state dump rolling archive length] - DELETED (OBSOLETE)
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [communication][base port] -> [suite servers][run ports] - Format as range list
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [communication][maximum number of ports] - DELETED (OBSOLETE)
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [suite host scanning] -> [suite servers] - value unchanged
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [suite servers][hosts] -> [suite servers][scan hosts] - value unchanged
2018-11-21T22:57:32+13:00 DEBUG -  * (7.8.0) [suite logging][roll over at start-up] - DELETED (OBSOLETE)
2018-11-21T22:57:32+13:00 DEBUG - Reading file /home/kinow/.cylc/global.rc
EXIT Oops!

[kinow]

Two approvals, and set to next-release. LGTM.