Isolated start-up and shut-down graphs.

[hjoliver]

Close #4912
Supersede #5036
(Really addresses the original #4903 but specifically just for start-up and shut-down graphs)

Implements isolated initial and final graphs as special non-advancing cycle points, to avoid the pain of eternal dependence on initial and final tasks.

The initial cycle point is called alpha and the final one omega.

• first and last letters of the greek alphabet
• avoids potential confusion with the initial/final/start/stop cycle points of the main graph
• (these names are a minor detail, subject to discussion!)

How it works:

• at start-up, the alpha graph loads and runs to completion
• then the main (cycling) graph loads, and runs to completion
• then the omega graph loads and runs to completion

Also:

• restart works as normal, in any of the graphs
• you can manually trigger tasks in any section, in which case (e.g.) alpha/foo will coexist in the task pool with 3/bar etc.
  • if you need to retrigger an initial install task (e.g.) part way through a workflow run, you may or may not want to pause the main flow while it runs - to handle this we should implement flow pause (which is intended anyway, but not done yet)

EXAMPLE

[scheduling]
    [[graph]]
        alpha = "a => b"
        omega = "x => y"
        R1 = "foo => bar"
[runtime]
    [[a, b, x, y, foo, bar]]

Result:

$ cylc cat-log junk | grep '=> running'
INFO - [alpha/a submitted job:01 flows:1] => running
INFO - [alpha/b submitted job:01 flows:1] => running
INFO - [1/foo submitted job:01 flows:2] => running
INFO - [1/bar submitted job:01 flows:2] => running
INFO - [omega/x submitted job:01 flows:3] => running
INFO - [omega/y submitted job:01 flows:3] => running

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg and conda-environment.yml.
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at Document alpha and omega graphs. cylc-doc#559
• If this is a bug fix, PRs raised to both master and the relevant maintenance branch.

[hjoliver]

I'm tentatively putting this one back to 8.1.0.

• It's ready to go and shouldn't be hard to review (and the underlying principles were discussed at length and agreed)
• It's a nice feature that many users will like (it makes special behaviour at start much easier to understand and configure)
• It has some nasty conflict potential if left too long

[hjoliver]

Merged master and deconflicted.

[hjoliver]

@dpmatthews - pinging you for a big-picture review (given your involvement in the discussions that led to this).

[hjoliver]

Squashed and rebased.

[hjoliver]

(Rebased)

[hjoliver]

(Rebased)

[hjoliver]

(Rebased)

[dwsutherland]

I'm annoyed I didn't review this one earlier.. A welcome feature for NIWA Ops.
(have cheekily added this to the 8.4.0 milestone)

[oliver-sanders]

8.4.0 is intended to be a small release filling in the remaining intervention holes:

• Cylc remove proposal.
• Skip mode proposal.

This is a large low-priority feature that requires some thought.

Cheekily bumping this back until we are able to give it the time it requires.

[hjoliver]

Yes the new intervention stuff is higher priority, but at the least I'd really like to prioritize this soon after 8.4.0.

This has been requested by many users for a long time. It looks "bigger" than it really is, in that it doesn't mess with the nasty bits of scheduler internals. And, we did discuss and agree it several years back.