Improve migration guide #490
Conversation
|
(I'll get onto this review pronto...) |
| Restarting a Cylc 7 workflow | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| Cylc 8 cannot *restart* a Cylc 7 workflow mid-run. Instead, :ref:`install | ||
| <Installing-workflows>` the workflow to a new run directory and start it | ||
| from scratch at the right cycle point or task(s): | ||
|
|
||
| To see if your workflow is compatible with Cylc 8, run ``cylc validate`` | ||
| **using Cylc 7**. | ||
| - ``cylc play --start-cycle-point=<cycle>`` (c.f. Cylc 7 *warm start*), or | ||
| - ``cylc play --start-task=<cycle/task>`` (Cylc 8 can start anywhere in the graph) | ||
|
|
||
| If tasks in your workflow call Cylc commands directly it might be necessary to | ||
| modify them to be compatible with command line interface changes | ||
| (unfortunately, this is not possible to detect at validation). | ||
| .. note:: | ||
|
|
||
| Any previous-cycle workflow data needed by the new run will need to be | ||
| manually copied over from the original run directory. | ||
|
|
||
|
|
There was a problem hiding this comment.
I think we should drop this, continuing a Cylc 7 workflow run with Cylc 8 is a tricky use case with caveats. Easier just to say Cylc 8 cannot restart Cylc 7 workflows.
There was a problem hiding this comment.
It will be a very common requirement though. Should we not provide advice on how to do it?
Maybe don't call it "restarting", to avoid confusion. Perhaps something like How to start your migrated workflow from where it got to under Cylc 7. And give general advice and caveats.
There was a problem hiding this comment.
Or "Restarting" a Cylc 7 workflow?
There was a problem hiding this comment.
It will be a very common requirement though.
I'm not aware of any requirement or desire at our end. I think the only use case would be really, really, long-running workflows, I'm not aware that we have any examples of this. If they exist I would advise finishing the workflow run with Cylc 7 rather than to attempting to migrate to Cylc 8 part-way through a run.
Migrating like this is hazardous, e.g. remote installation works differently, data might need to be relocated on multiple filesystems, etc. We discussed this in a meeting at our end?
There was a problem hiding this comment.
I've replaced this subsection with a bullet point saying you can't restart, use Cylc 7 to finish a Cylc 7 workflow
There was a problem hiding this comment.
I'm not aware of any requirement or desire at our end. I think the only use case would be really, really, long-running workflows, I'm not aware that we have any examples of this.
Most of our workflows are "really, really, long-running" - i.e. they have no final cycle point, or a very distant one. And most are "warm cycling" - i.e there are inter-cycle dependencies, and doing a cold start from scratch is not something you want to do unless you have no choice.
Anyhow, we may have got our wires crossed here. I just meant "advise users how to continue a warm-cycling Cylc 7 workflow with Cylc 8", not a literal in-place restart from the DB.
Along the lines of:
Shut down (with Cylc 7) after a particular cycle point, copy any inter-cycle data over to a new Cylc 8 install (taking account of the run-name if appropriate), and cold start that with Cylc 8 at the next cycle point.
| foo => bar | ||
| foo:fail => recover | ||
|
|
||
| # The fail case. | ||
| bar:fail => recover | ||
| foo | recover => bar | ||
|
|
||
| # Remove the "recover" task in the success case. | ||
| bar => ! recover | ||
|
|
||
| # Remove the "bar" task in the fail case. | ||
| recover => ! bar | ||
|
|
||
| # Downstream dependencies. | ||
| bar | recover => baz | ||
|
|
||
| Which results in the following logic: | ||
| foo => ! recover | ||
| # Remove the "foo" task in the fail case. | ||
| recover => ! foo |
There was a problem hiding this comment.
IMO this was clearer before.
There was a problem hiding this comment.
I have taken out the first task because it was unnecessary for the example.
That first edge was confusingly commented as the success case, but it wasn't really, the bar => baz was the success case
graph TD
subgraph before
A(foo) -->|unnecessary| B(bar) -->|success case| C(baz)
B -.->|fail case| D(recover) -.-> C
end
graph TD
subgraph after
A(foo) --> B(bar)
A -.-> C(recover) -.->B
end
What part of it would you say was clearer before?
There was a problem hiding this comment.
Yeah the first task was unnecessary. It was probably abstracted from an old example that was meant to look more realistic.
|
@wxtim You may want to poke at this one because it will conflict with your platforms PR. |
|
I have taken the (squashed) commits from this branch as-is and added to branch-8.0rc3 so I can re-deploy the docs. I will do that now, can always re-deploy again if any further changes are needed |
Ah, that explains why I was seeing your changes here, online already!! 😕 |
No worries - my PR is small enough that I'm happy to deal with the conflict. 😄 |
…n migration guide
Co-authored-by: Hilary James Oliver <hilary.j.oliver@gmail.com>
Co-authored-by: Hilary James Oliver <hilary.j.oliver@gmail.com>
| In Cylc 8 we mark the :term:`outputs <task output>` which are | ||
| :term:`optional <optional output>` (in this case ``bar:success`` and | ||
| ``bar:fail``) with a ``?`` in the graph. | ||
| In Cylc 8, all task outputs are :term:`expected <expected output>` to complete |
There was a problem hiding this comment.
(I'll update 492 after merging this)
|
Two approvals, and time is of the essence this week. I'll post a small follow-up. Thanks @MetRonnie |
|
Pushed squashed commit to branch-8.0rc3 and redeploying... Update: can't deploy right now, linkcheck failing because man.openbsd.org is down Update 2: deployed. (I imagine this will be the last for 8.0rc3) |
Add content to and reorganise into more logical order:
Also update slightly incorrect
CYLC_VERSIONJinja2 variable thingWould really like to merge this by Wed 6th morning (UTC) + redeploy 8.0rc3 docs in time for workshop, sorry for the short notice!
Requirements check-list
CONTRIBUTING.mdand added my name as a Code Contributor.