From 78578d3dfff529d3183b3894c0471668b6e407a6 Mon Sep 17 00:00:00 2001 From: Oliver Sanders Date: Tue, 27 Apr 2021 11:41:14 +0100 Subject: [PATCH 1/3] migration guide: add section for documenting cylc8 changes * move cylc 8 change overview into this section * add a cheat sheet containing highlighted cli changes * create a space for the "migration guide" * write the first guide on suicide triggers --- src/7-to-8/cheat-sheet.rst | 279 ++++++++++++++++++ src/7-to-8/index.rst | 21 ++ src/7-to-8/major-changes/index.rst | 11 + src/7-to-8/major-changes/suicide-triggers.rst | 91 ++++++ src/{overview.rst => 7-to-8/summary.rst} | 38 +-- src/index.rst | 2 +- 6 files changed, 417 insertions(+), 25 deletions(-) create mode 100644 src/7-to-8/cheat-sheet.rst create mode 100644 src/7-to-8/index.rst create mode 100644 src/7-to-8/major-changes/index.rst create mode 100644 src/7-to-8/major-changes/suicide-triggers.rst rename src/{overview.rst => 7-to-8/summary.rst} (94%) diff --git a/src/7-to-8/cheat-sheet.rst b/src/7-to-8/cheat-sheet.rst new file mode 100644 index 0000000000..41db45b635 --- /dev/null +++ b/src/7-to-8/cheat-sheet.rst @@ -0,0 +1,279 @@ +Cheat Sheet +=========== + +Quick summary of the command line changes between Cylc 7 / Rose 2019 and Cylc 8. + +.. highlight:: sub + + +Validating +---------- + +Check the workflow configuration for errors: + +.. list-table:: + :class: grid-table + + * - **Cylc 7** + - **Rose 2019** + - **Cylc8** (Rose 2) + * - :: + + cylc validate + - :: + + # validate from $PWD + rose suite-run --validate + - :: + + cylc validate + +Installing & Running +-------------------- + +Install a workflow from source and run it: + +.. list-table:: + :class: grid-table + + * - **Cylc 7** + - **Rose 2019** + - **Cylc8** (Rose 2) + * - :: + + # no installation capibility + # run from source + cylc run + - :: + + # install from $PWD + # then run + rose suite-run + - :: + + # install from $PWD + cylc install + + # run the installed workflow + cylc play + +Pausing & Unpausing +------------------- + +Tell a workflow not to submit any new jobs: + +.. list-table:: + :class: grid-table + + * - **Cylc 7** & Rose 2019 + - **Cylc8** (Rose 2) + * - :: + + cylc hold + + cylc unhold + - :: + + cylc pause + + cylc play + +Stopping +-------- + +Stop a running workflow:: + + cylc stop + +Restarting +---------- + +Restart a stopped workflow and pick up where it left off: + +.. list-table:: + :class: grid-table + + * - **Cylc 7** + - **Rose 2019** + - **Cylc8** (Rose 2) + * - :: + + # no installation capibility + # restart from source + cylc restart + - :: + + # regular restart + rose suite-restart + + Or alternatively:: + + # reinstall and restart + rose suite-run --restart + - :: + + # optionally reinstall + cylc reinstall + + # restart + cylc play + +Deleting +-------- + +Delete the workflow :term:`run directory` (leave source files untouched): + +.. list-table:: + :class: grid-table + + * - **Cylc 7** + - **Rose 2019** + - **Cylc8** (Rose 2) + * - :: + + rm -rf ~/cylc-run/ + - :: + + rose suite-clean + - :: + + cylc clean + + Or alternatively:: + + rm -rf ~/cylc-run/ + +Scanning +-------- + +List all running workflows:: + + cylc scan + +View A Workflow's Configuration +------------------------------- + +View the parsed workflow configuration: + +.. list-table:: + :class: grid-table + + * - **Cylc 7** + - **Rose 2019** + - **Cylc8** (Rose 2) + * - :: + + cylc get-config --sparse \ + + - :: + + # install workflow + rose suite-run -l + + # view installed config + cylc get-config --sparse \ + + - :: + + cylc config --sparse \ + + +Opening User Interfaces +----------------------- + +Opening the graphical user interface (GUI) or terminal user interface (TUI) +for monitoring / controling running workflows: + +.. list-table:: + :class: grid-table + + * - + - **Cylc 7** & Rose 2019 + - **Cylc8** (Rose 2) + * - Terminal + - :: + + cylc monitor + - :: + + cylc tui + * - Graphical + - :: + + cylc gui + - * Open the Cylc Hub's URL in a web browser. + + * Or alternatively run your own hub:: + + cylc hub + + Then open the address ``0.0.0.0:8000`` in a web browser. + +Static Graph Visualisation +-------------------------- + +Generate a visualisation for a workflow without running it: + +.. list-table:: + :class: grid-table + + * - **Cylc 7** & Rose 2019 + - **Cylc8** (Rose 2) + * - :: + + cylc graph + - Not yet implemnted. As a temporary workaround Graphviz can be used + to render ``cylc graph --reference`` output like so: + + .. code-block:: bash + + #!/usr/bin/env bash + + set -eu + + SUITE="$1" + FMT="$2" + + TMP=dotfile + + cylc graph --reference "$SUITE" 2>/dev/null > "$TMP.ref" + + sed \ + -e 's/node "\(.*\)" "\(.*\)"/"\1" [label="\2"]/' \ + -e 's/edge "\(.*\)" "\(.*\)"/"\1" -> "\2"/' \ + -e '1i digraph {' \ + -e '$a}' \ + -e '/^graph$/d' \ + -e '/^stop$/d' \ + "$TMP.ref" \ + > "$TMP.dot" + + dot \ + "$TMP.dot" \ + -T$FMT \ + -o "$TMP.$FMT" + + rm "$TMP.ref" "$TMP.dot" + echo "$TMP.$FMT" + +Rose Stem +--------- + +Run a :ref:`rose:Rose Stem` test suite. + +.. list-table:: + :class: grid-table + + * - **Rose 2019** + - **Rose 2** (Cylc8) + * - :: + + # install and start + rose stem + - :: + + # install + rose stem + + # start + cylc play diff --git a/src/7-to-8/index.rst b/src/7-to-8/index.rst new file mode 100644 index 0000000000..a05bef0b26 --- /dev/null +++ b/src/7-to-8/index.rst @@ -0,0 +1,21 @@ +Cylc 8 Migration Guide +====================== + +**A quick guide for Cylc 7 (and Rose 2019) users upgrading to Cylc 8.** + +.. tip:: + + If you need help using or understanding Cylc 8 please post questions to the + `Cylc 8 Migration category `_ + on the `Cylc Forum `_ + + +Cylc 8 differs from Cylc 7 in many ways: architecture, scheduling +algorithm, security, UIs, working practices, and more. + +.. toctree:: + :maxdepth: 2 + + summary + cheat-sheet + major-changes/index diff --git a/src/7-to-8/major-changes/index.rst b/src/7-to-8/major-changes/index.rst new file mode 100644 index 0000000000..d93e685e0c --- /dev/null +++ b/src/7-to-8/major-changes/index.rst @@ -0,0 +1,11 @@ +Major Changes +============= + +This section goes into detail on some of the major changes between Cylc 7 and +Cylc 8. + +.. toctree:: + :maxdepth: 1 + :glob: + + * diff --git a/src/7-to-8/major-changes/suicide-triggers.rst b/src/7-to-8/major-changes/suicide-triggers.rst new file mode 100644 index 0000000000..c826f32da3 --- /dev/null +++ b/src/7-to-8/major-changes/suicide-triggers.rst @@ -0,0 +1,91 @@ +Suicide Triggers +================ + +.. admonition:: Does This Change Affect Me? + :class: tip + + Suicide triggers are written using an exclamation mark ``!`` like so: + + .. code-block:: cylc-graph + + task_1 => ! task_2 + + Read this section if your workflows contain suicide triggers. + +Overview +-------- + +In Cylc 7 :term:`suicide triggers ` are used to remove +tasks from the graph automatically whilst a workflow is running. + +Cylc 8 handles :term:`graphs ` in an event-driven manner which means +that a workflow can follow different paths in different eventualities without +the need for suicide triggers. This is called :term:`graph branching`. + + +Changes +------- + +No changes are required, suicide triggers are still supported in Cylc 8, +however, they are most likely unnecessary. + + +Example +------- + +Here is an example Cylc 7 :term:`graph`: + +.. code-block:: cylc-graph + + # Regular graph. + foo => bar + + # The fail case. + bar:fail => recover + + # Remove the "recover" task in the success case. + bar => ! recover + + # Remove the "bar" task in the fail case. + recover => ! bar + + # Downstream dependencies. + bar | recover => baz + +Which results in the following logic: + +.. digraph:: Example + :align: center + + subgraph cluster_1 { + label = ":fail" + color = "red" + fontcolor = "red" + style = "dashed" + recover + } + + foo -> bar + bar -> recover + recover -> baz [arrowhead="onormal"] + bar -> baz [arrowhead="onormal"] + +In Cylc 8 the suicide triggers can be safely removed without changing the +workflow: + +.. code-block:: diff + + # Regular graph. + foo => bar + + # The fail case. + bar:fail => recover + + - # Remove the "recover" task in the success case. + - bar => ! recover + + - # Remove the "bar" task in the fail case. + - recover => ! bar + + # Downstream dependencies. + bar | recover => baz diff --git a/src/overview.rst b/src/7-to-8/summary.rst similarity index 94% rename from src/overview.rst rename to src/7-to-8/summary.rst index 13b8b05d0d..5f94f6f1e1 100644 --- a/src/overview.rst +++ b/src/7-to-8/summary.rst @@ -1,18 +1,8 @@ .. _overview: -What's New in Cylc 8.0b0? -========================= +Quick Summary Of Changes +======================== -**A quick guide for Cylc 7 users.** - -.. tip:: - - If you need help using or understanding Cylc 8 please post questions to the - `Cylc Forum `_ - - -Cylc 8 differs from Cylc 7 in many ways: architecture, scheduling -algorithm, security, UIs, working practices, and more. Backward Compatibility ---------------------- @@ -43,6 +33,11 @@ Architecture Cylc has been re-architected to support a remote web UI. The main Cylc 8 system components are: +- **Cylc Scheduler** + - The workflow engine core, Python 3 based + - Includes the **CLI** (Command Line Interface) + - And **TUI**, a new Terminal UI application + - **Cylc Hub** - Authenticates users, spawns and proxies Cylc UI Servers - (A `Jupyterhub `_ instance) @@ -53,11 +48,6 @@ components are: - Interacts with Schedulers and the filesystem - Serves the UI to users -- **Cylc Scheduler** - - The workflow engine core, Python 3 based - - Includes the **CLI** (Command Line Interface) - - And **TUI**, a new Terminal UI application - - **Cylc UI** - In-browser web UI, includes: - A dashboard with summary information and documentation links @@ -69,25 +59,25 @@ components are: - **Network layers** - Incremental push updates (c.f. polled full-state updates in Cylc 7) -.. figure:: img/hub.png +.. figure:: ../img/hub.png :figwidth: 100% :align: center Cylc 8 Hub authentication page -.. figure:: img/cylc-ui-dash.png +.. figure:: ../img/cylc-ui-dash.png :figwidth: 100% :align: center Cylc 8 UI dashboard -.. figure:: img/cylc-ui-tree.png +.. figure:: ../img/cylc-ui-tree.png :figwidth: 100% :align: center Cylc 8 UI workflow tree view -.. figure:: img/cylc-tui.png +.. figure:: ../img/cylc-tui.png :figwidth: 100% :align: center @@ -98,7 +88,7 @@ Scheduling Algorithm Cylc has to be able to manage infinite workflows of repeating tasks: -.. image:: img/cycling.png +.. image:: ../img/cycling.png :align: center See :ref:`Cylc 7 Scheduler Deficiencies Fixed by Cylc 8` @@ -126,7 +116,7 @@ Cylc 8 has only 8 task/job states. The Cylc 8 UI shows both task and jobs. Task icons are monochrome circles; job icons are coloured squares. The running task icon incorporates a radial progress indicator. -.. image:: img/task-job.png +.. image:: ../img/task-job.png :align: center The removed Cylc 7 task states have been absorbed into the *waiting* state, but @@ -137,7 +127,7 @@ associated jobs must be about to retry. Window on the Workflow ---------------------- -.. image:: img/n-window.png +.. image:: ../img/n-window.png :align: center The Cylc UI can't show "all the tasks" at once because the graph may be huge, diff --git a/src/index.rst b/src/index.rst index 7531603351..6423e4a965 100644 --- a/src/index.rst +++ b/src/index.rst @@ -14,7 +14,7 @@ indefinitely. .. toctree:: :maxdepth: 2 - overview + 7-to-8/index installation introduction/index tutorial/index From ebcb2aba51f8d413f1e24a9df62e881f6d44c6a3 Mon Sep 17 00:00:00 2001 From: Oliver Sanders Date: Wed, 28 Apr 2021 09:53:31 +0100 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: Hilary James Oliver --- src/7-to-8/cheat-sheet.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/7-to-8/cheat-sheet.rst b/src/7-to-8/cheat-sheet.rst index 41db45b635..dd6bcd2583 100644 --- a/src/7-to-8/cheat-sheet.rst +++ b/src/7-to-8/cheat-sheet.rst @@ -41,7 +41,7 @@ Install a workflow from source and run it: - **Cylc8** (Rose 2) * - :: - # no installation capibility + # no installation capability # run from source cylc run - :: @@ -98,7 +98,7 @@ Restart a stopped workflow and pick up where it left off: - **Cylc8** (Rose 2) * - :: - # no installation capibility + # no installation capability # restart from source cylc restart - :: @@ -207,7 +207,7 @@ for monitoring / controling running workflows: cylc hub - Then open the address ``0.0.0.0:8000`` in a web browser. + Then open the URL ``0.0.0.0:8000`` in a web browser. Static Graph Visualisation -------------------------- @@ -222,7 +222,7 @@ Generate a visualisation for a workflow without running it: * - :: cylc graph - - Not yet implemnted. As a temporary workaround Graphviz can be used + - Not yet implemented. As a temporary workaround Graphviz can be used to render ``cylc graph --reference`` output like so: .. code-block:: bash From b4626020858a7fd47223028df3d2df4dd74f11be Mon Sep 17 00:00:00 2001 From: Oliver Sanders Date: Wed, 28 Apr 2021 13:41:51 +0100 Subject: [PATCH 3/3] migration guide: document 8.0b* caveats --- src/7-to-8/caveats.rst | 143 +++++++++++++++++++++++++++++++++++++++++ src/7-to-8/index.rst | 1 + 2 files changed, 144 insertions(+) create mode 100644 src/7-to-8/caveats.rst diff --git a/src/7-to-8/caveats.rst b/src/7-to-8/caveats.rst new file mode 100644 index 0000000000..215d05a52e --- /dev/null +++ b/src/7-to-8/caveats.rst @@ -0,0 +1,143 @@ +Cylc |version| Caveats +====================== + +This is an early release of Cylc, there are some loose ends and features which +have not yet been (re)implemented. + +The documentation has not been fully reviewed and command line information may +be out of date in some cases. + + +Cylc Flow +--------- + +Run Numbers + By default when you install a workflow with ``cylc install`` it is installed + into a numbered subdirectory e.g. ``run1``. A subsequent installation + would go into ``run2``, etc. + + At present the full workflow registration *including* this run number must + be provided to Cylc commands e.g:: + + $ cylc play myflow/run1 + + We plan to infer the most recent run to make this easier e.g:: + + $ cylc play myflow + + * https://github.com/cylc/cylc-flow/issues/3895 +CLI Change + We plan to move to a new way of specifying workflow, cycles, tasks and jobs + on the command line (with back support for the old format). + + This will open up performing operations on multiple workflows in a single + command e.g:: + + $ cylc stop '*' # stop everything + + * https://github.com/cylc/cylc-flow/pull/3931 +Platform Selection + When Cylc needs to perform an action on a platform (e.g. submit a job) + it picks a random host from the platform. If this host is down the operation + will fail. + + In a later relase Cylc will pick another host in order to provide resilience + to system issues. + + * https://github.com/cylc/cylc-flow/issues/3827 +Trigger Edit + Functionality removed pending reimplementation. + + * https://github.com/cylc/cylc-flow/issues/3751 +Hold / Release + The ability to hold tasks is in place, however, doesn't work for most uses. + This will be fixed in a future beta release. + + * https://github.com/cylc/cylc-flow/issues/3743 +Reflow + The new "reflow" functionality which allows multiple + (potentially parallel) executions of the same workflow in a single + :term:`scheduler` is not fully supported by all commands and is + yet to be documented. + + +Browser Based UI +---------------- + +The old "GUI" has been replaced by the new browser-based "UI". + +Graph View + There is no graph view in the Cylc UI as yet. A new graph view will be + developed providing both "live" (AKA ``cylc gui``) and "offline" + (AKA ``cylc graph``) functionalities. + + * https://github.com/cylc/cylc-ui/issues/74 + * https://github.com/cylc/cylc-ui/issues/82 +Log Files + The ability to view job log and other files is yet to be implemented. +Multiple Selection + Multiple selection is yet to be implemented, however, it is possible + to issue action for multiple tasks (e.g. ``kill``) without using + multiple selection: + + * From the UI click on a workflow/cycle/task/job. + * Find the action you want to call (e.g. kill). + * Click the pencil symbol next to this action. + * Edit the workflows/cycles/tasks/jobs in the form and press submit. + + * https://github.com/cylc/cylc-ui/issues/434 +Installing Workflows + At present there is no way to view uninstalled workflows in the UI. + We will add the ability to view and install workflows from the UI. +Rose Edit + Rose Edit is awaiting reimplementation in the UI. +Trigger Edit + Functionality removed pending reimplementation. +Xtrigger Visibility + Xtriggers are not yet visible in the UI. + + * https://github.com/cylc/cylc-ui/issues/331 +Documentation / Orientation Guide + Some form of documentation will be provided within the UI itself. + + * https://github.com/cylc/cylc-ui/issues/155 + + +Terminal User Interface +----------------------- + +The ``cylc tui`` command (Tui) replaces the old ``cylc monitor``, it provides a +tree view that is very similar to the Cylc UI and supports some control +functionality. + +Performance + TUI currently refreshes its display every second. Large workflows which + change rapidly may evolve faster than TUI is able to keep pace with which + will cause TUI to freeze. + + A more performant implementation which does not rely on a scheduled global + update will follow in due course. + + * https://github.com/cylc/cylc-flow/issues/3527 +GScan + The old ``cylc gscan`` command has been removed. You can now find the gscan + display on the left-hand side of the Cylc UI. + + In a future release ``cylc tui`` will be able to list workflows in a similar + way. + + * https://github.com/cylc/cylc-flow/issues/3464 + + +UI Server +--------- + +Authorisation + Only "binary" authorisation (i.e. no access / full control) is currently + supported. + + * https://github.com/cylc/cylc-uiserver/issues/10 +CLI Via UIS + The ability to route Cylc commands via the UIS is planned for a future relase + + * https://github.com/cylc/cylc-flow/issues/3528 diff --git a/src/7-to-8/index.rst b/src/7-to-8/index.rst index a05bef0b26..eca1813093 100644 --- a/src/7-to-8/index.rst +++ b/src/7-to-8/index.rst @@ -17,5 +17,6 @@ algorithm, security, UIs, working practices, and more. :maxdepth: 2 summary + caveats cheat-sheet major-changes/index