diff --git a/cylc/doc/etc/tutorial/cylc-forecasting-suite/.validate b/cylc/doc/etc/tutorial/cylc-forecasting-suite/.validate index ad5d478caa..7f2bde5a0e 100755 --- a/cylc/doc/etc/tutorial/cylc-forecasting-suite/.validate +++ b/cylc/doc/etc/tutorial/cylc-forecasting-suite/.validate @@ -20,6 +20,6 @@ APIKEY="$(head --lines 1 ../api-keys)" FLOW_NAME="$(< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c6)" cylc install --flow-name "$FLOW_NAME" --no-run-name . sed -i "s/DATAPOINT_API_KEY/$APIKEY/" "$HOME/cylc-run/$FLOW_NAME/flow.cylc" -cylc validate --strict --icp=2000 "$FLOW_NAME" +cylc validate --check-circular --icp=2000 "$FLOW_NAME" cylc play --no-detach --abort-if-any-task-fails "$FLOW_NAME" cylc clean "$FLOW_NAME" diff --git a/cylc/doc/etc/tutorial/retries-tutorial/.validate b/cylc/doc/etc/tutorial/retries-tutorial/.validate index 028c1e8200..f54e75a091 100755 --- a/cylc/doc/etc/tutorial/retries-tutorial/.validate +++ b/cylc/doc/etc/tutorial/retries-tutorial/.validate @@ -18,4 +18,4 @@ set -eux -cylc validate --strict . --icp=2000 +cylc validate --check-circular . --icp=2000 diff --git a/cylc/doc/etc/tutorial/runtime-introduction/.validate b/cylc/doc/etc/tutorial/runtime-introduction/.validate index 5577b69ad4..d5640d0920 100755 --- a/cylc/doc/etc/tutorial/runtime-introduction/.validate +++ b/cylc/doc/etc/tutorial/runtime-introduction/.validate @@ -18,6 +18,6 @@ set -eux FLOW_NAME="$(< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c6)" cylc install --flow-name "$FLOW_NAME" --no-run-name . -cylc validate --strict --icp=2000 "$FLOW_NAME" +cylc validate --check-circular --icp=2000 "$FLOW_NAME" cylc play --no-detach --abort-if-any-task-fails "$FLOW_NAME" cylc clean "$FLOW_NAME" diff --git a/src/glossary.rst b/src/glossary.rst index 2af05e0bbb..5747ef02be 100644 --- a/src/glossary.rst +++ b/src/glossary.rst @@ -412,6 +412,28 @@ Glossary * :term:`job` * :term:`qualifier` + implicit task + An implicit task (previously known as a naked task) is a task in the + graph that does not have an explicit runtime definition. + For example, ``bar`` is an implicit task in the following workflow: + + .. code-block:: cylc + + [scheduling] + [[graph]] + R1 = foo & bar + [runtime] + [[foo]] + + Implicit tasks are not allowed by default, as they are often typos. + However, it is possible to allow them using + :cylc:conf:`flow.cylc[scheduler]allow implicit tasks` during + development of a workflow. + + See also: + + * :ref:`ImplicitTasks` + run directory When a :term:`suite ` is run a directory is created for all of the files generated whilst the suite is running. This is called the diff --git a/src/suite-design-guide/efficiency.rst b/src/suite-design-guide/efficiency.rst index 3c3603816a..b2399cf5f6 100644 --- a/src/suite-design-guide/efficiency.rst +++ b/src/suite-design-guide/efficiency.rst @@ -49,7 +49,7 @@ factored out into a task family inherited by all. inherit = OBSPROC If several families have settings in common, they can in turn can inherit -from higher-level families. +from higher-level families. Multiple inheritance allows efficient sharing even for overlapping categories of tasks. For example consider that some obs processing tasks in the following @@ -132,6 +132,8 @@ to allow chaining of dependencies: Family-to-Family Triggering ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. TODO: Is this section still true post-SoD? + .. code-block:: cylc [scheduling] @@ -142,7 +144,7 @@ This means every member of ``BIG_FAM_2`` depends on every member of ``BIG_FAM_1`` succeeding. For very large families this can create so many dependencies that it affects the performance of Cylc at run time, as well as cluttering graph visualizations with unnecessary edges. Instead, -interpose a dummy task that signifies completion of the first family: +interpose a blank task that signifies completion of the first family: .. code-block:: cylc @@ -150,7 +152,7 @@ interpose a dummy task that signifies completion of the first family: [[graph]] R1 = BIG_FAM_1:succeed-all => big_fam_1_done => BIG_FAM_2 -For families with ``M`` and ``N`` members respectively, this +For families with ``M`` and ``N`` members respectively, this reduces the number of dependencies from ``M*N`` to ``M+N`` without affecting the scheduling. @@ -265,6 +267,7 @@ example using suite parameters instead of Jinja2 loops: model => check """ [runtime] + [[pre, post, check]] [[model

]] script = echo "my parameter value is ${CYLC_TASK_PARAM_p}" [[model]] @@ -279,6 +282,7 @@ values: ``chunk => chunk

``. Here's a multi-parameter example: .. code-block:: cylc [scheduler] + allow implicit tasks = True [[parameters]] run = a, b, c m = 1..5 @@ -303,6 +307,7 @@ For example, this: [[graph]] R1 = pre => model => post [runtime] + [[pre, post]] [[MODELS]] [[model]] inherit = MODELS @@ -318,6 +323,7 @@ is equivalent to this: [[graph]] R1 = pre => MODELS:succeed-all => post [runtime] + [[pre, post]] [[MODELS]] [[model]] inherit = MODELS @@ -360,7 +366,7 @@ or by environment variable: ROSE_APP_OPT_CONF_KEYS = key1 key2 The environment variable is generally preferred in suites because you don't -have to repeat and override the root-level script configuration: +have to repeat and override the root-level script configuration: .. code-block:: cylc diff --git a/src/suites/empy/cities/flow.cylc b/src/suites/empy/cities/flow.cylc index 1338abfdcd..d2de162674 100644 --- a/src/suites/empy/cities/flow.cylc +++ b/src/suites/empy/cities/flow.cylc @@ -6,6 +6,9 @@ generation of groups of related dependencies and runtime properties. """ +[scheduler] + allow implicit tasks = True + @{ HOST = "SuperComputer" CITIES = 'NewYork', 'Philadelphia', 'Newark', 'Houston', 'SantaFe', 'Chicago' diff --git a/src/suites/inherit/single/one/flow.cylc b/src/suites/inherit/single/one/flow.cylc index d02facce50..d296ffd905 100644 --- a/src/suites/inherit/single/one/flow.cylc +++ b/src/suites/inherit/single/one/flow.cylc @@ -3,9 +3,6 @@ [meta] title = "User Guide [runtime] example." -[scheduler] - required run mode = simulation # (no task implementations) - [scheduling] initial cycle point = 20110101T06 final cycle point = 20110102T00 @@ -39,8 +36,5 @@ [[[environment]]] RUNNING_DIR = $HOME/running/ship # override OBS environment OUTPUT_DIR = $HOME/output/ship # add to OBS environment - [[foo]] - # (just inherits from root) - - # The task [[bar]] is implicitly defined by its presence in the - # graph; it is also a dummy task that just inherits from root. + [[foo, bar]] + # (just inherit from root) diff --git a/src/suites/inherit/single/two/flow.cylc b/src/suites/inherit/single/two/flow.cylc index 05507282e3..41ff8bfdce 100644 --- a/src/suites/inherit/single/two/flow.cylc +++ b/src/suites/inherit/single/two/flow.cylc @@ -20,6 +20,7 @@ [[root]] [[[environment]]] ROOT = "the quick brown fox" + [[foo]] [[GEN]] [[[environment]]] GEN_A = gen_a diff --git a/src/suites/jinja2/cities/flow.cylc b/src/suites/jinja2/cities/flow.cylc index 14768aeb03..6d94383098 100644 --- a/src/suites/jinja2/cities/flow.cylc +++ b/src/suites/jinja2/cities/flow.cylc @@ -6,6 +6,9 @@ generation of groups of related dependencies and runtime properties. """ +[scheduler] + allow implicit tasks = True + {% set HOST = "SuperComputer" %} {% set CITIES = 'NewYork', 'Philadelphia', 'Newark', 'Houston', 'SantaFe', 'Chicago' %} {% set CITYJOBS = 'one', 'two', 'three', 'four' %} @@ -55,4 +58,4 @@ cleaning = clean, cleanup [[node attributes]] cleaning = 'style=filled', 'fillcolor=yellow' - NewYork = 'style=filled', 'fillcolor=lightblue' + NewYork = 'style=filled', 'fillcolor=lightblue' diff --git a/src/suites/jinja2/defaults/flow.cylc b/src/suites/jinja2/defaults/flow.cylc index 7318282cea..37ed3dbf2b 100644 --- a/src/suites/jinja2/defaults/flow.cylc +++ b/src/suites/jinja2/defaults/flow.cylc @@ -17,6 +17,7 @@ """ [runtime] + [[foo, bar]] [[ENS]] {% for I in range( 0, N_MEMBERS | default( 3 )) %} [[ mem_{{ I }} ]] diff --git a/src/suites/queues/flow.cylc b/src/suites/queues/flow.cylc index 286d7de5bb..debe78e7cb 100644 --- a/src/suites/queues/flow.cylc +++ b/src/suites/queues/flow.cylc @@ -6,6 +6,9 @@ active tasks at once. """ +[scheduler] + allow implicit tasks = True + [scheduling] [[queues]] [[[default]]] diff --git a/src/tutorial/furthertopics/suicide-triggers.rst b/src/tutorial/furthertopics/suicide-triggers.rst index f31de1758c..195fb38a87 100644 --- a/src/tutorial/furthertopics/suicide-triggers.rst +++ b/src/tutorial/furthertopics/suicide-triggers.rst @@ -109,15 +109,17 @@ Paste the following code into the :cylc:conf:`flow.cylc` file: .. code-block:: cylc + [scheduler] + allow implicit tasks = True [scheduling] - cycling mode = integer - initial cycle point = 1 - runahead limit = P3 - [[graph]] - P1 = """ - make_cake_mixture => bake_cake => sell_cake - bake_cake:fail => eat_cake - """ + cycling mode = integer + initial cycle point = 1 + runahead limit = P3 + [[graph]] + P1 = """ + make_cake_mixture => bake_cake => sell_cake + bake_cake:fail => eat_cake + """ [runtime] [[root]] script = sleep 2 diff --git a/src/tutorial/runtime/configuration-consolidation/jinja2.rst b/src/tutorial/runtime/configuration-consolidation/jinja2.rst index 9f9a77f97e..d95f43759f 100644 --- a/src/tutorial/runtime/configuration-consolidation/jinja2.rst +++ b/src/tutorial/runtime/configuration-consolidation/jinja2.rst @@ -136,6 +136,9 @@ This would result in: 'heathrow': 3772, 'shetland': 3005} %} + [scheduler] + allow implicit tasks = True + [scheduling] [[graph]] T00/PT3H = """ diff --git a/src/tutorial/runtime/introduction.rst b/src/tutorial/runtime/introduction.rst index 53e4cfbca8..ebac0be387 100644 --- a/src/tutorial/runtime/introduction.rst +++ b/src/tutorial/runtime/introduction.rst @@ -37,6 +37,13 @@ The Task Section [runtime] [[hello_world]] +.. note:: + + This runtime sub-section is normally required, even if it is empty. However, + in the previous tutorials, we disabled this requirement using the setting + :cylc:conf:`flow.cylc[scheduler]allow implicit tasks`. + See :ref:`ImplicitTasks` for more details. + The ``script`` Setting ---------------------- diff --git a/src/tutorial/scheduling/datetime-cycling.rst b/src/tutorial/scheduling/datetime-cycling.rst index c3ebd7f75e..547612352e 100644 --- a/src/tutorial/scheduling/datetime-cycling.rst +++ b/src/tutorial/scheduling/datetime-cycling.rst @@ -439,6 +439,7 @@ Putting It All Together [scheduler] UTC mode = True + allow implicit tasks = True [scheduling] initial cycle point = 20000101T00Z [[graph]] @@ -466,6 +467,7 @@ Putting It All Together [scheduler] UTC mode = True + allow implicit tasks = True [scheduling] initial cycle point = 20000101T00Z [[graph]] @@ -511,6 +513,7 @@ Putting It All Together [scheduler] UTC mode = True + allow implicit tasks = True [scheduling] initial cycle point = 20000101T00Z [[graph]] diff --git a/src/tutorial/scheduling/graphing.rst b/src/tutorial/scheduling/graphing.rst index 2980cd022a..844e65450e 100644 --- a/src/tutorial/scheduling/graphing.rst +++ b/src/tutorial/scheduling/graphing.rst @@ -335,6 +335,8 @@ Cylc Graphs .. code-block:: cylc + [scheduler] + allow implicit tasks = True [scheduling] [[graph]] R1 = """ @@ -404,6 +406,8 @@ Cylc Graphs .. code-block:: cylc + [scheduler] + allow implicit tasks = True [scheduling] [[graph]] R1 = """ diff --git a/src/tutorial/scheduling/integer-cycling.rst b/src/tutorial/scheduling/integer-cycling.rst index 89d2eeb14a..5423e8e005 100644 --- a/src/tutorial/scheduling/integer-cycling.rst +++ b/src/tutorial/scheduling/integer-cycling.rst @@ -413,6 +413,8 @@ Recurrence Sections .. code-block:: cylc + [scheduler] + allow implicit tasks = True [scheduling] [[graph]] R1 = """ @@ -572,7 +574,8 @@ Recurrence Sections .. code-block:: cylc - + [scheduler] + allow implicit tasks = True [scheduling] cycling mode = integer initial cycle point = 1 diff --git a/src/user-guide/running-suites.rst b/src/user-guide/running-suites.rst index f92da9c637..1347ff147e 100644 --- a/src/user-guide/running-suites.rst +++ b/src/user-guide/running-suites.rst @@ -78,9 +78,9 @@ All workflows have an :term:`initial cycle point` and many have a :term:`final cycle point`. These determine the range between which Cylc will schedule tasks to run. -By default when you launch a Cylc :term:`scheduler` to run the workflow +By default when you launch a Cylc :term:`scheduler` to run the workflow, it will start at the :term:`initial cycle point` and stop at the -:term:`final cycle point`, however, it is possible to start and stop the +:term:`final cycle point`. However, it is possible to start and stop the scheduler at any arbitrary point. To do this we use a :term:`start cycle point` and/or :term:`stop cycle point` @@ -1057,12 +1057,12 @@ Several suite run modes allow you to simulate suite behaviour quickly without running the suite's real jobs - which may be long-running and resource-hungry: dummy mode - Runs dummy tasks as background jobs on configured job hosts. + Runs tasks as background jobs on configured job hosts. This simulates scheduling, job host connectivity, and generates all job files on suite and job hosts. dummy-local mode - Runs real dummy tasks as background jobs on the suite host, which allows + Runs real tasks as background jobs on the suite host, which allows dummy-running suites from other sites. This simulates scheduling and generates all job files on the suite host. @@ -1103,7 +1103,7 @@ directives in your live suite, or else use a custom live mode test suite. The dummy modes ignore all configured task ``script`` items including ``init-script``. If your ``init-script`` is required - to run even dummy tasks on a job host, note that host environment + to run even blank/empty tasks on a job host, note that host environment setup should be done elsewhere. diff --git a/src/user-guide/writing-suites/runtime.rst b/src/user-guide/writing-suites/runtime.rst index 1e80fc6984..0fc3e024b5 100644 --- a/src/user-guide/writing-suites/runtime.rst +++ b/src/user-guide/writing-suites/runtime.rst @@ -390,7 +390,7 @@ Remote Task Hosting ------------------- If a task declares a different platform to the one running the workflow, -Cylc will use non-interactive ssh to execute the task using the +Cylc will use non-interactive ssh to execute the task using the :term:`job runner` and one of the hosts from the :term:`platform` definition (platforms are defined in ``global.cylc[platforms]``). @@ -444,27 +444,30 @@ Remote task log directories, like local ones, are created on the fly, if necessary, during job submission. -Naked Dummy Tasks And Strict Validation ---------------------------------------- +.. _ImplicitTasks: -A *naked dummy task* appears in the suite graph but has no +Implicit Tasks +-------------- + +An :term:`implicit task` appears in the workflow graph but has no explicit runtime configuration section. Such tasks automatically -inherit the default "dummy task" configuration from the root -namespace. This is very useful because it allows functional suites to +inherit the configuration from the root namespace. +This is very useful because it allows functional suites to be mocked up quickly for test and demonstration purposes by simply defining the graph. It is somewhat dangerous, however, because there -is no way to distinguish an intentional naked dummy task from one -generated by typographic error: misspelling a task name in the graph -results in a new naked dummy task replacing the intended task in the -affected trigger expression; and misspelling a task name in a runtime -section heading results in the intended task becoming a dummy task +is no way to distinguish an intentional implicit task from one +caused by typographic error. Misspelling a task name in the graph +results in a new implicit task replacing the intended task in the +affected trigger expression, and misspelling a task name in a runtime +section heading results in the intended task becoming an implicit task itself (by divorcing it from its intended runtime config section). -To avoid this problem any dummy task used in a real suite should not be -naked - i.e. it should have an explicit entry in under the runtime -section of the suite configuration, even if the section is empty. This -results in exactly the same dummy task behaviour, via implicit -inheritance from root, but it allows use of -``cylc validate --strict`` -to catch errors in task names by failing the suite if any naked dummy -tasks are detected. +You can allow implicit tasks during development of a workflow using +:cylc:conf:`flow.cylc[scheduler]allow implicit tasks`. But, to avoid +the problems mentioned above, any task used in a production/operational +workflow should not be implicit, i.e. it should have an explicit entry in under +the runtime section of ``flow.cylc``, even if the section is empty. This +results in exactly the same task behaviour, via inheritance from root, +but adds a layer of protection against mistakes. Thus, it is recommended to +turn off :cylc:conf:`flow.cylc[scheduler]allow implicit tasks` when the +:cylc:conf:`flow.cylc[runtime]` section has been written. diff --git a/src/user-guide/writing-suites/scheduling.rst b/src/user-guide/writing-suites/scheduling.rst index c7d8c9a7cf..993eeb1b97 100644 --- a/src/user-guide/writing-suites/scheduling.rst +++ b/src/user-guide/writing-suites/scheduling.rst @@ -585,6 +585,7 @@ written using the ``R1`` notation. For example: [scheduler] UTC mode = True + allow implicit tasks = True [scheduling] initial cycle point = 20130808T00 final cycle point = 20130812T00 @@ -607,6 +608,7 @@ Let's suppose that we add the following section to the suite example above: [scheduler] UTC mode = True + allow implicit tasks = True [scheduling] initial cycle point = 20130808T00 final cycle point = 20130812T00 @@ -637,6 +639,7 @@ For example, we can write our suite like so, to produce the graph as shown: [scheduler] UTC mode = True + allow implicit tasks = True [scheduling] initial cycle point = 20130808T00 final cycle point = 20130812T00 @@ -692,6 +695,7 @@ the initial cycle point) and then repeat every ``PT6H`` (6 hours): [scheduler] UTC mode = True + allow implicit tasks = True [scheduling] initial cycle point = 20130808T00 final cycle point = 20130808T18 @@ -726,6 +730,7 @@ used as follows: [scheduler] UTC mode = True + allow implicit tasks = True [scheduling] initial cycle point = 20100101T03 [[graph]] @@ -1243,6 +1248,8 @@ failed): Efficient Inter-Family Triggering ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. TODO: Is this section still true post-SoD? + While Cylc allows writing :term:`dependencies ` between two :term:`families ` it is important to consider the number of dependencies this will generate. In the following example, each member of @@ -1262,9 +1269,9 @@ This can result in high memory use as the number of members of these families grows, potentially rendering the suite impractical for running on some systems. You can greatly reduce the number of dependencies generated in these situations -by putting dummy tasks in the graphing to represent the state of the family you +by putting blank tasks in the graphing to represent the state of the family you want to trigger off. For example, if ``FAM2`` should trigger off any -member of ``FAM1`` succeeding you can create a dummy task +member of ``FAM1`` succeeding you can create a blank task ``FAM1_succeed_any_marker`` and place a dependency on it as follows: .. code-block:: cylc @@ -1544,6 +1551,7 @@ workflow is skipped, if it is more than one day behind the wall-clock: .. code-block:: cylc [scheduler] + allow implicit tasks = True cycle point format = %Y-%m-%dT%H [scheduling] initial cycle point = 2015-08-15T00