Rose user guide rose tutorial #2170

oliver-sanders · 2018-03-28T11:06:02Z

~~Built on #2166, hold review until merged (will resolve conflicts after this merge).~~

Rose tutorial for the new documentation:

Main objectives for this content:

Explain rose configurations [in a generic sense] (purpose, format).
Convey the relationship between rose applications, suites and rose configurations.
Clarify the purpose of rose applications (namely encapsulation).
Clarify the purpose(s) of rose metadata.
Clarify the relationship between rose suite configurations and cylc suites (cylc suites with rose configurations).
Introduce and reinforce rose terminology.
Introduce the tools for evaluating rose configurations
- rose metadata-check
- ~~rose config~~
Introduce the tools for running rose configurations
- rose app-run
- rose suite-run
- rose task-run

To do in a future PR:

rose config?
rose config edit section?
rosie go section?
rose stem section

~~Ready for preliminary review (structure, approaches, etc).~~

oliver-sanders · 2018-04-09T17:56:45Z

This is now more or less finished (minus spit and polish) - the Rosie section could do with a critical eye and possibly some contributions.

oliver-sanders · 2018-04-13T08:59:57Z

Conflicts resolved. Ignore codacy issues regarding the use of !important in css files, it's not a nice thing to do but is required to reliably override theme values.

oliver-sanders · 2018-04-13T11:58:09Z

Remaining test battery failure is pycode style which will be fixed when rebased on-top of #2172.

matthewrmshin

Some quick initial comments. I'll give the change another eyeball in the afternoon.

matthewrmshin · 2018-04-23T10:55:09Z

sphinx/hyperlinks.rst

 .. _FCM: https://metomi.github.io/fcm/doc/
+.. _FCM User Guide: http://metomi.github.io/fcm/doc/user_guide/
+.. _Github: https://github.com/metomi/rose


Not the best label? Perhaps Rose at Github?

matthewrmshin · 2018-04-23T11:00:41Z

sphinx/tutorial/rose/furthertopics/polling.rst

@@ -26,7 +30,7 @@ file that looks like this:
           graph = """compose_letter => send_letter
                      bob => read_letter"""

-This sets up a simple suite which consists of the following:
+This is simple suite which consists of the following:


This is a simple ...

matthewrmshin · 2018-04-23T11:07:25Z

sphinx/tutorial/rose/rosie.rst

+      Switch back to the trunk then merge your change branch into the trunk::
+
+         fcm sw trunk
+         fcm merge change trunk


fcm merge change should be sufficient.

matthewrmshin · 2018-04-23T11:10:34Z

sphinx/tutorial/rose/summary.rst

+   See the :ref:`date-time tutorial <rose-tutorial-datetime-manipulation>`
+   for more information.
+
+:ref:`command-rose-host-select`


I think we should get rid of this one.

After cylc/cylc-flow#1885 and cylc/cylc-flow#2199?

sadielbartholomew

About to open a PR with a number of changes picked up from reading through the built tutorial pages. For now, note:

Everything builds fine on my environment.
Functionally, tutorial practicals are all great, except the second ('Rose Bush') practical on the Rose Suite Configurations page, where something has gone amiss. Will add in a fix to my PR.
PR: otherwise small changes, some trivial but many requested because I found certain wording/explanations unclear. Ask if you are unsure why I have asked to change something. A few comments that can't go in a PR.
Just need to look over small changes to other files now, will complete reviewing tomorrow.

sadielbartholomew · 2018-04-23T09:53:34Z

sphinx/tutorial/rose/rosie.rst

+In Rosie suites the :term:`suite directory` is added to `version control`_
+using `FCM`_.
+
+FCM is a `subversion <SVN>`_ (SVN) wrapper which provides a standard working


The SVN link is broken - it doesn't seem to be defined either in this file or in hyperlinks.rst. As discussed will leave to you to choose an appropriate link destination.

sadielbartholomew · 2018-04-23T10:43:37Z

sphinx/tutorial/rose/suites.rst

+#. The Cylc suite is run.
+#. The Cylc gui is launched.
+
+.. digraph:: Example


Great diagram - really clear & useful!

sadielbartholomew · 2018-04-23T11:37:20Z

sphinx/tutorial/rose/summary.rst

+
+The relationship between them is as follows:
+
+.. graph:: Example


Another really useful diagram - eventually I think it could be handy to collect such diagrams (as well as where they currently are) either on the Cheat Sheet or on a similar 'quick reference' type page.

Good point, for the moment I'll add diagrams / link the relevant sections through from the glossary.

oliver-sanders · 2018-04-24T11:33:16Z

@matthewrmshin Note I've added a few new glossary items in 307639c that could probably do with a quick scan.

lhuggett · 2018-04-24T13:41:14Z

Section: rose applications
The bit about creating a new directory called forecast. The directory
doesn't seem to be used in this section.
The rose-app.conf file: where is this supposed to be created? I put it in the
current directory (~/rose-tutorial/application-tutorial) which seems to work.
Viewing of map: works in firefox, but not in knoqueror.

Section: metadata
Having clicked on the link to the full list of metadata items,
it is quite confusing at this stage of the training.
Typing all the '[env=WHATEVER]' and 'description=some comment or other' stuff into
the rose-meta.conf file to set it up is a bit tedious, even when copying and pasting
most of it from the rose-app.conf file.

Section: rose suite configurations
I would suggest changing 'When we run rose applications in cylc suites we use the
rose task-run command rather than the rose app-run command.' to
When we run rose applications from within cylc suites we use the
rose task-run command rather than the rose app-run command.'
To avoid possible ambiguity.

The bit where it says 'TODO - remove triple quotes in cylc tutorial'
is that intended for the tuttee or the developer?

The changes to the forecast task - it doesn't say where to find this.
I have assumed in the suite.rc file.

With the setting of initial and final cycle points, it doesn't seem
to work with copying and pasting in the results from the cylc training.

The 'post_process_exeter' task currently fails due to not being able to
find file "+PT01H00M.csv"

Cheat sheet:
what is the difference between --path and -C?

matthewrmshin · 2018-04-24T14:53:54Z

sphinx/glossary.rst

+      * :term:`warm start`
+
+   warm start
+      In a :term:`datetime cycling` suite


Not necessarily datetime.

matthewrmshin · 2018-04-24T14:57:02Z

sphinx/glossary.rst

+      configuration.
+
+      Reloading changes to the ``[runtime]`` section is safe, reloading changes
+      to the ``[scheduling]`` section has caveats, see the `Cylc User Guide`_.


Note: [runtime] changes that affect the family inheritance hierarchy may affect scheduling as well.

Oh, good point.

See new commit.

Review feedback amendments

oliver-sanders · 2018-04-24T14:59:10Z

Section: rose applications

The bit about creating a new directory called forecast. The directory
doesn't seem to be used in this section.

This should now be fixed.

The rose-app.conf file: where is this supposed to be created? I put it in the
current directory (~/rose-tutorial/application-tutorial) which seems to work.

The tutorial now gets you to cd application-tutorial then create a rose-app.conf file so I hope this is now implicit, we could add a touch rose-app.conf step if it's still unclear.

Viewing of map: works in firefox, but not in knoqueror.

Note sure why this is happening, I'll take a look at this one later.

Section: metadata

Having clicked on the link to the full list of metadata items, it is quite confusing at this stage of the training.

Agreed, the API pages are much more technical. I'm not entirely sure what to do about this really. We could remove the link from the tutorial as it's not really necessary or update the text to read something like.

For future reference a full list of metadata items can be found ...

Typing all the '[env=WHATEVER]' and 'description=some comment or other' stuff into
the rose-meta.conf file to set it up is a bit tedious, even when copying and pasting
most of it from the rose-app.conf file.

I was a bit worried about this. For the moment I'll put in a TODO for future work. Suggestions welcome.

Section: rose suite configurations

I would suggest changing 'When we run rose applications in cylc suites we use the
rose task-run command rather than the rose app-run command.' to
When we run rose applications from within cylc suites we use the
rose task-run command rather than the rose app-run command.'
To avoid possible ambiguity.

Good point, done.

The bit where it says 'TODO - remove triple quotes in cylc tutorial'
is that intended for the tuttee or the developer?

I've now moved this into a comment.

The changes to the forecast task - it doesn't say where to find this.
I have assumed in the suite.rc file.

I've re-phrased this to note that the task is in the suite.rc file.

With the setting of initial and final cycle points, it doesn't seem
to work with copying and pasting in the results from the cylc training.

Fixed.

The 'post_process_exeter' task currently fails due to not being able to
find file "+PT01H00M.csv"

I'll need to see this error in context to advise, possibly a typo or the likes or a misleading instruction.

Cheat sheet:

what is the difference between --path and -C?

One is correct and the other is a strange figment of my immigration. Well spotted, not quite sure where that came from, fixed.

sadielbartholomew · 2018-04-24T15:07:20Z

sphinx/tutorial/rose/suites.rst

+            WIND_FILE_TEMPLATE=test-data/wind_{cycle}_{xy}.csv
+            RAINFALL_FILE=test-data/rainfall.csv
+            MAP_FILE=map.html
+            CYLC_TASK_CYCLE_POINT=20171101T0000Z


Noting for future improvements brief conversation in person regarding setting this variable to 20171101T0000Z being a potential cause of confusion for readers, given it is a random date-time tied to the test data.

I've inserted a better explanation of the test data in the application tutorial (a42675a).

matthewrmshin

Some minor comments. Looks good otherwise.

sadielbartholomew

Minor comments for commits made after my first review (a01e3b6 onwards, inclusive). Previous feedback addressed. Now I just need to go over all existing sphinx files with small changes as opposed to being completely new (all files under etc/ & bin/ i.e. non-doc files look fine).

sadielbartholomew · 2018-04-25T11:45:45Z

sphinx/installation.rst

@@ -6,8 +6,8 @@ Installation

 .. _archives: https://github.com/metomi/rose/tags

-The source code for Rose is available via `GitHub`_, code releases
-are available in ``zip`` and ``tar.gz`` `archives`_.
+The source code for Rose is available via `GitHub <Rose GitHub>`_, code


This link is broken. I seem to recall you cannot provide custom text for a hyperlink created from another file, so will need to rename directly in sphinx/hyperlinks.rst.

The link to SVN in sphinx/tutorial/rose/rosie.rst is also broken for the same reason.

Doh, I keep walking into that one.

sadielbartholomew · 2018-04-25T11:53:19Z

sphinx/tutorial/rose/applications.rst

@@ -181,9 +181,9 @@ can be used to provide the path to the Rose application:
      .. code-block:: rose

         # The weighting to give to the wind file from each WIND_CYCLE
-         # (should add up to 1).
+         # (comma separated list, values should add up to 1).


Much clearer comments, but for consistency, this change should really be also made in the config files that are provided for the user to use in other practicals (i.e. etc/tutorial/metadata-tutorial/rose-app.conf & etc/tutorial/rose-suite-tutorial/app/forecast/meta/rose-meta.conf).

sadielbartholomew · 2018-04-25T11:55:05Z

sphinx/tutorial/rose/applications.rst

         WEIGHTING=1
-         # List of cycle points to get wind data from.
+         # Comma separated list of cycle points to get wind data from.


As comment above, & also in this case etc/tutorial/runtime-tutorial/runtime needs adapting accordingly.

sadielbartholomew · 2018-04-25T12:48:22Z

sphinx/glossary.rst

+
+      .. _daemon: https://en.wikipedia.org/wiki/Daemon_(computing)
+
+      By default a suite server program is a `daemon`_ meaning that is runs in


Typo 'is it'

sadielbartholomew · 2018-04-25T12:49:46Z

sphinx/glossary.rst

+      When we say that a :term:`suite` is "running" we mean that the cylc
+      suite server program is running.
+
+      The suite server program is responsible for running the suite, it submits


'... the suite. It submits ...' is better grammar & reads better.

sadielbartholomew · 2018-04-25T12:58:05Z

sphinx/glossary.rst

+
+   reload
+      Any changes made to the ``suite.rc`` file whilst the suite is running
+      will not have any effect untill the suite is:


Typo: 'untill'. Also, I suggest appending 'either' onto the end of the sentence here, to be fully clear that these things are not all necessary!

sadielbartholomew · 2018-04-25T13:00:32Z

sphinx/glossary.rst

+      * :term:`Shutdown` and :term:`restarted <restart>`
+      * "Reloaded"
+
+      Reloading does not require the suite to be :term:`shutdown`. When a suite


:term:`shut down <shutdown>` :)

sadielbartholomew · 2018-04-25T13:03:35Z

sphinx/glossary.rst

+
+      Reloading does not require the suite to be :term:`shutdown`. When a suite
+      is reloaded any currently "active" :term:`tasks <task>` will continue with
+      their "pre-reload" configuration, new tasks will use the new


'configuration, whilst new tasks...' perhaps?

sadielbartholomew · 2018-04-25T13:10:14Z

sphinx/tutorial/rose/applications.rst

@@ -212,19 +222,13 @@ application:

      .. code-block:: rose

-         # The cycle point for the test data.
+         # The date when the test data was gathered.


Much clearer! There should be no confusion about this now.

sadielbartholomew · 2018-04-25T13:12:23Z

sphinx/glossary.rst

-      to the ``[scheduling]`` section has caveats, see the `Cylc User Guide`_.
+      Reloading changes is safe providing they don't affect the
+      :term:`suite's <suite>` :term:`graph`. Changes to the graph have certain
+      caveats attached, see the `Cylc User Guide`_.


attached. See the ... [for more information].' reads better.

sadielbartholomew

Review complete. I have some final comments, all minor. My last set of feedback has been addressed suitably.

Overall the new Rose Tutorial section is extremely good & should be very instructive to anyone who reads it!

sadielbartholomew · 2018-04-25T14:32:42Z

sphinx/api/configuration/site-and-user.rst

@@ -14,10 +14,18 @@ site configuration at ``$ROSE_HOME/etc/rose.conf`` where
 search for the user configuration at ``$HOME/.metomi/rose.conf`` where
 ``$HOME`` is the home directory of the current user.

-You can also override many internal constants of the ``rose config edit`` and
+.. note::


This note is no longer applicable, the rose.conf settings section having been added in now.

sadielbartholomew · 2018-04-25T14:34:25Z

sphinx/tutorial/cylc/scheduling/graphing.rst

         foo -> bar -> baz -> qux
         pub -> bar -> wop

-   #. **Use** ``cylc graph`` **to visualise the workflow.**
+   #. **Use** ``Cylc graph`` **to visualise the workflow.**


This is a command, so 'cylc' still needs to be lower-case here.

sadielbartholomew · 2018-04-25T14:34:38Z

sphinx/tutorial/cylc/scheduling/integer-cycling.rst

@@ -406,15 +401,15 @@ starting 5 cycles *after* the initial cycle point.

   #. **Visualise the suite.**

-      Try visualising the suite using ``cylc graph``.
+      Try visualising the suite using ``Cylc graph``.


This is a command, so 'cylc' still needs to be lower-case here.

sadielbartholomew · 2018-04-25T14:37:46Z

sphinx/api/configuration/rose-configuration-format.rst

@@ -13,9 +13,9 @@ following:
 * (optionally) files containing data that cannot easily be represented by the
  INI format.

-We have added the following conventions into the rose configuraiton format:
+We have added the following conventions into the Rose configuraiton format:


I've noticed a typo ('configuraiton') - it's outside of the PR changes but we should get it fixed now.

oliver-sanders self-assigned this Mar 28, 2018

matthewrmshin added this to the soon milestone Apr 9, 2018

oliver-sanders force-pushed the rose-user-guide-rose-tutorial branch 3 times, most recently from ab9526a to 0667754 Compare April 13, 2018 08:57

oliver-sanders requested review from matthewrmshin and sadielbartholomew April 13, 2018 08:58

oliver-sanders added 20 commits April 13, 2018 17:02

Remove cylc logic from tutorial scripts.

4471984

rose introduction.

0f78267

rose applications.

09fcb60

rose metadata.

b8658df

rose/cylc cheat sheet.

00606e2

tutorial suite, pass cycle point offsets.

887f7ed

rose suite configurations.

16a8f07

rose bush - temporary tutorial addition

a0a89a9

add directive for practical extensions

16b7310

rosie

8016d5b

rose summary

2080324

style code-block captions.

4322518

move all tutorials into the rose-tutorial directory.

761215a

erratum - first pass

b89d3ec

pep8.

27668cf

Capitalise all rose instances.

78df244

Capitalise all cylc instances.

61feafe

Add tutorial file tests.

1ca206d

Correct tutorial directory names.

83790e8

Set graphviz defaults.

93581cb

oliver-sanders added this to the next-release milestone Apr 23, 2018

matthewrmshin reviewed Apr 23, 2018

View reviewed changes

oliver-sanders added 2 commits April 23, 2018 14:21

feedback

feb3ec8

remove rogue control character

2b223ca

oliver-sanders force-pushed the rose-user-guide-rose-tutorial branch from f03d1ae to 2b223ca Compare April 23, 2018 13:23

sadielbartholomew reviewed Apr 23, 2018

View reviewed changes

sadielbartholomew mentioned this pull request Apr 23, 2018

Review feedback amendments oliver-sanders/rose#19

Merged

oliver-sanders and others added 3 commits April 24, 2018 09:48

erratum

18ac5f5

Review feedback minor changes: fix typos, improve clarity etc

400ee9c

feedback 2

307639c

sadielbartholomew added 2 commits April 24, 2018 14:51

Resolve merge conflicts

ccac7f1

Respond to feedback

f458a4f

matthewrmshin reviewed Apr 24, 2018

View reviewed changes

Merge pull request #19 from sadielbartholomew/copy-rug-rose-tut

24bc586

Review feedback amendments

feedback 3

cf3439a

sadielbartholomew reviewed Apr 24, 2018

View reviewed changes

matthewrmshin approved these changes Apr 24, 2018

View reviewed changes

oliver-sanders added 2 commits April 24, 2018 16:25

explain the forecast app test data

a42675a

feedback 4

5fc9e36

This was referenced Apr 24, 2018

rose make-docs: add --strict option #2176

Merged

Rose user guide rose stem #2177

Merged

sadielbartholomew reviewed Apr 25, 2018

View reviewed changes

feedback 5

d4e9baa

sadielbartholomew approved these changes Apr 25, 2018

View reviewed changes

feedback 6

332c976

oliver-sanders merged commit 9de891f into metomi:rose-user-guide-overhaul Apr 25, 2018

oliver-sanders deleted the rose-user-guide-rose-tutorial branch May 18, 2018 09:20


		The relationship between them is as follows:

		.. graph:: Example


		.. _daemon: https://en.wikipedia.org/wiki/Daemon_(computing)

		By default a suite server program is a `daemon`_ meaning that is runs in

Rose user guide rose tutorial #2170

Rose user guide rose tutorial #2170

Conversation

oliver-sanders commented Mar 28, 2018 • edited Loading

oliver-sanders commented Apr 9, 2018 • edited Loading

oliver-sanders commented Apr 13, 2018

oliver-sanders commented Apr 13, 2018

matthewrmshin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadielbartholomew left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oliver-sanders commented Apr 24, 2018

lhuggett commented Apr 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oliver-sanders commented Apr 24, 2018

Section: rose applications

Section: metadata

Section: rose suite configurations

Cheat sheet:

Choose a reason for hiding this comment

oliver-sanders Apr 24, 2018 • edited Loading

Choose a reason for hiding this comment

matthewrmshin left a comment

Choose a reason for hiding this comment

sadielbartholomew left a comment • edited Loading

Choose a reason for hiding this comment

sadielbartholomew Apr 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadielbartholomew left a comment

Choose a reason for hiding this comment

sadielbartholomew Apr 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oliver-sanders commented Mar 28, 2018 •

edited

Loading

oliver-sanders commented Apr 9, 2018 •

edited

Loading

sadielbartholomew left a comment •

edited

Loading

oliver-sanders Apr 24, 2018 •

edited

Loading

sadielbartholomew left a comment •

edited

Loading

sadielbartholomew Apr 25, 2018 •

edited

Loading

sadielbartholomew Apr 25, 2018 •

edited

Loading