Suggestion for structure and management of NorESM system testing baselines

[TomasTorsvik]

@mvertens , @gold2718
I thought it could be useful to include some guidelines for the NorESM system testing baselines directory in
/cluster/shared/noresm/noresm_baselines, and have therefore included the following README file in that directory. Please comment/change if you think something should be commented on/changed.

NorESM system testing baselines
===============================

This directory contains baselines for comparing output from standard system test runs.


Directory structure
-------------------

The root directory contains baselines created from NorESM code releases. Each baseline
is named according to the corresponding release name, and contains baselines for both
fully coupled and component-specific tests.

noresm-develop directory contains baselines for development versions of NorESM.
It is recommended to follow a naming convention that makes it possible to
identify the basic model configuration from the case name, e.g.:
- <noresm_tag> : Prefered, if testing a tag version (not code release) of NorESM.
- <noresm-version>_PR<###> : Identify with a pull request number from GitHub.
- dev-<some_label>_PR<###> : When testing cross-component dependencies, which does not
                             correspond to a specific NorESM version.

<component>-develop directories contain baselines for development versions of model
components. It is recommended to follow a naming convention that makes it possible to
identify the basic model configuration from the case name, e.g.:
- <noresm-version>_<component_tag> : Prefered, if testing a tag version of a component.
- <noresm-version>_PR<###> : Identify with a pull request number from GitHub.
- dev-<some_label>_PR<###> : When testing cross-component dependencies, which does not
                             correspond to a specific NorESM version.


Data sharing
------------

Baseline simulations should have open read access and use the "noresm" group setting
with write access.

chgrp -R noresm <casename>
chmod g+w -R <casename>

So that a long list output will be 

drwxrwxr-x.  ##  <username>  noresm  ####  <date>  <casename>


Cleaning up old baselines
-------------------------

The baseline creators are responsible for removing old baseline runs once they are no
longer needed/relevant. For instance, development versions can ususally be removed once
a release version is available.

[mvertens]

@TomasTorsvik - thanks so much for putting this together. This is really helpful moving forwards.
Where it gets confusing is not just the component tag - but what other changes to the sandbox were made to duplicate this testing. As an example - what if the BLOM tag requires changes to CMEPS and CAM (say for new forcing such as dms exchange). How should we describe the baseline given the other changes to the model components.

[gold2718]

I think this could get complex in a hurry so maybe the easiest is to just add the relevant PR number at the end (e.g., <noresm-version>_<component_tag>_PR<###>).

[TomasTorsvik]

Good points. Maybe in the case there isn't a component tag, the PR<###> should be referenced instead.

The situation with multiple component changes is more complicated. Maybe this should not go in the <component>-develop directory, but instead into noresm-develop directory?
Alternatively, can these "cross-component" dependent tests be labeled in some way that signals that it's not a straight forward component test (e.g. dev-<some-relevant-development-label>_PR<###>)?

[gold2718]

@TomasTorsvik, thanks for starting this, I think it will really help keep that area organized.

One thought about write access. We have had issues in the past where disk space became tight and the author was not immediately available to remove some older directories (it's really easy to forget!). For instance, this Summer, I was asked to remove some old baseline directories from an internal NCAR test machine (that were written well before I left NCAR nearly 2 years ago). Luckily, I was still able to log in and remove them.

Is there a group we all share, such as noresm? Then, the recommendation could be to set the group to that and make it also group writable.

[TomasTorsvik]

Agree. It would be good if all output is shared writable with the noresm group setting.