Agree content of cylc-flow.rc

[wxtim]

Reference #44

Aim

Before carrying out cylc-flow ticket #3260 get as much agreement as possible on the content of the new cylc-flow.rc.

Background

Before to #2199 (redefining how Cylc submits jobs to compute resources/clusters) we should consider the design of the .rc settings files.

The most logical way to support installation of suites on suite clusters and
job hosts is to migrate our host-based configuration logic to a cluster-based
configuration logic. Some points to consider:

• Deprecate global.rc [hosts] section in favour of a [job platforms]
  section. The new section should allow us to configure:
  • Login hosts.
  • Batch system or workload manager, and related settings.
  • Directory locations for various usages by suites.
  • Communication protocols:
    • Forward (install, submit, poll, kill, view log, log retrieval, etc).
    • Backward (message and manipulation commands).
• Deprecate suite.rc relevant settings under [runtime][__MANY__], but have
  a top level [job platforms] section that simply mirrors that of global.rc to allow individual suites to override the site/user settings.
• Users will configure tasks to run on clusters instead of hosts/batch systems.
• If relevant, improve alignment with DRMAA Open Grid Forum API?

So, for example, a suite flow.rc might look like this:

[cylc]
    ...

[scheduling]
    ...

[job platforms]
    [[spice]]
        [[[batch system]]]
            class = slurm
            before command = sshare -u %(user_id)           # Blue Sky thinking
            after command = sacct -j %(jobid)s --format=""
    [[xcs]]
        login hosts = xcslr0, xcslr1
        retrieve job logs = True
        [[[batch system]]]
            class = pbs
        [[[default directives]]]
            --some-directive="directive here!"
    [[desktop]]
        login hosts = vld392

[runtime]
    [[Alice]]
        ...

    [[Bob]]
        ...
        [[[job platform]]]
            platform = desktop

    [[Charlie]]
        ...
        [[[job platform]]]
            platform = spice
        [[[directives]]]
            --mem=5
            --ntasks=1

    [[Dai]]
        ...
        [[[job platform]]]
            platform = cray

Ideally the suite.rc and the global.rc should be replaced by a single configuration, probably called cylc-flow.rc. It is envisaged that some of it's settings are more likely to be set in a global context, and some in a suite context, but the same settings should be available in both cases.

[dpmatthews]

Superseded by cylc/cylc-flow#3422