Python API

[hjoliver]

We could easily provide a Python API for defining suites (and also reformulate our current suite.rc file parser a little to plug into it). For normal use, I can't see this being the recommended way of writing suites, compared with the current paradigm of config file + Jinja2 support, but maybe there's a case for defining workflows in Python programs as well. (And a few people seem to view it as a deficiency that we don't currently have this). This could even extend to skipping the graph string as a way to define dependencies.

[benfitzpatrick]

We've talked about this informally in Exeter before (or is there a proper issue?). There are suites that justifiably push the configuration file interface to the limits. My memory is that we were a bit divided about a Python API, but I think we thought that if someone else developed it, we wouldn't mind! The others may chip in.

This could even extend to skipping the graph string as a way to define dependencies.

I quite like the idea of also being able to define file-based dependencies for tasks, since that is what our dependencies nearly always are based around. Once you know what files a task needs, you can wrap it all up and push it to a platform with a different filesystem. Obviously this doesn't lend itself to the full dependency syntax power (no OR, no failure triggers, no suicide triggers).

[arjclark]

... I think we thought that if someone else developed it, we wouldn't mind!

That is pretty much my recollection too.

[matthewrmshin]

This does not have to be complicated. All we really need is an easy and documented way for users to generate the internal data structure of a parsed suite.rc in Python.

[matthewrmshin]

Just a bit more thoughts on this one. We can have decent APIs and/or documented ways to do these in Python programs:

• Populate a data structure that represents a suite configuration.
  • (Need something better than our current SuiteConfig class!)
  • (At suite design time and at run time.)
  • A programmatic way to work with a piece of dependency graph.
  • Introspection interface.
  • (See also Better suite validation. #1136, Add, subtract, or override semantics for repeated config items #1363, graph parsing: space removal bug #2534.)
• Call commands (that are currently provided via CLI and/or GUI).
  • Suite server client commands.
  • Non-client commands.
    • E.g. validate a suite or start up a suite server program.
  • Suite API - On-The-Fly #2123, GUI should access suite items only via the API #2466.
• Specify pieces of logic (that are currently specified as shell logic by users).
  • Runtime task environment and scripts.
  • Suite and task event handlers.
  • Task job host selection.
  • DRMAA? #2205, Runtime configuration static/dynamic variants #2415, Event handling functions (for central reporting). #2442.
• Query the suite run database #2483.

[hjoliver]

Comment from Matt on Riot.im:

http://regent-lang.org/
This is work flow at the language level. We can probably learn something from this when we design the Python API.

[hjoliver]

Example from @oliver-sanders of multiple distinct graph sections:

    with cylc.graph() as start_up:
        make_something >> install_cold(platform)

    with cylc.graph() as workflow:
        foo >> bar >> baz
        bar & qux >> pub

    with cylc.graph() as wrap_up:
        post_process >> tidy

    cylc.run(
        start_up >> workflow.every('P1D') >> wrap_up
    )

[oliver-sanders]

(example taken from the wiki)

[hjoliver]

(example taken from the wiki)

(Ah, brilliant, forgot where that write-up was 👍 )