Publish data store updates

[dwsutherland]

These changes partially address #3207
Closes #3014

Dependant PR: cylc/cylc-uiserver#95

Functional/Done:

• Network components added (Publisher/Subscriber components)
• Data components (deltas) for publish.
• Security/Authentication provided by zmq.auth (Workflow service network layer auth & encryption  #3359 ).
• Module name changes.

Utility included (have been using it for testing):
cylc-subscribe and the publish of the protobuf topics: i.e. PbWorkflow data element, which can be subscribed to via the topic workflow:

(flow) sutherlander@cortex-vbox:~$ cylc run --hold noz
            ._.                                                       
            | |                 The Cylc Suite Engine [8.0a1]         
._____._. ._| |_____.           Copyright (C) 2008-2019 NIWA          
| .___| | | | | .___|   & British Crown (Met Office) & Contributors.  
| !___| !_! | | !___.  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
!_____!___. |_!_____!  This program comes with ABSOLUTELY NO WARRANTY.
      .___! |              It is free software, you are welcome to    
      !_____!             redistribute it under certain conditions;   
                        see `COPYING' in the Cylc source distribution. 
                                                                       

*** listening on tcp://cortex-vbox:43078/ ***
*** publishing on tcp://cortex-vbox:43031 ***

To view suite server program contact information:
 $ cylc get-suite-contact noz

Other ways to see if the suite is still running:
 $ cylc scan -n 'noz' cortex-vbox
 $ cylc ping -v --host=cortex-vbox noz
 $ ps -opid,args 13994  # on cortex-vbox

(flow) sutherlander@cortex-vbox:suite$ cylc subscribe noz workflow
Connecting to tcp://cortex-vbox:43031
Received:  workflow
{
    "stamp": "sutherlander|noz@1569677013.108348",
    "id": "sutherlander|noz",
    "name": "noz",
    "status": "held",
    "host": "cortex-vbox",
    "port": 43078,
    "owner": "sutherlander",
    "tasks": [
        "sutherlander|noz|bar",
        "sutherlander|noz|baz",
        "sutherlander|noz|foo",
.
.
.

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Appropriate tests are included (unit and/or functional).
• Appropriate change log entry included (needed?).
• No documentation update required.

[dwsutherland]

Made some significant changes to how data-store updates are managed in the WorkflowService/cylc-flow:

• Changes/Deltas are generated as elements with only updated fields populated.
• Deltas are gathered and applied to the local data-store (so the same application of changes can be used with the UIS)
• Deltas are published as a multi-part messages.

cylc subscribe [OPTIONS] REG [TOPIC] can be used the the following topics:

• edges
• families
• family_proxies
• jobs
• tasks
• task_proxies
• workflow

for example:

(flow) sutherlander@cortex-vbox:cylc-flow$ cylc subscribe noz jobs
Connecting to tcp://cortex-vbox:43063
Received:  jobs
{
    "time": 1570594467.2935598,
    "checksum": "4230119217",
    "deltas": [
        {
            "stamp": "sutherlander|noz|20190101T00|baz|1@1570594467.2907376",
            "id": "sutherlander|noz|20190101T00|baz|1",
            "submitNum": 1,
            "state": "ready",
            "taskProxy": "sutherlander|noz|20190101T00|baz",
            "batchSysName": "background",
            "host": "localhost",
            "jobLogDir": "/home/sutherlander/cylc-run/noz/log/job/20190101T00/baz/01",
            "script": "sleep 15; echo \"$CYLC_TASK_NAME\"",
            "name": "baz",
            "cyclePoint": "20190101T00"
        },
        {
            "stamp": "sutherlander|noz|20190102T00|baa|1@1570594467.292317",
            "id": "sutherlander|noz|20190102T00|baa|1",
            "state": "succeeded",
            "finishedTime": "2019-10-09T04:14:26Z",
            "messages": [
                "succeeded"
            ]
        }
    ]
}
Received:  jobs
{
    "time": 1570594467.8002422,
    "checksum": "1037995818",
    "deltas": [
        {
            "stamp": "sutherlander|noz|20190102T00|bar|1@1570594467.7975347",
            "id": "sutherlander|noz|20190102T00|bar|1",
            "submitNum": 1,
            "state": "ready",
            "taskProxy": "sutherlander|noz|20190102T00|bar",
            "batchSysName": "background",
            "host": "localhost",
            "jobLogDir": "/home/sutherlander/cylc-run/noz/log/job/20190102T00/bar/01",
            "script": "sleep 15; echo \"$CYLC_TASK_NAME\"",
            "name": "bar",
            "cyclePoint": "20190102T00"
        }
    ]
}
Received:  jobs
{
    "time": 1570594468.7934058,
    "checksum": "1102876461",
    "deltas": [
        {
            "stamp": "sutherlander|noz|20190101T00|baz|1@1570594468.7916427",
            "id": "sutherlander|noz|20190101T00|baz|1",
            "state": "submitted",
            "submittedTime": "2019-10-09T04:14:28Z",
            "batchSysJobId": "3656",
            "messages": [
                "submitted"
            ]
        }
    ]
}
.
.
.

More info to come..

[kinow]

Looks promising! 🎉

[kinow]

This is a WIP details to come...

Just checking, is it still WIP?

[dwsutherland]

This is a WIP details to come...

Just checking, is it still WIP?

Just changed it to fully fledged PR..
However, there is still tests to add and there may be more changes if the UIS requires it..

[oliver-sanders]

Tested the full stack, working as expected 🚀!

Thanks @dwsutherland this is a pretty gnarly part of the system, you've managed to get it going nicely.

Testing I did find one oddity which is that with cycling suites, cylc subscribe will list tasks each time a new cycle is added which is superfluous as the task definitions haven't changed. I guess we're getting an erroneous but harmless delta?

[dwsutherland]

Testing I did find one oddity which is that with cycling suites, cylc subscribe will list tasks each time a new cycle is added which is superfluous as the task definitions haven't changed. I guess we're getting an erroneous but harmless delta?

This is probably intentional, because with new edges you get new task proxies, and with new task proxies you get changes to task definition elements (as the relationships have to be set (IDs added to the proxies field)).

[kinow]

Two approvals, but probably good idea to get @hjoliver eyes on this too.

[dwsutherland]

Two approvals, but probably good idea to get @hjoliver eyes on this too.

Got one more commit, API version, module names, change log.

[hjoliver]

Two approvals, but probably good idea to get @hjoliver eyes on this too.

Yeah, I'll prioritize this ASAP

[dwsutherland]

Ok, finishing touches in:

• API version to scan items (for UI Server, and cylc scan to know API version)
• Module names changed (as discussed on Riot), done here to avoid an extra pair of breaking changes and API version increment.
• Change entry added

[hjoliver]

Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- cylc/flow/data_store_mgr.py  24
- cylc/flow/network/publisher.py  3
- cylc/flow/daemonize.py  4
- cylc/flow/network/subscriber.py  5
         

Complexity decreasing per file
==============================
+ cylc/flow/network/client.py  -1
         

See the complete overview on Codacy

[hjoliver]

Alright, this is going in (tested as working, the update/sync methodology seems sound and robust, and no code issues that I've seen). 🎉