Node (Network) Coordinated Subscriptions (part 1/3) - Centralise REP and PUB into single runtime server

[dwsutherland]

This change:

• partially addresses graphql: subscriptions at the scheduler #3731
• partially addresses zmq: implement REQ-REP + PUB-SUB comms #3329
• closes Calling publisher coroutine from another thread #3564
• closes Pytest Unhandled Thread Exception Warning #4706

This PR completes step one of the following changes needed for GraphQL subscriptions at the workflow:

1. Centralise REP and PUB into single runtime server.
2. Setup coordination scheme for REQ of PUB topics to SUB to.
3. Implement a GraphQL-WS style handler for GraphQL subscriptions.

In order to coordinate between PUB and REP sockets it makes sense to revive the single runtime server concept to initialize both and manage requests (REQ) for publishing (PUB) topics to subscribe (SUB) to.

• Step 1 - Can now publish in response to a REQ (in addition to REP).

This also results in less network code in the scheduler.

At present we have each network object (publisher, replier/server) instantiated by the scheduler in the main thread, then their startup/socket-bind happening in a separate thread:

This PR reduces the number of threads, with the publisher, subscriber coordinated in the same thread by the parent server object (instantiated by the scheduler still):

Published articles/data/deltas are now put into a (threadsafe) queue, instead of called directly by the main thread...

Since the threading is setup by the scheduler now, the client no longer needs to import the threading library.

Running the complex workflow takes about the same time as the current master:

        Command being timed: "cylc play complex --color=never -n"
        User time (seconds): 515.45
        System time (seconds): 36.81
        Percent of CPU this job got: 36%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 25:18.28
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 117568
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 20396247
        Voluntary context switches: 182301
        Involuntary context switches: 172133
        Swaps: 0
        File system inputs: 8
        File system outputs: 3725008
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.py and
  conda-environment.yml.
• Appropriate tests are included (unit and/or functional).
• Already covered by existing tests.
• No change log entry required (invisible to users).
• No documentation update required.

[MetRonnie]

Thanks

[oliver-sanders]

Looks good, need to go through this one very thoroughly and do some hands-on-testing, so far so good.

[hjoliver]

This also results in less network code in the scheduler.

Well that's nice 😁

[oliver-sanders]

Works great, cheers.

I've poked and prodded this as best I'm able, works great.

[oliver-sanders]

Suggested change

	# TODO: If requested, coordinate publishing response/stream.
	# TODO: If requested, coordinate publishing response/stream.
	# https://github.com/cylc/cylc-flow/issues/3329