reload the global configuration during a workflow run

[oliver-sanders]

[edit] This issue was re-written to broaden its scope from a main-loop plugin to also cover workflow config reload.

At present the global config is static. It is read in once when the scheduler starts.

There may be use cases for "reloading" the global configuration (not to be mistaken for the workflow config) while the workflow is running, e.g:

• Dynamic run hosts to support elastically scaling Cylc server clusters.
• Dynamic platforms to allow admins to update platform configs on-the-fly.
• Discourse post.

There is one part of Cylc which is already reloading the global config, the auto-restart main-loop plugin. This plugin "re"loads the global config periodically in order to check the [run hosts]available and [run hosts]condemned settings. These settings are used to control the auto-restart functionality. However, this "refreshing" is constrained to the auto-restart plugin, it does not affect the cached instance that is used elsewhere throughout the Cylc codebase.

The "refreshing" of the global config could occur:

• Periodically (e.g. with a main loop plugin).
• On demand, with workflow config reload.
• [HO] On demand, separately from workflow config reload
  (workflow config reload can be intensive, and may not be needed along with a global reload)

This should be technically simple to accomplish, however, there is a high potential for caveats / inconsistencies with different parts of the system working with different versions of the config:

• Make sure code isn't caching values from glbl_cfg() calls.
• Clear and rebuild any state which is derived from configs pull in from the global config.
• There are some configs that we will not be able "refresh" in this way, e.g. [symlink dirs] or UTC mode. We need to ensure these configs remain cached.

We will need to define the scope of this feature (i.e. what things can or can not be changed by global config reload), and communicate this somehow. I.e, we should add a note to global config entries to mark them as reloadable or not.

Pull requests welcome!

[hjoliver]

Should this plugin be included in the default set, if so with what interval.

IMO yes it should. Infrequent - 10 min?

[oliver-sanders]

Note: with #5182 platforms are cached so this plugin will need to wipe the cache for changes to get picked up.

[ScottWales]

I'll take a look at this

[hjoliver]

Note, I added a bullet point to the description above:

[HO] On demand, separately from workflow config reload
(workflow config reload can be intensive, and may not be needed along with a global reload)

[ScottWales]

Config items that have a persistent effect are mainly under [scheduler] and would be simplest to handle by requiring restarting the server to change:

• cylc/flow/network/server.py#L192 scheduler/run hosts/ports
• cylc/flow/config.py#L635 scheduler/utc mode
• cylc/flow/loggingutil.py#L164-L170 scheduler/logging
• cylc/flow/scheduler.py#L1270 scheduler
• cylc/flow/subprocpool.py#L191 scheduler/process pool size

Install paths for a platform have a persistent effect on the filesystem, changing them after the platform has been created is not going to have any effect (even if the server is stopped and restarted)

• cylc/flow/pathutil.py#L247 install/symlink dirs

Platforms are attached to task proxies rather than individual jobs. Currently the original platform attached to a task proxy persists through reloads. Changing the platform of a running job could break things, e.g. changing the host of a background task, but other modifications might want to affect existing jobs, e.g. changing the global init-script and retrying a failed job.

• cylc/flow/task_proxy.py#L347

I'm not sure if the global and workflow reloads can be easily separated - if the global reload affects platforms then the task proxies need to be re-created - is that what takes up all the time for a workflow reload? Or is there a separate costly step?

All the other glbl_cfg() references I could find appear to be restricted to inside a function where just changing the global config object is sufficient.

[hjoliver]

I'm not sure if the global and workflow reloads can be easily separated -

If that's the case, I think it's fine to do both at once, at least as a first cut (and maybe for the long run, given your next point).

if the global reload affects platforms then the task proxies need to be re-created - is that what takes up all the time for a workflow reload?

That's a good point. Yes I think recreating all the task proxies is the main sink for reload time. Maybe it doesn't matter too much anymore though. In Cylc 7 there were typical way more task proxies in a large workflow, than in Cylc 8 (which has just the "active window" tasks).