reduce the impact of FS scans

[oliver-sanders]

At present we are regularly performing file system (FS) scans to detect new suites. This functionality is required to pick up on suites started from the CLI.

These FS scans are quite regular and can be slow as they involve walking the directory tree.

• Is there a more light-weight alternative to a full FS scan?
  • To pick up new suites we only need to listen for directory creation.
  • However, due to hierarchical suites these directories could be created in different places.
  • Note a solution cannot be Linux-bound (though adding something like fswatch to the dependency base shouldn't be an issue)
• Can we do some kind of incremental scan to avoid re-walking trodden ground?
  • Perhaps an arrangement of "quick scans" and "full scans".

Note: For suites started via the UI we can do things in a more responsive manner (cylc/cylc-flow#3451).

[oliver-sanders]

Is there a more light-weight alternative to a full FS scan?

Yes, however, sadly we cannot depend on filesystem events as we need to support network filesystems.

I did quickly build a filesystem event driven scan, it's pretty simple, but alas, it's not an option for us so the best bet we have for now is just to make the scan more intelligent.

Can we do some kind of incremental scan to avoid re-walking trodden ground?

Yes, and cylc/cylc-flow#3724 opens up an interface to doing so, just use a suite name filter which skips flows you already know about.