Changes to support executing EPP configurations

[cpelley]

• Fixed dagrunner logger (sometimes there was duplicate entries).
• Remote data polling plugin separating glob patterns and grouping non-glob patterns which fixes an issue where 1 file found would obscure another missing file.
• Rationalised the plugin executer call args UI (users no longer need to configure empty dictionaries).

More detailed explanation of changes

1. Running of scheduler now managed outside dagrunner execution (through dagrunner-logger script).

   • More flexible to manage the scheduler outside the execution of the configuration by dagrunner.
   • I don't think the ThreadingTCPServer socket server was ever intended to be run within its own thread of the same python instance that the client attaches to the socket. The ThreadingTCPServer creates a new thread for each client connection.
   • Minimal downside to managing the server independently where we launch the logging server and kill it after we are done. Basic illustration to how this can be done:

     dagrunner-logger --sqlite-filepath ... &
     pid1=$!
     dagrunner-execute-graph graph_config.graphs.gen_networkx ...
     kill $pid1
     

2. Overhaul remote data polling plugin (final time hopefully).

   • Bugfix: Ensuring that globular pattern matching is differentiated with matches against non-globular matches. Not differentiating between the two could cause the data polling plugin to return True when not ALL files were actually found.
   • Grouping filepath polling into groups: by host and glob/non-glob. This ensures that:
     1. Minimise ssh calls (ssh calls are comparatively slow).
     2. Ensuring that any glob calls are independently verified as expanding to a result. Where globular patterns were included in the same call as non-globular patterns, we wouldn't know whether each pattern was successfully expanding based on filepaths found (hence the bug mentioned above).

3. Refactored logic around plugin_executor handling of the call parameter. This makes it a lot clearer how keyword arguments are passed along to the underlying 'plugin'. This change also means that an empty dictionary being defined to represent keyword arguments is no longer necessary (i.e. a technical implementation detail imposed on end users).
   Illustration of what this means (where 'kwargs' represents a dictionary of keyword arguments to pass):

   For 'plugin' callables (functions etc.):

   no kwargs:

   call={callable_obj}
   res = callable_obj()

   kwargs passed to callable_obj:

   call={callable_obj, kwargs}
   res = callable_obj(**kwargs)

   For callables, we check that we are given only 0 or 1 item under 'call' in addition to the callable_obj ref.

   ---

   For plugin classes:

   no kwargs:

   call={callable_obj}
   res = callable_obj()()

   kwargs passed for initialisation of callable_obj:

   call={callable_obj, kwargs}
   res = callable_obj(**kwargs)()

   kwargs passed to plugin object callable:

   call={callable_obj, {}, kwargs}
   res = callable_obj()(**kwargs)

   kwargs passed for initialisation and kwargs2 passed for object callable:

   call={callable_obj, kwargs, kwargs2}
   res = callable_obj(**kwargs2)(**kwargs2)

   For classes, we check that we are given only 0, 1 or 2 items under 'call' in addition to the callable_obj ref.

Issues

• https://github.com/MetOffice/improver_suite/issues/2253
• https://github.com/MetOffice/epp_workflows/pull/55