Usage of daemon workers, dask config, and tests

[lilyminium]

Daemon workers and dask config

I tried to set up a DaskSLURMBackend and received this error:

distributed.worker - WARNING -  Compute Failed
Function:  _wrapped_function
args:      (<function ProtocolGraph._execute_protocol at 0x7f952bf3f8b0>, 'evaluator_working-data/SimulationLayer/f4683d534fda4bd6bb6d8722fafff554/456cf9ec8e1649c1a15d61042b188733_build_solvated_coordinates', '{"id": "456cf9ec8e1649c1a15d61042b188733|build_solvated_coordinates", "type": "BuildCoordinatesPackmol", "inputs": {".allow_merging": true, ".max_molecules": 2000, ".count_exact_amount": true, ".mass_density": {"value": 0.95, "unit": "g / ml", "@type": "openff.evaluator.unit.Quantity"}, ".box_aspect_ratio": [1.0, 1.0, 1.0], ".substance": {"components": [{"smiles": "CCCCCCCCCC", "role": {"value": "solv", "@type": "openff.evaluator.substances.components.Component.Role"}, "@type": "openff.evaluator.substances.components.Component"}, {"smiles": "CCCCCCCCCC", "role": {"value": "sol", "@type": "openff.evaluator.substances.components.Component.Role"}, "@type": "openff.evaluator.substances.components.Component"}], "amounts": {"CCCCCCCCCC{solv}": [{"value": 1.0, "@type": "openff.evaluator.substances
kwargs:    {'safe_exceptions': True, 'available_resources': <openff.evaluator.backends.backends.QueueWorkerResources object at 0x7f95353d5160>, 'registered_workflow_protocols': ['openff.evaluator.workflow.protocols.ProtocolGroup', 'openff.evaluator.protocols.analysis.AverageObservable', 'openff.evaluator.protocols.analysis.AverageDielectricConstant', 'openff.evaluator.protocols.analysis.AverageFreeEnergies', 'openff.evaluator.protocols.analysis.ComputeDipoleMoments', 'openff.evaluator.protocols.analysis.DecorrelateTrajectory', 'openff.evaluator.protocols.analysis.DecorrelateObservables', 'openff.evaluator.protocols.coordinates.BuildCoordinatesPackmol', 'openff.evaluator.protocols.coordinates.SolvateExistingStructure', 'openff.evaluator.protocols.coordinates.BuildDockedCoordinates', 'openff.evaluator.protocols.forcefield.BaseBuildSystem', 'openff.evaluator.protocols.forcefield.TemplateBuildSystem', 'openff.evaluator.protocols.forcefield.BuildSmirnoffSystem', 'openff.evaluator.protocols.forcefield.
Exception: AssertionError('daemonic processes are not allowed to have children')

This is addressed by Evaluator's documentation, which recommends setting up a dask distributed configuration that switches daemons off: https://openff-evaluator.readthedocs.io/en/stable/backends/daskbackends.html#configuration

However, IMO recommendations should be left for settings that optimize program execution, not are essential to function. Also, needing to set a global configuration is not ideal if you want to run dask with other configurations. You can set environment variables in your script, but doing that for something that's essential for function is also not ideal.

Dask has a configuration context manager: https://docs.dask.org/en/stable/configuration.html#dask.config.set
Would it be possible to use this in Evaluator? Alternatively, would it be possible to emit a warning or error upon seeing known bad configurations? While the Dask worker job dies on SLURM, the overall manager job continues, so it's easy to incur some wastage of computer time.

Tests

I would have looked into this further out of interest, but was a bit stumped by the tests not working. The backend tests are currently skipped because of Travis. After removing the skip, they continued not working because GH runners aren't an HPC environment, I guess (https://github.com/lilyminium/openff-evaluator/tree/lily/dask-backends). Is OpenFF interested in bringing these tests back? The Dask Jobqueue library sets up Docker images for testing.