swarm: use docker images for local tests

[oliver-sanders]

Related:

• Use CI to run tests for batch systems #3372
• swarm: use docker compose #3935
• swarm: build against different python versions #3936

The Docker “swarm” used by the test battery uses containers to emulate remote job platforms and runs the tests themselves on the local platform.

This largely replicates the old system; it allows us to run the test battery on the system it will be deployed on including the job platforms available which is very useful.

However:

• This does introduce some complexity in that the host system needs to be configured to see these containers as remote hosts. This is highly useful as it allows us to use these containers not just for testing but also for live development, however, it is a pain where this functionality is not desired.
• Also, we don’t have an integrated solution to testing against a build matrix of Python/Bash versions, etc.
• Testing job runners would require installation and configuration of the job submission system on the host machine in order to be able to submit jobs to the container which would be very, very messy.

I suggest that we make it possible to run the whole test battery in containers where desired whilst still supporting running tests locally. We can then install and configure the job runner stuff on these images allowing testing of slurm, pbs, etc. You wouldn’t be able to use these containers from your local system (without doing the heavy lifting yourself) but would still be able to use the background/at containers as well as available platforms for local testing.

Something along these lines:

$ ctb -c cylc_py37_bash_5 -p ‘_remote_background_indep_tcp’ tests/f
$ docker cylc_py37_bash_5 exec pytest tests/u

[oliver-sanders]

See comment on this issue #4197 about building images with both pip/conda to test dependency changes.

[oliver-sanders]

Elaboration on the above comment...

• We use two containers.
• On the first container:
  • We install Conda and Conda pack (can use the miniforge or mambaforge base image)
  • We use this to build a Cylc environment using the conda-environment.yml.
  • We conda-pack this into a tarball.
• On the second container:
  • We unpack this tarball.
  • We somehow make this the system env (this is the tricky bit).
    • This would allow us to test different versions of Bash, Coreutils, etc.
    • The rough and ready approach is to replace the /bin/ execs with links to the Conda bin/ but that's messy.
    • Otherwise we are left bodging shell scripts (only works for login scripts).
  • We pip install -e the cylc-flow repo.

We then dispose of the first container (job done).