Make CIs obey environment.yaml

[riedel]

Actually environment.yaml is installed outside the actual docker images and mostly even does not make sense.

• use the yaml file inside the dockerfile for building the CI images
• no need to install dependencies outside the docker for cluster CIs
• use pip versions of dask and distributed since the conda versions are outdated
• make htcondor use conda

At the moment one needs to edit tons of files if you actually want to make sure a test dependency is working

[guillaumeeb]

I agree that we should aim to use only one conda/pip build and one environment.yaml, same for inside and outside docker images. So we probably can simplify things!

I'm not sure if we should build it outside or inside the Docker image. I agree that I don't see a lot of sense at first in building conda env outside the image, except it can be updated without touching the images, maybe?

So it allows to decouple Docker images and conda env, and just mounting the env to be used in CI. I did not look back at the code since a long time.

use pip versions of dask and distributed since the conda versions are outdated

Also a bit surprised about this point. Outdated even in conda-forge?

[guillaumeeb]

use pip versions of dask and distributed since the conda versions are outdated

I think this is not true using conda-forge.

[riedel]

use pip versions of dask and distributed since the conda versions are outdated

I think this is not true using conda-forge.

I was just thinking of doing the same as distributed does with dask: https://github.com/dask/distributed/blob/d72fd2186256651dad63ada1d73b9d4697b573df/continuous_integration/environment-3.9.yaml#L50-L52

[guillaumeeb]

Oh OK, I didn't understand. I think this is not necessary for us, relying on official releases of Dask and Distributed seems the right choice to me.