rebuild the conda environment

[prjemian]

Using pip, we ruined the conda environment. Rebuild the the env.yml file.

[prjemian]

Notice that conda is very slow to solve the package requirements. micromamba solves this list much faster and also downloads packages in parallel. But micromamba environments aren't activated by conda activate packagename.

Use micromamba to solve the package requirements and then create a conda environment from the explicit list. This is still faster than only using conda to solve the requirements.

• Refine the env.yml file by iteration:
  • Create a test environment: micromamba create -y -n testenvironment-f ./env.yml
  • Activate with micromamba activate testenvironment
  • Test that your python code works
• create explicit packages file: conda list --explicit | tee /tmp/conda-list-explicit.txt
• create the desired conda environment: conda create --name bdp2022 --file /tmp/conda-list-explicit.txt
• Activate with conda activate bdp2022
• pip install any remaining packages from the env.yml file
• test that your python code still works

[prjemian]

@sveseli This might interest you.

[sveseli]

@prjemian Thanks, very interesting... we might actually be able to use micromamba for DM if it speeds things up considerably

[prjemian]

Seems that pvapy requires a PVA client library from epics-base that is provided in the version from https://anaconda.org/epics/epics-base but that library is not packaged with https://anaconda.org/conda-forge/epics-base

(bdptest) bdp@terrier ~/DM/workflows/example-05 $ python blueskyImageServer.py --input-file /gdata/bdp/BDP/bdp-test-02/bdp_000001.tif
Traceback (most recent call last):
  File "/clhome/BDP/DM/workflows/example-05/blueskyImageServer.py", line 5, in <module>
    import pvaccess as pva
  File "/clhome/BDP/micromamba/envs/bdptest/lib/python3.9/site-packages/pvaccess/__init__.py", line 1,                 in <module>
    from .pvaccess import *
ImportError: libpvaClient.so.4.8.0: cannot open shared object file: No such file or directory

It is necessary to re-arrange the order of the channels in the env.yml file accordingly.

[sveseli]

If you install pvapy via pip, it has all epics libraries included (no need for epics-base conda packages). Epics base conda package is needed for conda pvapy packages.

[prjemian]

That's what ruined the conda environment.

[prjemian]

Specifically, the pip install did not consider the existing package requirements and managed to replace some conda installed packages. Not sure which ones.

[prjemian]

(bdp2022) bdp@terrier ~/DM/workflows/example-05 $ conda list | wc -l
408

That's a lot of things to consider. (3 header lines, so 40 other packages to consider)

[prjemian]

I've measured that micromamba is faster:

• measured 2x faster in unit test workflow
• measured almost 2x faster in unit test workflow
• recall about 15x faster to create some environment (can't find that now)

[prjemian]

With ~30 minutes per iteration using conda to refine env.yml, use of micromamba made this much faster.