Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rebuild the conda environment #50

Closed
prjemian opened this issue Jul 8, 2022 · 10 comments
Closed

rebuild the conda environment #50

prjemian opened this issue Jul 8, 2022 · 10 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@prjemian
Copy link
Contributor

prjemian commented Jul 8, 2022

Using pip, we ruined the conda environment. Rebuild the the env.yml file.

@prjemian prjemian added the bug Something isn't working label Jul 8, 2022
@prjemian prjemian added this to the BDP M6 milestone Jul 8, 2022
@prjemian prjemian self-assigned this Jul 8, 2022
@prjemian
Copy link
Contributor Author

prjemian commented Jul 8, 2022

Notice that conda is very slow to solve the package requirements. micromamba solves this list much faster and also downloads packages in parallel. But micromamba environments aren't activated by conda activate packagename.

Use micromamba to solve the package requirements and then create a conda environment from the explicit list. This is still faster than only using conda to solve the requirements.

  • Refine the env.yml file by iteration:
    • Create a test environment: micromamba create -y -n testenvironment-f ./env.yml
    • Activate with micromamba activate testenvironment
    • Test that your python code works
  • create explicit packages file: conda list --explicit | tee /tmp/conda-list-explicit.txt
  • create the desired conda environment: conda create --name bdp2022 --file /tmp/conda-list-explicit.txt
  • Activate with conda activate bdp2022
  • pip install any remaining packages from the env.yml file
  • test that your python code still works

@prjemian
Copy link
Contributor Author

prjemian commented Jul 8, 2022

@sveseli This might interest you.

@sveseli
Copy link
Collaborator

sveseli commented Jul 8, 2022

@prjemian Thanks, very interesting... we might actually be able to use micromamba for DM if it speeds things up considerably

@prjemian
Copy link
Contributor Author

prjemian commented Jul 8, 2022

Seems that pvapy requires a PVA client library from epics-base that is provided in the version from https://anaconda.org/epics/epics-base but that library is not packaged with https://anaconda.org/conda-forge/epics-base

(bdptest) bdp@terrier ~/DM/workflows/example-05 $ python blueskyImageServer.py --input-file /gdata/bdp/BDP/bdp-test-02/bdp_000001.tif
Traceback (most recent call last):
  File "/clhome/BDP/DM/workflows/example-05/blueskyImageServer.py", line 5, in <module>
    import pvaccess as pva
  File "/clhome/BDP/micromamba/envs/bdptest/lib/python3.9/site-packages/pvaccess/__init__.py", line 1,                 in <module>
    from .pvaccess import *
ImportError: libpvaClient.so.4.8.0: cannot open shared object file: No such file or directory

It is necessary to re-arrange the order of the channels in the env.yml file accordingly.

@sveseli
Copy link
Collaborator

sveseli commented Jul 8, 2022

If you install pvapy via pip, it has all epics libraries included (no need for epics-base conda packages). Epics base conda package is needed for conda pvapy packages.

@prjemian
Copy link
Contributor Author

prjemian commented Jul 8, 2022

That's what ruined the conda environment.

@prjemian
Copy link
Contributor Author

prjemian commented Jul 8, 2022

Specifically, the pip install did not consider the existing package requirements and managed to replace some conda installed packages. Not sure which ones.

@prjemian
Copy link
Contributor Author

prjemian commented Jul 8, 2022

(bdp2022) bdp@terrier ~/DM/workflows/example-05 $ conda list | wc -l
408

That's a lot of things to consider. (3 header lines, so 40 other packages to consider)

@prjemian
Copy link
Contributor Author

prjemian commented Jul 8, 2022

I've measured that micromamba is faster:

  • measured 2x faster in unit test workflow
  • measured almost 2x faster in unit test workflow
  • recall about 15x faster to create some environment (can't find that now)

@prjemian
Copy link
Contributor Author

prjemian commented Jul 8, 2022

With ~30 minutes per iteration using conda to refine env.yml, use of micromamba made this much faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
No open projects
Development

No branches or pull requests

2 participants