Locking fails when pypi package depends on package being installed via conda

[NatPRoach]

Hello, I'm attempting to lock the following environment.

name: example
channels:
  - conda-forge
  - bioconda
  - default
dependencies:
  - pybedlite=0.0.2
  - pip:
    - fgpyo==0.0.3

The dependencies are, simplified, something like this:

graph TD;
    A[.];
    B[fgpyo==0.0.3];
    C[pybedlite=0.0.2];
    D[samwell>=0.0.2];
    E[pybedlite>=0.0.1];
    A-->|pip|B;
    A-->|conda|C;
    B-->D;
    D-->E;

Loading

conda-lock is failing attempting to handle it, seemingly because pybedlite v0.0.2 isn't on pip due to having a conda-dependency. And apparently conda-lock doesn't register that pybedlite >= 0.0.1 is already being installed via conda.

Is there a work-around I'm missing that could possibly fix this?

[mariusvniekerk]

No, the pip resolver will probably always fail (like 95% sure, but without a stack trace its hard to guess)

I would recommend that you get fgpyo into conda-forge if possible

[nh13]

I am a bit confused, if conda env install -f env.yml, it would be unintuitive if conda-lock can't lock versions here. E.g. conda env export is able to give package versions. There's clearly PEBKAC on my end.

[Kirill888]

I think I'm having similar problem, but in my case things get solved just fine, but installation fails as windows and macos specific packages get pulled into linux-64 install via transitive pypi dependencies

name: test
channels:
  - https://conda.anaconda.org/conda-forge
dependencies:
  - python=3.10
  - pip=22.3
  - jupyterlab=3
  - pip:
    - jupyterlab-code-formatter

In the above jupyterlab-code-formatter depend on jupyterlab and nothing else, when installing directly with conda, things are fine. When generating lock file I would expect to only have jupyterlab-code-formatter to be installed via pip, but I also end up with things like this in the lock file:

- dependencies: {}
  hash:
    sha256: 01b6400dd79212f50a2f01af1c65b781290ff39610853db99bf03962eb9a615f
  manager: pip
  name: pywinpty
  platform: linux-64
  source: null
  url: https://files.pythonhosted.org/packages/22/77/59645ee028f410ef1d582fc2923d2cb61016ed38e6ae5022f405227ca2ad/pywinpty-2.0.9.tar.gz
  version: 2.0.9

this package is not needed and can not be installed on linux. I believe this is pulled in by ipython or ipykernel, which are pretty common dependencies to have, and even though these are installed via conda, their pypi dependencies are brought into the environment if any of the pip packages have them as direct or transitive dependencies.

I would assume that dependency traversal when looking at pip packages would just stop as soon as dependency available on conda is found, but it looks like tree traversal continues. Separate issue is that pip platform guards are not respected by the looks of it.

Simply deleting those pypi entries from lock file manually works fine, and then environment can be built and functions fine pip check and all that. But deleting things after solving is not great, are there built-in features like "ignore these packages" or similar that I can use to overcome those issues?

[Kirill888]

Alternatively and option like "do not attempt to solve pip, but still install what's listed with an assumption that human solved those" would be nice to have for situations like this. I kind of global --no-deps for all pip packages.

[maresb]

Making the pip dependencies robust would be extremely difficult since there are so many ambiguous edge cases. But I think meaningful progress could be made in this direction.

For the particular jupyterlab-code-formatter case, the best and simplest solution seems to me like adding it to conda-forge. And actually, it already exists on conda-forge with underscores: jupyterlab_code_formatter.

[Kirill888]

thanks, inconsistent use of _ vs - between pip and conda always trip me up. That and copying environment files from years ago without checking conda-forge. That solves my problem, thanks again

[maresb]

thanks, inconsistent use of _ vs - between pip and conda always trip me up.

Yes, me too! I wish I could do something about it, but I'm not sure what.

[weiji14]

thanks, inconsistent use of _ vs - between pip and conda always trip me up.

Yes, me too! I wish I could do something about it, but I'm not sure what.

You could document the underscore/hyphen name aliases in https://github.com/conda-incubator/grayskull/blame/main/grayskull/strategy/config.yaml 😉