Pip breaks when downgrading the version of an already installed dependency

[buhrmann]

Description

When installing a package with a dependency that is already installed but needs downgrading, pip downgrades the required dependency ,but then fails and complains about not finding the original version of the dependency anymore.

Expected behavior

Pip shouldn't look for a previously installed version if it has itself just changed that version, I guess, but not sure about the details of what's going on.

pip version

23.1.2

Python version

3.9.16

OS

Linux/Amd64

How to Reproduce

I'm working on a mac with apple silicon, but this occurs on linux/amd64, so the minimal steps to reproduce for me are:

docker pull --platform linux/amd64 condaforge/mambaforge:latest
docker run -it --rm --platform linux/amd64 docker.io/condaforge/mambaforge:latest

Inside the container create a minimal python environment:

mamba init && source /root/.bashrc
mamba create -y -n test python=3.9 wrapt=1.15.0
mamba activate test

Check what's inside:

mamba env export
ls /opt/conda/envs/test/lib/python3.9/site-packages/wrapt-1.15.0.dist-info/
cat /opt/conda/envs/test/lib/python3.9/site-packages/wrapt-1.15.0.dist-info/INSTALLER

Try pip install of these two packages:

pip install -v tensorflow-cpu==2.10.0 git+https://github.com/graphext/chars2vec.git@master#egg=chars2vec

Output

The above commands lead to this initial python environment:

$mamba env export

name: test
  channels:
    - conda-forge
  dependencies:
    - _libgcc_mutex=0.1=conda_forge
    - _openmp_mutex=4.5=2_gnu
    - bzip2=1.0.8=h7f98852_4
    - ca-certificates=2023.5.7=hbcca054_0
    - ld_impl_linux-64=2.40=h41732ed_0
    - libffi=3.4.2=h7f98852_5
    - libgcc-ng=13.1.0=he5830b7_0
    - libgomp=13.1.0=he5830b7_0
    - libnsl=2.0.0=h7f98852_0
    - libsqlite=3.42.0=h2797004_0
    - libuuid=2.38.1=h0b41bf4_0
    - libzlib=1.2.13=hd590300_5
    - ncurses=6.4=hcb278e6_0
    - openssl=3.1.1=hd590300_1
    - pip=23.1.2=pyhd8ed1ab_0
    - python=3.9.16=h2782a2a_0_cpython
    - python_abi=3.9=3_cp39
    - readline=8.2=h8228510_1
    - setuptools=68.0.0=pyhd8ed1ab_0
    - tk=8.6.12=h27826a3_0
    - tzdata=2023c=h71feb2d_0
    - wheel=0.40.0=pyhd8ed1ab_0
    - wrapt=1.15.0=py39h72bdee0_0
    - xz=5.2.6=h166bdaf_0
  prefix: /opt/conda/envs/test

Note that we have wrapt=1.15.0 installed, with these files:

$ ls /opt/conda/envs/test/lib/python3.9/site-packages/wrapt-1.15.0.dist-info/
INSTALLER  LICENSE  METADATA  RECORD  REQUESTED  WHEEL  direct_url.json  top_level.txt

INSTALLER AND REQUESTED are empty though, in case that's important.

The important bits from pip's verbose log are the following snippets (full log attached below):

Collecting wrapt<1.15,>=1.11.0 (from tensorflow->-r requirements.txt (line 3))
    Downloading wrapt-1.14.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (77 kB)
...

Installing collected packages: libclang, flatbuffers, zipp, wrapt, urllib3, typing-extensions, termcolor, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, pyasn1, protobuf, packaging, oauthlib, numpy, MarkupSafe, keras, idna, grpcio, gast, charset-normalizer, certifi, cachetools, absl-py, werkzeug, scipy, rsa, requests, pyasn1-modules, opt-einsum, ml-dtypes, importlib-metadata, h5py, google-pasta, astunparse, requests-oauthlib, markdown, jax, google-auth, google-auth-oauthlib, tensorboard, tensorflow
    Attempting uninstall: wrapt
      Found existing installation: wrapt 1.15.0
      Uninstalling wrapt-1.15.0:
        Successfully uninstalled wrapt-1.15.0
  Successfully installed MarkupSafe-2.1.3 absl-py-1.4.0 astunparse-1.6.3 cachetools-5.3.1 certifi-2023.5.7 charset-normalizer-3.1.0 flatbuffers-23.5.26 gast-0.4.0 google-auth-2.21.0 google-auth-oauthlib-1.0.0 google-pasta-0.2.0 grpcio-1.56.0 h5py-3.9.0 idna-3.4 importlib-metadata-6.7.0 jax-0.4.13 keras-2.12.0 libclang-16.0.0 markdown-3.4.3 ml-dtypes-0.2.0 numpy-1.23.5 oauthlib-3.2.2 opt-einsum-3.3.0 packaging-23.1 protobuf-4.23.3 pyasn1-0.5.0 pyasn1-modules-0.3.0 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 scipy-1.11.1 six-1.16.0 tensorboard-2.12.3 tensorboard-data-server-0.7.1 tensorflow-2.12.0 tensorflow-estimator-2.12.0 tensorflow-io-gcs-filesystem-0.32.0 termcolor-2.3.0 typing-extensions-4.7.0 urllib3-1.26.16 werkzeug-2.3.6 wrapt-1.14.1 zipp-3.15.0
...

Requirement already satisfied: wrapt>=1.11.0 in /opt/conda/envs/test/lib/python3.9/site-packages (from tensorflow-cpu==2.10.0) (1.15.0)
...

FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/envs/test/lib/python3.9/site-packages/wrapt-1.15.0.dist-info/METADATA'

pip-log.txt

So what seems to happen is that we start with wrapt=1.15.0. One of the two packages to be installed requires wrapt<=1.15 and the other wrapt>=1.11, which seem to be compatible. Pip first satisfies wrapt<=1.15 by uninstalling the current wrapt version and installing wrapt=1.14. So far so good. But then, when it seems to get to the second package to be installed, it isn't aware of having just replaced the wrapt version, and keeps looking for the previously installed version (1.15) which obviously it can't find anymore, and therefore fails the install.

Is this just a limitation of how Pip resolves (or doesn't resolve) dependencies, or is it a bug? At first the issue sounded related to #9129, but in this case conda doesn't really come into play, apart from creating the base environment.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

chars2vec runs a pip install command in its setup.py. That seems like a very bad idea. It shouldn't be trying to install build dependencies itself, but rather should define them in its own build-system.requires setting.

I don't think we'd support running nested copies of pip like this. As a temporary fix, you could try building a wheel for chars2vec first and then using that in your install command, but the better solution is to get chars2vec to fix their build process.

[buhrmann]

Great catch, I hadn't noticed, and will try without that manual installation. I think I've seen the same happening with other packages though, so I'm not sure that's the cause. But I'll try and come back with more info 👍🏼

[buhrmann]

That sort of seems to work in this case, thanks so much! It doesn't really work, because it install both tensorflow-cpu and tensorflow (the GPU variant) and then crashes at runtime when trying to import tensorflow, but that's a different problem and I think expected under the rules of pip's dependency resolution (?). In any case, I'll close this issue and if the original problem pops up in another context (without the manual pip call in setup.py), I'll open a new one.