Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip takes too long to resolve conflicting dependencies #9517

Closed
pabloa opened this issue Jan 25, 2021 · 21 comments
Closed

pip takes too long to resolve conflicting dependencies #9517

pabloa opened this issue Jan 25, 2021 · 21 comments

Comments

@pabloa
Copy link

pabloa commented Jan 25, 2021

What did you want to do?
We run the command:

pip install -e .

on a CI node with no previous cache using a virtual environment.

There was NO cache and there were NO previous packages. We are using a new virtual environment

Python 3.7.3

setup.py

        'spacy==2.2.3',  # Also run: # sudo python -m spacy download en
        'sentence-transformers',
        'sentencepiece==0.1.85', 
        'scispacy>=0.2.0',
        'nltk',
        'python-docx',
        'langdetect',
        'fuzzywuzzy==0.15.0',
        'python-Levenshtein==0.12.0',
        'annoy==1.16.3',
        'pyDatalog',
        'srsly==1.0.1',
        'nest_asyncio==1.4.2', 

Output

Collecting Flask==0.12.2
  Using cached Flask-0.12.2-py2.py3-none-any.whl (83 kB)
INFO: pip is looking at multiple versions of apispec to determine which version is compatible with other requirements. This could take a while.
Collecting apispec==0.35.0
  Using cached apispec-0.35.0-py2.py3-none-any.whl (22 kB)
INFO: pip is looking at multiple versions of modelling2 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of langdetect to determine which version is compatible with other requirements. This could take a while.
Collecting langdetect
  Using cached langdetect-1.0.7.zip (998 kB)
  Using cached langdetect-1.0.6.zip (995 kB)
  Using cached langdetect-1.0.5.zip (993 kB)
  Using cached langdetect-1.0.4.zip (960 kB)
  Using cached langdetect-1.0.3.zip (934 kB)
  Using cached langdetect-1.0.2.zip (934 kB)
  Using cached langdetect-1.0.1.zip (934 kB)
  Using cached langdetect-1.0.0.zip (934 kB)
  Using cached langdetect-0.1.0.zip (924 kB)
INFO: pip is looking at multiple versions of sympy to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of statsmodels to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of redis to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pyjwt to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pika to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of parsimonious to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of matplotlib to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of marshmallow to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of language-tags to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of html5lib to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of flask-testing to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of flask-swagger-ui to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of flask-cors to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of flask-compress to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of flask-apispec to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of flask to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of apispec to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of modelling2 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of langdetect to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of conllu to determine which version is compatible with other requirements. This could take a while.
Collecting conllu
  Using cached conllu-4.2.2-py2.py3-none-any.whl (14 kB)
  Using cached conllu-4.2.1-py2.py3-none-any.whl (14 kB)
  Using cached conllu-4.2-py2.py3-none-any.whl (14 kB)
  Using cached conllu-4.1-py2.py3-none-any.whl (14 kB)
  Using cached conllu-4.0-py2.py3-none-any.whl (13 kB)
  Using cached conllu-3.1.1-py2.py3-none-any.whl (14 kB)
  Using cached conllu-3.1-py2.py3-none-any.whl (14 kB)
  Using cached conllu-3.0-py2.py3-none-any.whl (14 kB)
  Using cached conllu-2.3.2-py2.py3-none-any.whl (13 kB)
  Using cached conllu-2.3-py2.py3-none-any.whl (13 kB)
  Using cached conllu-2.2.2-py2.py3-none-any.whl (13 kB)
  Using cached conllu-2.2.1-py2.py3-none-any.whl (12 kB)
  Using cached conllu-2.2-py2.py3-none-any.whl (12 kB)
  Using cached conllu-2.1.1-py2.py3-none-any.whl (12 kB)
  Using cached conllu-2.1-py2.py3-none-any.whl (12 kB)
  Using cached conllu-2.0-py2.py3-none-any.whl (10 kB)
  Using cached conllu-1.5-py2.py3-none-any.whl (9.8 kB)
  Using cached conllu-1.4.1-py2.py3-none-any.whl (9.5 kB)
  Using cached conllu-1.4-py2.py3-none-any.whl (9.5 kB)
  Using cached conllu-1.3.2-py2.py3-none-any.whl (9.3 kB)
  Using cached conllu-1.3.1-py2.py3-none-any.whl (9.3 kB)
  Using cached conllu-1.3-py2.py3-none-any.whl (9.3 kB)
  Using cached conllu-1.2.3-py2.py3-none-any.whl (9.2 kB)
  Using cached conllu-1.2.2-py2.py3-none-any.whl (9.2 kB)
  Using cached conllu-1.2.1-py2.py3-none-any.whl (9.1 kB)
  Using cached conllu-1.2-py2.py3-none-any.whl (9.1 kB)
  Using cached conllu-1.1-py2.py3-none-any.whl (8.8 kB)
  Using cached conllu-1.0.1-py2.py3-none-any.whl (8.3 kB)
  Using cached conllu-1.0-py2.py3-none-any.whl (8.2 kB)
  Using cached conllu-0.11-py2.py3-none-any.whl (6.8 kB)
  Using cached conllu-0.10.7-py2.py3-none-any.whl (6.3 kB)
  Using cached conllu-0.10.6-py2.py3-none-any.whl (6.2 kB)
  Using cached conllu-0.10.5-py2.py3-none-any.whl (6.2 kB)
  Using cached conllu-0.10.4-py2.py3-none-any.whl (6.2 kB)
  Using cached conllu-0.10.3-py3-none-any.whl (5.4 kB)
  Using cached conllu-0.10.2-py3-none-any.whl (5.4 kB)
  Using cached conllu-0.10.1.tar.gz (5.4 kB)
  Using cached conllu-0.10.tar.gz (5.3 kB)
  Using cached conllu-0.9.tar.gz (4.5 kB)
  Using cached conllu-0.8.tar.gz (4.5 kB)
  Using cached conllu-0.7.tar.gz (4.4 kB)
  Using cached conllu-0.6.1.tar.gz (4.3 kB)
  Using cached conllu-0.6.tar.gz (4.2 kB)
  Using cached conllu-0.5.tar.gz (3.8 kB)
  Using cached conllu-0.4.tar.gz (2.1 kB)
  Using cached conllu-0.3.tar.gz (2.0 kB)
  Using cached conllu-0.2-py2.py3-none-any.whl (8.2 kB)
  Using cached conllu-0.1.tar.gz (1.6 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of conllu to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.1-py2.py3-none-any.whl (136 kB)
  Using cached urllib3-1.26.0-py2.py3-none-any.whl (136 kB)
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
  Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
  Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
  Using cached urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
  Using cached urllib3-1.25.7-py2.py3-none-any.whl (125 kB)
  Using cached urllib3-1.25.6-py2.py3-none-any.whl (125 kB)
  Using cached urllib3-1.25.5-py2.py3-none-any.whl (125 kB)
  Using cached urllib3-1.25.4-py2.py3-none-any.whl (125 kB)
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.
Collecting six>=1.5
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
  Using cached six-1.13.0-py2.py3-none-any.whl (10 kB)
  Using cached six-1.12.0-py2.py3-none-any.whl (10 kB)
  Using cached six-1.11.0-py2.py3-none-any.whl (10 kB)
  Using cached six-1.10.0-py2.py3-none-any.whl (10 kB)
  Using cached six-1.9.0-py2.py3-none-any.whl (10 kB)
  Using cached six-1.8.0-py2.py3-none-any.whl (9.7 kB)
  Using cached six-1.7.3-py2.py3-none-any.whl (9.5 kB)
  Using cached six-1.7.2-py2.py3-none-any.whl (9.6 kB)
  Using cached six-1.7.1-py2.py3-none-any.whl (9.6 kB)
  Using cached six-1.7.0-py2.py3-none-any.whl (9.3 kB)
  Using cached six-1.6.1-py2.py3-none-any.whl (8.6 kB)
  Using cached six-1.6.0-py2.py3-none-any.whl (8.5 kB)
  Using cached six-1.5.2-py2.py3-none-any.whl (8.4 kB)
  Using cached six-1.5.1-py2.py3-none-any.whl (8.4 kB)
  Using cached six-1.5.0-py2.py3-none-any.whl (8.2 kB)
INFO: pip is looking at multiple versions of six to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Build timed out (after 210 minutes). Marking the build as aborted.
Build was aborted

Additional information

ipaddress==1.0.23
modelling2==0.11.22-builder.310def657e15f16895db81aa427271b04f08230b
  - apispec [required: ==0.35.0, installed: 0.35.0]
    - PyYAML [required: >=3.10, installed: 5.3.1]
  - colorama [required: >=0.3.9, installed: 0.4.3]
  - cytoolz [required: >=0.10.1, installed: 0.11.0]
    - toolz [required: >=0.8.0, installed: 0.11.1]
  - dataIngest [required: Any, installed: 1.0.0]
    - argcomplete [required: >=0.8.4, installed: 1.12.2]
      - importlib-metadata [required: >=0.23,<4, installed: 3.4.0]
        - typing-extensions [required: >=3.6.4, installed: 3.7.4.3]
        - zipp [required: >=0.5, installed: 3.4.0]
    - boto3 [required: ==1.9.129, installed: 1.9.129]
      - botocore [required: >=1.12.129,<1.13.0, installed: 1.12.129]
        - docutils [required: >=0.10, installed: 0.15.2]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
          - six [required: >=1.5, installed: 1.10.0]
        - urllib3 [required: >=1.20,<1.25, installed: 1.24.1]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
      - s3transfer [required: >=0.2.0,<0.3.0, installed: 0.2.1]
        - botocore [required: >=1.12.36,<2.0.0, installed: 1.12.129]
          - docutils [required: >=0.10, installed: 0.15.2]
          - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
            - six [required: >=1.5, installed: 1.10.0]
          - urllib3 [required: >=1.20,<1.25, installed: 1.24.1]
    - botocore [required: ==1.12.129, installed: 1.12.129]
      - docutils [required: >=0.10, installed: 0.15.2]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.10.0]
      - urllib3 [required: >=1.20,<1.25, installed: 1.24.1]
    - cachetools [required: ==2.1.0, installed: 2.1.0]
    - Cython [required: ==0.28.5, installed: 0.28.5]
    - fuzzywuzzy [required: >=0.15.0, installed: 0.15.0]
    - llvmlite [required: <0.32.0, installed: 0.31.0]
    - lxml [required: >=3.3.3, installed: 4.6.2]
    - Markdown [required: >=2.0.1, installed: 3.3.3]
      - importlib-metadata [required: Any, installed: 3.4.0]
        - typing-extensions [required: >=3.6.4, installed: 3.7.4.3]
        - zipp [required: >=0.5, installed: 3.4.0]
    - numba [required: ==0.47.0, installed: 0.47.0]
      - llvmlite [required: >=0.31.0dev0, installed: 0.31.0]
      - numpy [required: Any, installed: 1.16.1]
      - setuptools [required: Any, installed: 39.0.1]
    - numpy [required: ==1.16.1, installed: 1.16.1]
    - pandas [required: >=0.22.0, installed: 0.24.2]
      - numpy [required: >=1.12.0, installed: 1.16.1]
      - python-dateutil [required: >=2.5.0, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.10.0]
      - pytz [required: >=2011k, installed: 2020.5]
    - paramiko [required: >=2.4.1, installed: 2.4.1]
      - bcrypt [required: >=3.1.3, installed: 3.2.0]
        - cffi [required: >=1.1, installed: 1.14.4]
          - pycparser [required: Any, installed: 2.20]
        - six [required: >=1.4.1, installed: 1.10.0]
      - cryptography [required: >=1.5, installed: 3.3.1]
        - cffi [required: >=1.12, installed: 1.14.4]
          - pycparser [required: Any, installed: 2.20]
        - six [required: >=1.4.1, installed: 1.10.0]
      - pyasn1 [required: >=0.1.7, installed: 0.4.8]
      - pynacl [required: >=1.0.1, installed: 1.4.0]
        - cffi [required: >=1.4.1, installed: 1.14.4]
          - pycparser [required: Any, installed: 2.20]
        - six [required: Any, installed: 1.10.0]
    - pathlib2 [required: >=2.1.0, installed: 2.3.5]
      - six [required: Any, installed: 1.10.0]
    - psycopg2-binary [required: >=2.7.3.2, installed: 2.8.6]
    - pyhocon [required: >=0.3.33, installed: 0.3.57]
      - pyparsing [required: >=2.0.3, installed: 2.4.7]
    - python-dateutil [required: >=2.6.0, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.10.0]
    - requests [required: Any, installed: 2.25.1]
      - certifi [required: >=2017.4.17, installed: 2020.12.5]
      - chardet [required: >=3.0.2,<5, installed: 3.0.4]
      - idna [required: >=2.5,<3, installed: 2.10]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.24.1]
    - sqlalchemy [required: >=1.1.5, installed: 1.3.22]
    - statsmodels [required: ==0.9.0, installed: 0.9.0]
      - pandas [required: Any, installed: 0.24.2]
        - numpy [required: >=1.12.0, installed: 1.16.1]
        - python-dateutil [required: >=2.5.0, installed: 2.8.1]
          - six [required: >=1.5, installed: 1.10.0]
        - pytz [required: >=2011k, installed: 2020.5]
      - patsy [required: Any, installed: 0.5.1]
        - numpy [required: >=1.4, installed: 1.16.1]
        - six [required: Any, installed: 1.10.0]
    - urllib3 [required: ==1.24.1, installed: 1.24.1]
  - dill [required: >=0.2.7.1, installed: 0.3.3]
  - diskcache [required: <5.0.0, installed: 4.1.0]
  - Flask [required: ==0.12.2, installed: 0.12.2]
    - click [required: >=2.0, installed: 7.1.2]
    - itsdangerous [required: >=0.21, installed: 1.1.0]
    - Jinja2 [required: >=2.4, installed: 2.11.2]
      - MarkupSafe [required: >=0.23, installed: 1.1.1]
    - Werkzeug [required: >=0.7, installed: 0.16.1]
  - flask-apispec [required: ==0.6.0.post0, installed: 0.6.0.post0]
    - apispec [required: >=0.17.0, installed: 0.35.0]
      - PyYAML [required: >=3.10, installed: 5.3.1]
    - flask [required: >=0.10.1, installed: 0.12.2]
      - click [required: >=2.0, installed: 7.1.2]
      - itsdangerous [required: >=0.21, installed: 1.1.0]
      - Jinja2 [required: >=2.4, installed: 2.11.2]
        - MarkupSafe [required: >=0.23, installed: 1.1.1]
      - Werkzeug [required: >=0.7, installed: 0.16.1]
    - marshmallow [required: >=2.0.0, installed: 3.0.0b12]
    - six [required: >=1.9.0, installed: 1.10.0]
    - webargs [required: >=0.18.0, installed: 1.8.1]
      - marshmallow [required: >=2.7.0, installed: 3.0.0b12]
  - Flask-Compress [required: ==1.4.0, installed: 1.4.0]
    - Flask [required: Any, installed: 0.12.2]
      - click [required: >=2.0, installed: 7.1.2]
      - itsdangerous [required: >=0.21, installed: 1.1.0]
      - Jinja2 [required: >=2.4, installed: 2.11.2]
        - MarkupSafe [required: >=0.23, installed: 1.1.1]
      - Werkzeug [required: >=0.7, installed: 0.16.1]
  - Flask-Cors [required: ==3.0.3, installed: 3.0.3]
    - Flask [required: >=0.9, installed: 0.12.2]
      - click [required: >=2.0, installed: 7.1.2]
      - itsdangerous [required: >=0.21, installed: 1.1.0]
      - Jinja2 [required: >=2.4, installed: 2.11.2]
        - MarkupSafe [required: >=0.23, installed: 1.1.1]
      - Werkzeug [required: >=0.7, installed: 0.16.1]
    - Six [required: Any, installed: 1.10.0]
  - flask-swagger-ui [required: ==3.20.9, installed: 3.20.9]
    - flask [required: Any, installed: 0.12.2]
      - click [required: >=2.0, installed: 7.1.2]
      - itsdangerous [required: >=0.21, installed: 1.1.0]
      - Jinja2 [required: >=2.4, installed: 2.11.2]
        - MarkupSafe [required: >=0.23, installed: 1.1.1]
      - Werkzeug [required: >=0.7, installed: 0.16.1]
  - Flask-Testing [required: ==0.7.1, installed: 0.7.1]
    - Flask [required: Any, installed: 0.12.2]
      - click [required: >=2.0, installed: 7.1.2]
      - itsdangerous [required: >=0.21, installed: 1.1.0]
      - Jinja2 [required: >=2.4, installed: 2.11.2]
        - MarkupSafe [required: >=0.23, installed: 1.1.1]
      - Werkzeug [required: >=0.7, installed: 0.16.1]
  - fuzzywuzzy [required: ==0.15.0, installed: 0.15.0]
  - googletrans [required: Any, installed: 3.0.0]
    - httpx [required: ==0.13.3, installed: 0.13.3]
      - certifi [required: Any, installed: 2020.12.5]
      - chardet [required: ==3.*, installed: 3.0.4]
      - hstspreload [required: Any, installed: 2020.12.22]
      - httpcore [required: ==0.9.*, installed: 0.9.1]
        - h11 [required: >=0.8,<0.10, installed: 0.9.0]
        - h2 [required: ==3.*, installed: 3.2.0]
          - hpack [required: >=3.0,<4, installed: 3.0.0]
          - hyperframe [required: >=5.2.0,<6, installed: 5.2.0]
        - sniffio [required: ==1.*, installed: 1.2.0]
      - idna [required: ==2.*, installed: 2.10]
      - rfc3986 [required: >=1.3,<2, installed: 1.4.0]
      - sniffio [required: Any, installed: 1.2.0]
  - html5lib [required: ==0.9999999, installed: 0.9999999]
    - six [required: Any, installed: 1.10.0]
  - ipdb [required: Any, installed: 0.13.4]
    - ipython [required: >=5.1.0, installed: 7.19.0]
      - backcall [required: Any, installed: 0.2.0]
      - decorator [required: Any, installed: 4.4.2]
      - jedi [required: >=0.10, installed: 0.18.0]
        - parso [required: >=0.8.0,<0.9.0, installed: 0.8.1]
      - pexpect [required: >4.3, installed: 4.8.0]
        - ptyprocess [required: >=0.5, installed: 0.7.0]
      - pickleshare [required: Any, installed: 0.7.5]
      - prompt-toolkit [required: >=2.0.0,<3.1.0,!=3.0.1,!=3.0.0, installed: 3.0.14]
        - wcwidth [required: Any, installed: 0.2.5]
      - pygments [required: Any, installed: 2.7.4]
      - setuptools [required: >=18.5, installed: 39.0.1]
      - traitlets [required: >=4.2, installed: 5.0.5]
        - ipython-genutils [required: Any, installed: 0.2.0]
    - setuptools [required: Any, installed: 39.0.1]
  - joblib [required: ==0.11, installed: 0.11]
  - language-tags [required: ==0.4.3, installed: 0.4.3]
    - six [required: Any, installed: 1.10.0]
  - llvmlite [required: !=0.32.0, installed: 0.31.0]
  - marshmallow [required: ==3.0.0b12, installed: 3.0.0b12]
  - matplotlib [required: ==2.1.0, installed: 2.1.0]
    - cycler [required: >=0.10, installed: 0.10.0]
      - six [required: Any, installed: 1.10.0]
    - numpy [required: >=1.7.1, installed: 1.16.1]
    - pyparsing [required: >=2.0.1,!=2.1.6,!=2.1.2,!=2.0.4, installed: 2.4.7]
    - python-dateutil [required: >=2.0, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.10.0]
    - pytz [required: Any, installed: 2020.5]
    - six [required: >=1.10, installed: 1.10.0]
  - multiprocess [required: ==0.70.4, installed: 0.70.4]
    - dill [required: >=0.2.5, installed: 0.3.3]
  - numexpr [required: >=2.6.2, installed: 2.7.2]
    - numpy [required: >=1.7, installed: 1.16.1]
  - numpy [required: ==1.16.1, installed: 1.16.1]
  - paho-mqtt [required: >=1.4.0, installed: 1.5.1]
  - pandas [required: ==0.24.2, installed: 0.24.2]
    - numpy [required: >=1.12.0, installed: 1.16.1]
    - python-dateutil [required: >=2.5.0, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.10.0]
    - pytz [required: >=2011k, installed: 2020.5]
  - parsimonious [required: ==0.8.1, installed: 0.8.1]
    - six [required: >=1.9.0, installed: 1.10.0]
  - pika [required: ==1.1.0, installed: 1.1.0]
  - psutil [required: Any, installed: 5.8.0]
  - psutil [required: >=5.2.2, installed: 5.8.0]
  - psycopg2-binary [required: >=2.8.4, installed: 2.8.6]
  - pydevd-pycharm [required: ~=192.6603.28, installed: 192.6603.34]
  - pyjwt [required: ==1.5.3, installed: 1.5.3]
  - python-Levenshtein [required: ==0.12.0, installed: 0.12.0]
    - setuptools [required: Any, installed: 39.0.1]
  - redis [required: ==2.10.6, installed: 2.10.6]
  - requests [required: Any, installed: 2.25.1]
    - certifi [required: >=2017.4.17, installed: 2020.12.5]
    - chardet [required: >=3.0.2,<5, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.24.1]
  - riemann-client [required: Any, installed: 6.5.0]
    - click [required: >=3.1, installed: 7.1.2]
    - protobuf [required: >=3.2.0,<4.0.0, installed: 3.14.0]
      - six [required: >=1.9, installed: 1.10.0]
  - scikit-learn [required: >=0.20.3, installed: 0.24.1]
    - joblib [required: >=0.11, installed: 0.11]
    - numpy [required: >=1.13.3, installed: 1.16.1]
    - scipy [required: >=0.19.1, installed: 1.2.0]
      - numpy [required: >=1.8.2, installed: 1.16.1]
    - threadpoolctl [required: >=2.0.0, installed: 2.1.0]
  - scipy [required: ==1.2.0, installed: 1.2.0]
    - numpy [required: >=1.8.2, installed: 1.16.1]
  - six [required: ==1.10.0, installed: 1.10.0]
  - SQLAlchemy [required: >=1.2.17, installed: 1.3.22]
  - statsmodels [required: ==0.9.0, installed: 0.9.0]
    - pandas [required: Any, installed: 0.24.2]
      - numpy [required: >=1.12.0, installed: 1.16.1]
      - python-dateutil [required: >=2.5.0, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.10.0]
      - pytz [required: >=2011k, installed: 2020.5]
    - patsy [required: Any, installed: 0.5.1]
      - numpy [required: >=1.4, installed: 1.16.1]
      - six [required: Any, installed: 1.10.0]
  - sympy [required: ==1.3, installed: 1.3]
    - mpmath [required: >=0.19, installed: 1.1.0]
  - tabulate [required: Any, installed: 0.8.7]
  - urllib3 [required: ==1.24.1, installed: 1.24.1]
  - watchdog [required: ==0.10.2, installed: 0.10.2]
    - pathtools [required: >=0.1.1, installed: 0.1.2]
  - webargs [required: ==1.8.1, installed: 1.8.1]
    - marshmallow [required: >=2.7.0, installed: 3.0.0b12]
  - Werkzeug [required: ==0.16.1, installed: 0.16.1]
pipdeptree==2.0.0
  - pip [required: >=6.0.0, installed: 21.0]
pkg-resources==0.0.0
pycfg==0.0.1
  - argcomplete [required: Any, installed: 1.12.2]
    - importlib-metadata [required: >=0.23,<4, installed: 3.4.0]
      - typing-extensions [required: >=3.6.4, installed: 3.7.4.3]
      - zipp [required: >=0.5, installed: 3.4.0]
  - awscli [required: Any, installed: 1.18.219]
    - botocore [required: ==1.19.59, installed: 1.12.129]
      - docutils [required: >=0.10, installed: 0.15.2]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.10.0]
      - urllib3 [required: >=1.20,<1.25, installed: 1.24.1]
    - colorama [required: >=0.2.5,<0.4.4, installed: 0.4.3]
    - docutils [required: >=0.10,<0.16, installed: 0.15.2]
    - PyYAML [required: >=3.10,<5.4, installed: 5.3.1]
    - rsa [required: >=3.1.2,<=4.5.0, installed: 4.5]
      - pyasn1 [required: >=0.1.3, installed: 0.4.8]
    - s3transfer [required: >=0.3.0,<0.4.0, installed: 0.2.1]
      - botocore [required: >=1.12.36,<2.0.0, installed: 1.12.129]
        - docutils [required: >=0.10, installed: 0.15.2]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
          - six [required: >=1.5, installed: 1.10.0]
        - urllib3 [required: >=1.20,<1.25, installed: 1.24.1]
  - boto3 [required: Any, installed: 1.9.129]
    - botocore [required: >=1.12.129,<1.13.0, installed: 1.12.129]
      - docutils [required: >=0.10, installed: 0.15.2]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.10.0]
      - urllib3 [required: >=1.20,<1.25, installed: 1.24.1]
    - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
    - s3transfer [required: >=0.2.0,<0.3.0, installed: 0.2.1]
      - botocore [required: >=1.12.36,<2.0.0, installed: 1.12.129]
        - docutils [required: >=0.10, installed: 0.15.2]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
          - six [required: >=1.5, installed: 1.10.0]
        - urllib3 [required: >=1.20,<1.25, installed: 1.24.1]
  - botocore [required: Any, installed: 1.12.129]
    - docutils [required: >=0.10, installed: 0.15.2]
    - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
    - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.10.0]
    - urllib3 [required: >=1.20,<1.25, installed: 1.24.1]
  - cryptography [required: Any, installed: 3.3.1]
    - cffi [required: >=1.12, installed: 1.14.4]
      - pycparser [required: Any, installed: 2.20]
    - six [required: >=1.4.1, installed: 1.10.0]
  - ecdsa [required: Any, installed: 0.16.1]
    - six [required: >=1.9.0, installed: 1.10.0]
  - ecdsa [required: Any, installed: 0.16.1]
    - six [required: >=1.9.0, installed: 1.10.0]
  - graphyte [required: Any, installed: 1.6.0]
  - jedi [required: ==0.9.0, installed: 0.18.0]
    - parso [required: >=0.8.0,<0.9.0, installed: 0.8.1]
  - jmespath [required: Any, installed: 0.10.0]
  - munch [required: Any, installed: 2.5.0]
    - six [required: Any, installed: 1.10.0]
  - paho-mqtt [required: Any, installed: 1.5.1]
  - paramiko [required: ==2.4.1, installed: 2.4.1]
    - bcrypt [required: >=3.1.3, installed: 3.2.0]
      - cffi [required: >=1.1, installed: 1.14.4]
        - pycparser [required: Any, installed: 2.20]
      - six [required: >=1.4.1, installed: 1.10.0]
    - cryptography [required: >=1.5, installed: 3.3.1]
      - cffi [required: >=1.12, installed: 1.14.4]
        - pycparser [required: Any, installed: 2.20]
      - six [required: >=1.4.1, installed: 1.10.0]
    - pyasn1 [required: >=0.1.7, installed: 0.4.8]
    - pynacl [required: >=1.0.1, installed: 1.4.0]
      - cffi [required: >=1.4.1, installed: 1.14.4]
        - pycparser [required: Any, installed: 2.20]
      - six [required: Any, installed: 1.10.0]
  - psycopg2-binary [required: >=2.7.3.2, installed: 2.8.6]
  - pycrypto [required: Any, installed: 2.6.1]
  - pyopenssl [required: Any, installed: 20.0.1]
    - cryptography [required: >=3.2, installed: 3.3.1]
      - cffi [required: >=1.12, installed: 1.14.4]
        - pycparser [required: Any, installed: 2.20]
      - six [required: >=1.4.1, installed: 1.10.0]
    - six [required: >=1.5.2, installed: 1.10.0]
  - python-dateutil [required: >=2.5.2, installed: 2.8.1]
    - six [required: >=1.5, installed: 1.10.0]
  - pytz [required: Any, installed: 2020.5]
  - redis [required: Any, installed: 2.10.6]
  - requests [required: Any, installed: 2.25.1]
    - certifi [required: >=2017.4.17, installed: 2020.12.5]
    - chardet [required: >=3.0.2,<5, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.24.1]
  - requests [required: Any, installed: 2.25.1]
    - certifi [required: >=2017.4.17, installed: 2020.12.5]
    - chardet [required: >=3.0.2,<5, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.24.1]
  - s3transfer [required: Any, installed: 0.2.1]
    - botocore [required: >=1.12.36,<2.0.0, installed: 1.12.129]
      - docutils [required: >=0.10, installed: 0.15.2]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.10.0]
      - urllib3 [required: >=1.20,<1.25, installed: 1.24.1]
  - urllib3 [required: Any, installed: 1.24.1]
pyswiplus==0.0.2
wheel==0.36.2Warning!!! Possibly conflicting dependencies found:
* awscli==1.18.219
 - botocore [required: ==1.19.59, installed: 1.12.129]
 - s3transfer [required: >=0.3.0,<0.4.0, installed: 0.2.1]
* pycfg==0.0.1
 - jedi [required: ==0.9.0, installed: 0.18.0]
@witalij-s

This comment has been minimized.

@uranusjr

This comment has been minimized.

@uranusjr uranusjr reopened this Jan 26, 2021
@uranusjr

This comment has been minimized.

@pabloa
Copy link
Author

pabloa commented Jan 26, 2021

Yes. All I described happens in Python 3.7.x

@pradyunsg
Copy link
Member

This seems to be a case where the resolver is trying too hard to find a solution that doesn't exist. I think that's the case because of the warning at the end of the pip deptree that's been posted.

I think this can be bundled into #9187, since nothing about this is specific to 21.0 AFAICT.

@pradyunsg pradyunsg changed the title pip 21.0 is broken pip takes too long to resolve conflicting dependencies Jan 27, 2021
@pabloa
Copy link
Author

pabloa commented Jan 27, 2021

It used to work with the old resolver. The current behavior is clearly a pip==21.0 issue. It does not happen with pip 20.x, for example, or older versions.

If pip tries to find a solution and it does not exist, it should print a warning and solves it like before or by doing an educated guess, or something. There are several projects like ours. They use/depend on packages with a lack of information. We still must deploy to production.

@uranusjr
Copy link
Member

The old resolver “works” in the same sense that a random person assess situations faster than an expert. The rando guesses randomly, while the expert needs to actually analyse the situation. If the old resolver gets things right, you’re lucky. But that’s not a very useful baseline to evaluate performance.

@pabloa
Copy link
Author

pabloa commented Jan 27, 2021

@uranusjr I am not saying pip must use the old resolver.
I am saying: If the new resolver cannot solve something because there are 1 or more bugs somewhere on the dependent packages, it should behave for that execution as the old resolver. Or at least pick something. And print a warning.

I recognize the new resolver is predictable and better. Still, there are scenarios like the one I reported where this resolver does not work. It was running for 3 hours and could not install all the dependencies!

Is it unreasonable to pick the best candidate package on those cases? The new resolver does not event report which package is unsolvable.

It has to be a better solution for this. And we are not the only ones having problems.

If picking one version of a package is not acceptable when the resolver cannot resolve, please suggest some alternative: print the package having issues? Restarting the installation using the old resolver?

Being unable to install a project with new versions of pip does not seem a solution.

@notatallshaw
Copy link
Member

notatallshaw commented Feb 20, 2021

I was trying to reproduce this example to find ways that pip resolver could be improved but it seems that is is no longer reproducible?

I used Python 3.7.3 and Pip 21.0.1 with this setup.py to try to reproduce it:

from setuptools import setup


setup(
    name="foo",
    version="1.0",
    install_requires=[
        'spacy==2.2.3',
        'sentence-transformers',
        'sentencepiece==0.1.85', 
        'scispacy>=0.2.0',
        'nltk',
        'python-docx',
        'langdetect',
        'fuzzywuzzy==0.15.0',
        'python-Levenshtein==0.12.0',
        'annoy==1.16.3',
        'pyDatalog',
        'srsly==1.0.1',
        'nest_asyncio==1.4.2']
)

And it successfully built:

(pip_test) PS test_pip> pip install -e .
Obtaining test_pip
Collecting spacy==2.2.3
  Downloading spacy-2.2.3-cp37-cp37m-win_amd64.whl (9.7 MB)
     |████████████████████████████████| 9.7 MB 3.3 MB/s
Collecting sentence-transformers
  Downloading sentence-transformers-0.4.1.2.tar.gz (64 kB)
     |████████████████████████████████| 64 kB 4.5 MB/s
Collecting sentencepiece==0.1.85
  Downloading sentencepiece-0.1.85-cp37-cp37m-win_amd64.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB ...
Collecting scispacy>=0.2.0
  Downloading scispacy-0.4.0-py3-none-any.whl (44 kB)
     |████████████████████████████████| 44 kB ...
Collecting nltk
  Downloading nltk-3.5.zip (1.4 MB)
     |████████████████████████████████| 1.4 MB 6.4 MB/s
Collecting python-docx
  Downloading python-docx-0.8.10.tar.gz (5.5 MB)
     |████████████████████████████████| 5.5 MB 6.8 MB/s
Collecting langdetect
  Downloading langdetect-1.0.8.tar.gz (981 kB)
     |████████████████████████████████| 981 kB 6.4 MB/s
Collecting fuzzywuzzy==0.15.0
  Downloading fuzzywuzzy-0.15.0-py2.py3-none-any.whl (28 kB)
Collecting python-Levenshtein==0.12.0
  Using cached python-Levenshtein-0.12.0.tar.gz (48 kB)
Collecting annoy==1.16.3
  Downloading annoy-1.16.3.tar.gz (644 kB)
     |████████████████████████████████| 644 kB 6.4 MB/s
Collecting pyDatalog
  Downloading pyDatalog-0.17.1.zip (304 kB)
     |████████████████████████████████| 304 kB ...
Collecting srsly==1.0.1
  Downloading srsly-1.0.1-cp37-cp37m-win_amd64.whl (179 kB)
     |████████████████████████████████| 179 kB ...
Collecting nest_asyncio==1.4.2
  Downloading nest_asyncio-1.4.2-py3-none-any.whl (5.3 kB)
Requirement already satisfied: setuptools in pip_test\lib\site-packages (from python-Levenshtein==0.12.0->foo==1.0) (52.0.0.post20210125)
Collecting preshed<3.1.0,>=3.0.2
  Downloading preshed-3.0.5-cp37-cp37m-win_amd64.whl (108 kB)
     |████████████████████████████████| 108 kB ...
Collecting blis<0.5.0,>=0.4.0
  Downloading blis-0.4.1-cp37-cp37m-win_amd64.whl (5.0 MB)
     |████████████████████████████████| 5.0 MB 6.8 MB/s
Collecting cymem<2.1.0,>=2.0.2
  Downloading cymem-2.0.5-cp37-cp37m-win_amd64.whl (35 kB)
Collecting catalogue<1.1.0,>=0.0.7
  Downloading catalogue-1.0.0-py2.py3-none-any.whl (7.7 kB)
Collecting thinc<7.4.0,>=7.3.0
  Downloading thinc-7.3.1-cp37-cp37m-win_amd64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 6.8 MB/s
Collecting numpy>=1.15.0
  Downloading numpy-1.20.1-cp37-cp37m-win_amd64.whl (13.6 MB)
     |████████████████████████████████| 13.6 MB 3.3 MB/s
Collecting murmurhash<1.1.0,>=0.28.0
  Downloading murmurhash-1.0.5-cp37-cp37m-win_amd64.whl (20 kB)
Collecting requests<3.0.0,>=2.13.0
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting plac<1.2.0,>=0.9.6
  Downloading plac-1.1.3-py2.py3-none-any.whl (20 kB)
Collecting wasabi<1.1.0,>=0.4.0
  Downloading wasabi-0.8.2-py3-none-any.whl (23 kB)
Collecting importlib-metadata>=0.20
  Downloading importlib_metadata-3.4.0-py3-none-any.whl (10 kB)
Collecting zipp>=0.5
  Downloading zipp-3.4.0-py3-none-any.whl (5.2 kB)
Collecting typing-extensions>=3.6.4
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
     |████████████████████████████████| 137 kB ...
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: certifi>=2017.4.17 in pip_test\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy==2.2.3->foo==1.0) (2020.12.5)
Collecting scispacy>=0.2.0
  Downloading scispacy-0.3.0-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 3.2 MB/s
Collecting nmslib>=1.7.3.6
  Downloading nmslib-2.1.1-cp37-cp37m-win_amd64.whl (662 kB)
     |████████████████████████████████| 662 kB ...
Collecting scispacy>=0.2.0
  Downloading scispacy-0.2.5-py3-none-any.whl (38 kB)
  Downloading scispacy-0.2.4.tar.gz (38 kB)
Collecting awscli
  Downloading awscli-1.19.12.tar.gz (1.4 MB)
     |████████████████████████████████| 1.4 MB 6.8 MB/s
Collecting conllu
  Downloading conllu-4.4-py2.py3-none-any.whl (15 kB)
Collecting joblib
  Downloading joblib-1.0.1-py3-none-any.whl (303 kB)
     |████████████████████████████████| 303 kB ...
Collecting scikit-learn>=0.20.3
  Downloading scikit_learn-0.24.1-cp37-cp37m-win_amd64.whl (6.8 MB)
     |████████████████████████████████| 6.8 MB 6.4 MB/s
Collecting pysbd
  Downloading pysbd-0.3.4-py3-none-any.whl (71 kB)
     |████████████████████████████████| 71 kB ...
Collecting psutil
  Downloading psutil-5.8.0-cp37-cp37m-win_amd64.whl (244 kB)
     |████████████████████████████████| 244 kB ...
Collecting pybind11<2.6.2
  Downloading pybind11-2.6.1-py2.py3-none-any.whl (188 kB)
     |████████████████████████████████| 188 kB 6.8 MB/s
Collecting scipy>=0.19.1
  Downloading scipy-1.6.1-cp37-cp37m-win_amd64.whl (32.6 MB)
     |████████████████████████████████| 32.6 MB 6.4 MB/s
Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)
Collecting tqdm<5.0.0,>=4.10.0
  Downloading tqdm-4.57.0-py2.py3-none-any.whl (72 kB)
     |████████████████████████████████| 72 kB 688 kB/s
Collecting botocore==1.20.12
  Downloading botocore-1.20.12-py2.py3-none-any.whl (7.2 MB)
     |████████████████████████████████| 7.2 MB 1.7 MB/s
Collecting docutils<0.16,>=0.10
  Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
     |████████████████████████████████| 547 kB ...
Collecting s3transfer<0.4.0,>=0.3.0
  Downloading s3transfer-0.3.4-py2.py3-none-any.whl (69 kB)
     |████████████████████████████████| 69 kB ...
Collecting PyYAML<5.5,>=3.10
  Downloading PyYAML-5.4.1-cp37-cp37m-win_amd64.whl (210 kB)
     |████████████████████████████████| 210 kB ...
Collecting colorama<0.4.4,>=0.2.5
  Downloading colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting rsa<=4.5.0,>=3.1.2
  Downloading rsa-4.5-py2.py3-none-any.whl (36 kB)
Collecting jmespath<1.0.0,>=0.7.1
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six>=1.5
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting pyasn1>=0.1.3
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting click
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 483 kB/s
Collecting regex
  Downloading regex-2020.11.13-cp37-cp37m-win_amd64.whl (269 kB)
     |████████████████████████████████| 269 kB ...
Collecting lxml>=2.3.2
  Downloading lxml-4.6.2-cp37-cp37m-win_amd64.whl (3.5 MB)
     |████████████████████████████████| 3.5 MB 3.3 MB/s
Collecting transformers<5.0.0,>=3.1.0
  Downloading transformers-4.3.2-py3-none-any.whl (1.8 MB)
     |████████████████████████████████| 1.8 MB ...
Collecting torch>=1.6.0
  Downloading torch-1.7.1-cp37-cp37m-win_amd64.whl (184.1 MB)
     |████████████████████████████████| 184.1 MB 4.6 kB/s
Collecting sacremoses
  Downloading sacremoses-0.0.43.tar.gz (883 kB)
     |████████████████████████████████| 883 kB 6.4 MB/s
Collecting tokenizers<0.11,>=0.10.1
  Downloading tokenizers-0.10.1-cp37-cp37m-win_amd64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 6.8 MB/s
Collecting filelock
  Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting packaging
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB ...
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
WARNING: The candidate selected for download or install is a yanked version: 'python-levenshtein' candidate (version 0.12.0 at https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz#sha256=033a11de5e3d19ea25c9302d11224e1a1898fe5abd23c61c7c360c25195e3eb1 (from https://pypi.org/simple/python-levenshtein/))
Reason for being yanked: Insecure, upgrade to 0.12.1
Building wheels for collected packages: annoy, python-Levenshtein, scispacy, awscli, langdetect, nltk, pyDatalog, python-docx, sentence-transformers, sacremoses
  Building wheel for annoy (setup.py) ... done
  Created wheel for annoy: filename=annoy-1.16.3-cp37-cp37m-win_amd64.whl size=48250 sha256=6d2d542f78b946e1cdb75f2727bf5474042c8bcf1c72d3b44386ba24bf6b6af6
  Stored in directory: pip\cache\wheels\39\36\d4\ee348a7240ca3e8d1fcbf04ebe46d45f2879ccb094a40f5706
  Building wheel for python-Levenshtein (setup.py) ... done
  Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp37-cp37m-win_amd64.whl size=81989 sha256=7ef7a28f7b36c10d775dc493addc63fd7ed936de8db65b2631724853ab6a4ee9
  Stored in directory: pip\cache\wheels\f0\9b\13\49c281164c37be18343230d3cd0fca29efb23a493351db0009
  Building wheel for scispacy (setup.py) ... done
  Created wheel for scispacy: filename=scispacy-0.2.4-py3-none-any.whl size=35204 sha256=08c8526b27c5c4b6f2f8b78bba973fe2a1d617959cf4573a880ba9b510453e35
  Stored in directory: pip\cache\wheels\96\24\54\8c2bf5a6804275431a74bd256821500b3f2911d9ba8470a846
  Building wheel for awscli (setup.py) ... done
  Created wheel for awscli: filename=awscli-1.19.12-py2.py3-none-any.whl size=3564391 sha256=095fbe4e66cdde5eba3d4edd7801a61d9dfce726337cb88d6999ed05cc4801c1
  Stored in directory: pip\cache\wheels\ae\6d\8e\fe9fcfdc1d0bb4a01c0ab812eb1f6d191d872314bf3ae71279
  Building wheel for langdetect (setup.py) ... done
  Created wheel for langdetect: filename=langdetect-1.0.8-py3-none-any.whl size=993190 sha256=8d5ea79fa6c45a021f03f4dab474e1cc35ca186f855dd4af125a99afa28bf92c
  Stored in directory: pip\cache\wheels\59\f6\9d\85068904dba861c0b9af74e286265a08da438748ee5ae56067
  Building wheel for nltk (setup.py) ... done
  Created wheel for nltk: filename=nltk-3.5-py3-none-any.whl size=1434677 sha256=a2d7879ea0e39222e4ec56fa2793c4256da60c774e4c1ddb45f8c1a8356688c8
  Stored in directory: pip\cache\wheels\45\6c\46\a1865e7ba706b3817f5d1b2ff7ce8996aabdd0d03d47ba0266
  Building wheel for pyDatalog (setup.py) ... done
  Created wheel for pyDatalog: filename=pyDatalog-0.17.1-py2.py3-none-any.whl size=74197 sha256=537a7766fbb9a172e10b2363d373f33855bbb9134aac45024246397ea5ea195d
  Stored in directory: pip\cache\wheels\04\e7\e1\8fa2c8dfe74d0b9e20818c8ec00c20657baa0738976c93e46e
  Building wheel for python-docx (setup.py) ... done
  Created wheel for python-docx: filename=python_docx-0.8.10-py3-none-any.whl size=184489 sha256=d1aee6204eb909a271ee69dc52a9ea4423859489aa39ebf7cdc0954cbf8a0481
  Stored in directory: pip\cache\wheels\75\c6\69\05491f32dc052cd70476b65f5bf7082a9b274045f6b001b821
  Building wheel for sentence-transformers (setup.py) ... done
  Created wheel for sentence-transformers: filename=sentence_transformers-0.4.1.2-py3-none-any.whl size=103066 sha256=2eee4fc5b4566a2e54fa330bcf05cb5c39946934f1a83b0a909b77b98379d6a6
  Stored in directory: pip\cache\wheels\1c\a2\db\5d9e7c4aa8dbd82718c202fb1bc7118c2d3bf0925af92943f2
  Building wheel for sacremoses (setup.py) ... done
  Created wheel for sacremoses: filename=sacremoses-0.0.43-py3-none-any.whl size=893258 sha256=1a11654e208885cc7fa2c6f64fd4d0e3a9cc5c338d1b78dc26282e91ab1ac355
  Stored in directory: pip\cache\wheels\69\09\d1\bf058f7d6fa0ecba2ce7c66be3b8d012beb4bf61a6e0c101c0
Successfully built annoy python-Levenshtein scispacy awscli langdetect nltk pyDatalog python-docx sentence-transformers sacremoses
Installing collected packages: six, zipp, urllib3, typing-extensions, python-dateutil, numpy, murmurhash, jmespath, cymem, wasabi, tqdm, srsly, regex, pyparsing, pyasn1, preshed, plac, joblib, importlib-metadata, idna, click, chardet, botocore, blis, tokenizers, threadpoolctl, thinc, scipy, sacremoses, s3transfer, rsa, requests, PyYAML, pybind11, psutil, packaging, filelock, docutils, colorama, catalogue, transformers, torch, spacy, sentencepiece, scikit-learn, pysbd, nmslib, nltk, lxml, conllu, awscli, sentence-transformers, scispacy, python-Levenshtein, python-docx, pyDatalog, nest-asyncio, langdetect, fuzzywuzzy, annoy, foo
  Running setup.py develop for foo
Successfully installed PyYAML-5.4.1 annoy-1.16.3 awscli-1.19.12 blis-0.4.1 botocore-1.20.12 catalogue-1.0.0 chardet-4.0.0 click-7.1.2 colorama-0.4.3 conllu-4.4 cymem-2.0.5 docutils-0.15.2 filelock-3.0.12 foo fuzzywuzzy-0.15.0 idna-2.10 importlib-metadata-3.4.0 jmespath-0.10.0 joblib-1.0.1 langdetect-1.0.8 lxml-4.6.2 murmurhash-1.0.5 nest-asyncio-1.4.2 nltk-3.5 nmslib-2.1.1 numpy-1.20.1 packaging-20.9 plac-1.1.3 preshed-3.0.5 psutil-5.8.0 pyDatalog-0.17.1 pyasn1-0.4.8 pybind11-2.6.1 pyparsing-2.4.7 pysbd-0.3.4 python-Levenshtein-0.12.0 python-dateutil-2.8.1 python-docx-0.8.10 regex-2020.11.13 requests-2.25.1 rsa-4.5 s3transfer-0.3.4 sacremoses-0.0.43 scikit-learn-0.24.1 scipy-1.6.1 scispacy-0.2.4 sentence-transformers-0.4.1.2 sentencepiece-0.1.85 six-1.15.0 spacy-2.2.3 srsly-1.0.1 thinc-7.3.1 threadpoolctl-2.1.0 tokenizers-0.10.1 torch-1.7.1 tqdm-4.57.0 transformers-4.3.2 typing-extensions-3.7.4.3 urllib3-1.26.3 wasabi-0.8.2 zipp-3.4.0

Actually it's interesting that it installs python-Levenshtein 0.12.0 which has been yanked: https://pypi.org/project/python-Levenshtein/0.12.0/

I guess if the requirements explicitly state that version it gets installed anyway.

@uranusjr
Copy link
Member

Looks like langdetect 1.0.8 has a better dependency tree. Maybe changing it to langdetect<1.0.8 would reproduce. (This is one of the beautiful thing working on this, it's a pain to reproduce anything since there are so many moving parts.)

@notatallshaw
Copy link
Member

notatallshaw commented Feb 20, 2021

Looks like langdetect 1.0.8 has a better dependency tree. Maybe changing it to langdetect<1.0.8 would reproduce. (This is one of the beautiful thing working on this, it's a pain to reproduce anything since there are so many moving parts.)

Indeed, tried adding that but still installed fine. Rereading the initial report it looks like they didn't fully list their requirements as the first item in the log states Collecting Flask==0.12.2 but that's not listed in the requirements they have given. But even using the following:

from setuptools import setup

setup(
    name="foo",
    version="1.0",
    install_requires=[
        'Flask==0.12.2',
        'spacy==2.2.3',
        'sentence-transformers',
        'sentencepiece==0.1.85', 
        'scispacy>=0.2.0',
        'nltk',
        'python-docx',
        'langdetect<1.0.8',
        'fuzzywuzzy==0.15.0',
        'python-Levenshtein==0.12.0',
        'annoy==1.16.3',
        'pyDatalog',
        'srsly==1.0.1',
        'nest_asyncio==1.4.2']
)

It still installed fine, in fact reading the original post further I think there are many more packages that haven't listed, but reverse engineering their requirements from their logs is too laborious for me to try further.

I thought I had a good idea for speeding up these kind of situations, but finding a good test case sure is hard!

@jayqi
Copy link

jayqi commented Feb 25, 2021

I'm encountering this issue on a very small requirements.txt with only two packages:

spacy==2.3.5
spacy-pkuseg

From inspecting their dependencies manually, I can see very clearly that there is an incompatible dependency on srsly:

It looks like pip gets stuck backtracking on a whole bunch of different things. If there is no valid solution and if we don't think pip should install an invalid solution, then it should at least fail faster so I can move on. There are only three versions of spacy-pkguseg to check yet pip goes into a crazy spiral of downloading dozens of versions of other dependencies.

  • pip v21.0.1
  • Python v3.8.8
  • macOs
Expand for logs. I canceled it after 6 minutes.

Collecting spacy==2.3.5
  Using cached spacy-2.3.5-cp38-cp38-macosx_10_9_x86_64.whl (10.2 MB)
Collecting spacy-pkuseg
  Using cached spacy_pkuseg-0.0.28-cp38-cp38-macosx_10_9_x86_64.whl (2.4 MB)
Collecting cymem<2.1.0,>=2.0.2
  Using cached cymem-2.0.5-cp38-cp38-macosx_10_9_x86_64.whl (31 kB)
Collecting numpy>=1.15.0
  Using cached numpy-1.20.1-cp38-cp38-macosx_10_9_x86_64.whl (16.0 MB)
Collecting thinc<7.5.0,>=7.4.1
  Using cached thinc-7.4.5-cp38-cp38-macosx_10_9_x86_64.whl (982 kB)
Collecting wasabi<1.1.0,>=0.4.0
  Using cached wasabi-0.8.2-py3-none-any.whl (23 kB)
Collecting catalogue<1.1.0,>=0.0.7
  Using cached catalogue-1.0.0-py2.py3-none-any.whl (7.7 kB)
Requirement already satisfied: setuptools in /Users/jqi/miniconda3/envs/pip-conflict/lib/python3.8/site-packages (from spacy==2.3.5->-r req-test.txt (line 1)) (49.6.0.post20210108)
Collecting murmurhash<1.1.0,>=0.28.0
  Using cached murmurhash-1.0.5-cp38-cp38-macosx_10_9_x86_64.whl (18 kB)
Collecting plac<1.2.0,>=0.9.6
  Using cached plac-1.1.3-py2.py3-none-any.whl (20 kB)
Collecting tqdm<5.0.0,>=4.38.0
  Using cached tqdm-4.57.0-py2.py3-none-any.whl (72 kB)
Collecting preshed<3.1.0,>=3.0.2
  Using cached preshed-3.0.5-cp38-cp38-macosx_10_9_x86_64.whl (105 kB)
Collecting requests<3.0.0,>=2.13.0
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting blis<0.8.0,>=0.4.0
  Using cached blis-0.7.4-cp38-cp38-macosx_10_9_x86_64.whl (5.8 MB)
Collecting srsly<1.1.0,>=1.0.2
  Using cached srsly-1.0.5-cp38-cp38-macosx_10_9_x86_64.whl (177 kB)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: certifi>=2017.4.17 in /Users/jqi/miniconda3/envs/pip-conflict/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy==2.3.5->-r req-test.txt (line 1)) (2020.12.5)
Collecting spacy-pkuseg
  Using cached spacy_pkuseg-0.0.27-cp38-cp38-macosx_10_9_x86_64.whl (2.0 MB)
  Using cached spacy_pkuseg-0.0.26-cp38-cp38-macosx_10_9_x86_64.whl (2.0 MB)
INFO: pip is looking at multiple versions of wasabi to determine which version is compatible with other requirements. This could take a while.
Collecting wasabi<1.1.0,>=0.4.0
  Using cached wasabi-0.8.1-py3-none-any.whl (23 kB)
  Using cached wasabi-0.8.0-py3-none-any.whl (23 kB)
  Using cached wasabi-0.7.1.tar.gz (22 kB)
  Using cached wasabi-0.7.0.tar.gz (22 kB)
  Using cached wasabi-0.6.0-py3-none-any.whl (20 kB)
  Using cached wasabi-0.5.0-py3-none-any.whl (20 kB)
  Using cached wasabi-0.4.2-py3-none-any.whl (20 kB)
  Using cached wasabi-0.4.1-py3-none-any.whl (20 kB)
  Using cached wasabi-0.4.0-py3-none-any.whl (20 kB)
INFO: pip is looking at multiple versions of wasabi to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
  Using cached urllib3-1.26.1-py2.py3-none-any.whl (136 kB)
  Using cached urllib3-1.26.0-py2.py3-none-any.whl (136 kB)
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
  Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
  Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
  Using cached urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
  Using cached urllib3-1.25.7-py2.py3-none-any.whl (125 kB)
  Using cached urllib3-1.25.6-py2.py3-none-any.whl (125 kB)
  Using cached urllib3-1.25.5-py2.py3-none-any.whl (125 kB)
  Using cached urllib3-1.25.4-py2.py3-none-any.whl (125 kB)
  Using cached urllib3-1.25.3-py2.py3-none-any.whl (150 kB)
  Using cached urllib3-1.25.2-py2.py3-none-any.whl (150 kB)
  Using cached urllib3-1.24.3-py2.py3-none-any.whl (118 kB)
  Using cached urllib3-1.24.2-py2.py3-none-any.whl (131 kB)
  Using cached urllib3-1.24.1-py2.py3-none-any.whl (118 kB)
  Using cached urllib3-1.24-py2.py3-none-any.whl (117 kB)
  Using cached urllib3-1.23-py2.py3-none-any.whl (133 kB)
  Using cached urllib3-1.22-py2.py3-none-any.whl (132 kB)
  Using cached urllib3-1.21.1-py2.py3-none-any.whl (131 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while.
Collecting tqdm<5.0.0,>=4.38.0
  Using cached tqdm-4.56.2-py2.py3-none-any.whl (72 kB)
  Using cached tqdm-4.56.1-py2.py3-none-any.whl (72 kB)
  Using cached tqdm-4.56.0-py2.py3-none-any.whl (72 kB)
  Using cached tqdm-4.55.2-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.55.1-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.55.0-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.54.1-py2.py3-none-any.whl (69 kB)
  Using cached tqdm-4.54.0-py2.py3-none-any.whl (69 kB)
  Using cached tqdm-4.53.0-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.52.0-py2.py3-none-any.whl (71 kB)
  Using cached tqdm-4.51.0-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.50.2-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.50.1-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.50.0-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.49.0-py2.py3-none-any.whl (69 kB)
  Using cached tqdm-4.48.2-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.48.1-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.48.0-py2.py3-none-any.whl (67 kB)
  Using cached tqdm-4.47.0-py2.py3-none-any.whl (66 kB)
  Using cached tqdm-4.46.1-py2.py3-none-any.whl (63 kB)
  Using cached tqdm-4.46.0-py2.py3-none-any.whl (63 kB)
  Using cached tqdm-4.45.0-py2.py3-none-any.whl (60 kB)
  Using cached tqdm-4.44.1-py2.py3-none-any.whl (60 kB)
  Using cached tqdm-4.44.0-py2.py3-none-any.whl (60 kB)
  Using cached tqdm-4.43.0-py2.py3-none-any.whl (59 kB)
  Using cached tqdm-4.42.1-py2.py3-none-any.whl (59 kB)
  Using cached tqdm-4.42.0-py2.py3-none-any.whl (59 kB)
  Using cached tqdm-4.41.1-py2.py3-none-any.whl (56 kB)
  Using cached tqdm-4.41.0-py2.py3-none-any.whl (56 kB)
  Using cached tqdm-4.40.2-py2.py3-none-any.whl (55 kB)
  Using cached tqdm-4.40.1-py2.py3-none-any.whl (55 kB)
  Using cached tqdm-4.40.0-py2.py3-none-any.whl (54 kB)
  Using cached tqdm-4.39.0-py2.py3-none-any.whl (53 kB)
  Using cached tqdm-4.38.0-py2.py3-none-any.whl (53 kB)
INFO: pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of thinc to determine which version is compatible with other requirements. This could take a while.
Collecting thinc<7.5.0,>=7.4.1
  Downloading thinc-7.4.3-cp38-cp38-macosx_10_9_x86_64.whl (981 kB)
     |████████████████████████████████| 981 kB 3.4 MB/s 
  Downloading thinc-7.4.2-cp38-cp38-macosx_10_9_x86_64.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 3.4 MB/s 
  Using cached thinc-7.4.1-cp38-cp38-macosx_10_9_x86_64.whl (2.1 MB)
Collecting blis<0.8.0,>=0.4.0
  Using cached blis-0.4.1-cp38-cp38-macosx_10_9_x86_64.whl (3.7 MB)
^CERROR: Operation cancelled by user

@notatallshaw
Copy link
Member

notatallshaw commented Mar 2, 2021

@uranusjr Maybe I'm being naive but it seems looking at the above example pip should be able to fail / resolve much faster.

Here is the test case:

from setuptools import setup

setup(
    name="foo",
    version="1.0",
    install_requires=[
        'spacy==2.3.5',
        'spacy-pkuseg'
    ]
)

Currently as explained above pip gets stuck here. By hand I can resolve this as impossible by doing the following:

  1. Perform a search on the requirements looking for where only a single version of the package is available.
  2. Collect dependencies of packages where only a single version is available, repeat step 1 until all packages where only a single version available have had their requirements collected.
  3. Check for any conflicts in the set of all dependencies, if there are then the resolution is impossible
  4. Performing a breadth first search on the requirements collected iterate through each package one at a time.
  5. Either:
    5a. Exhaust all versions of a particular package and throw ResolutionImpossible
    5b. Find the more recent version of the package which matches the requirements and continue with step 4 going to the next package in the breadth first search.

In this example it would look like this:

  • spacy==2.3.5 has one version available on PyPi, collect requirements: "murmurhash>=0.28.0,<1.1.0", "cymem>=2.0.2,<2.1.0", "preshed>=3.0.2,<3.1.0", "thinc>=7.4.1,<7.5.0", "blis>=0.4.0,<0.8.0", "wasabi>=0.4.0,<1.1.0", "srsly>=1.0.2,<1.1.0", "catalogue>=0.0.7,<1.1.0", "tqdm>=4.38.0,<5.0.0", "setuptools", "numpy>=1.15.0", "plac>=0.9.6,<1.2.0", "requests>=2.13.0,<3.0.0"
  • spacy-pkuseg has multiple versions available on PyPi, not collecting any requirements yet
  • All of these have multiple versions available on PyPi so not collecting yet: "murmurhash>=0.28.0,<1.1.0", "cymem>=2.0.2,<2.1.0", "preshed>=3.0.2,<3.1.0", "thinc>=7.4.1,<7.5.0", "blis>=0.4.0,<0.8.0", "wasabi>=0.4.0,<1.1.0", "srsly>=1.0.2,<1.1.0", "catalogue>=0.0.7,<1.1.0", "tqdm>=4.38.0,<5.0.0", "setuptools", "numpy>=1.15.0", "plac>=0.9.6,<1.2.0", "requests>=2.13.0,<3.0.0"
  • All packages which have only 1 available version have had their requirements collected, are there any conflicts? No
  • Performing a breadth first search on remaining packages (spacy-pkuseg -> requirements of spacy(murmurhash -> cymem -> ...) -> requirements of spacy-pkuseg (...) -> requirements of murmurhash(...) --> ...)
  • Collecting requirements of latest version of spacy-pkuseg (0.0.28): "cython>=0.25", "numpy>=1.16.0", "srsly>=2.3.0,<3.0.0"
  • Any conflicts? Yes, these 2 are logically impossible: "srsly>=2.3.0,<3.0.0" and "srsly>=1.0.2,<1.1.0"
  • Collecting requirements of next latest version of spacy-pkuseg (0.0.27): "cython", "numpy>=1.16.0", "srsly>=2.3.0,<3.0.0"
  • Any conflicts? Yes, these 2 are logically impossible: "srsly>=2.3.0,<3.0.0" and "srsly>=1.0.2,<1.1.0"
  • Collecting requirements of next latest version of spacy-pkuseg (0.0.26): "cython", "numpy>=1.16.0", "srsly>=2.3.0,<3.0.0"
  • Any conflicts? Yes, these 2 are logically impossible: "srsly>=2.3.0,<3.0.0" and "srsly>=1.0.2,<1.1.0"
  • No more versions of spacy-pkuseg available, resolution impossible!

Would this breadth first search through the requirements decency tree not be a much faster resolution in lots of failure cases? Or am I being naive in the scope of the problem or the complexity to implement?

@uranusjr
Copy link
Member

uranusjr commented Mar 2, 2021

Breadth-first search is definitely an interesting proposal (and IIRC was raised before), but the idea of a “depth” is a bit more complicated past the most simplistic examples because the dependencies usually form a graph instead of a tree. Nowhere technically infeasible (not even that difficult) though, so I’d definitely encourage you to look into the possibility.

The biggest issue to the approach in my mind is not about the implementation, but to somehow know whether it is an improvement. It obviously improves the example here, and likely many others showing a similar failure pattern, but would it worsen cases that run just fine right now? If it does, is the improvement enough to justify the degradation? How are we even supposed to put cases on a scale? Those are the more important questions to answer than the implementation itself.

@notatallshaw
Copy link
Member

notatallshaw commented Mar 2, 2021

Yeah, I'm not sure that the breadth first search would be better in all/most cases. I think the main thing is collecting all packages where there can only be 1 possible version installed and then trying not to be "dumb" in the search methodology through the remaining packages (breadth first was my first idea but I have other ideas).

Understood on the problems you've outlined, it's hard to know if you're making a net positive or a net negative or if you've made certain edge cases 1000x worse. But I have some ideas on testing this as well.

I'll start taking a look at the resolver.resolve and see if I can figure out how to tweak the implementation to test ideas.

@pradyunsg
Copy link
Member

Last I'd tried breadth first search, I'd found the results to be a bit surprising and mote difficult to reason about than depth first search. I also didn't find any package manager that approached the problem that way. That was 2 years ago tho.

@pradyunsg pradyunsg added the S: needs triage Issues/PRs that need to be triaged label Mar 6, 2021
@notatallshaw
Copy link
Member

notatallshaw commented Sep 17, 2021

FYI I'm no longer able to reproduce the issue with:

spacy==2.3.5
spacy-pkuseg

I've got a new optimization I am testing, so if anyone has a reproducible example let me know.

@daniellepintz
Copy link

@notatallshaw which version of pip are you using? I am encountering this issue :/

@pradyunsg
Copy link
Member

pradyunsg commented Sep 24, 2021

@daniellepintz They're working on making improvements to pip's behaviour in this area, and I believe that was a comment about the improved behaviour they're seeing on the local in-development copy of pip that they have.

@notatallshaw
Copy link
Member

@notatallshaw which version of pip are you using? I am encountering this issue :/

Do you have a reproducible set of requirements?

As @pradyunsg mentions I am working on what should be a performance improvement when people encounter this issue, but I am always looking for more reproducible examples to confirm it actually works.

@notatallshaw
Copy link
Member

notatallshaw commented Dec 13, 2021

Someone just posted an example here of azure-cli with Python 3.9 on Linux but then deleted the comment. I tried in a fresh virtual environment but wasn't albe to reproduce.

In general I think this issue should be closed, all reported and reproducible examples were fixed by #10479

New examples should be new issues, and feel free to CC me and I'll check if I can reproduce and see what the issue is.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 14, 2022
@pradyunsg pradyunsg removed the S: needs triage Issues/PRs that need to be triaged label Mar 17, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants