Pipenv's dependency resolution fails to resolve botocore version, while pip succeeds #4781

ltn100 · 2021-09-02T09:55:38Z

Issue description

Using this Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
dvc = {extras = ["s3"], version = "*"}
awscli = "*"

[dev-packages]

[requires]
python_version = "3.8"

Running pipenv install fails to resolve the dependencies for botocore:

$ pipenv --version
pipenv, version 2021.5.29
$ ls
Pipfile
$ pipenv install
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
[ResolutionFailure]:   File "/Users/lnethert/.pyenv/versions/3.8.9/lib/python3.8/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[...]
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches botocore<1.20.107,<2.0a.0,==1.21.34,>=1.12.36,>=1.20.106 (from awscli==1.20.34->-r /var/folders/vb/qxkdpv5s5bvbrcs92zvq8kljdfzp9p/T/pipenvqs__kzo5requirements/pipenv-_6pq4hm2-constraints.txt (line 2))
Tried: 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, [...], 1.21.34
Skipped pre-versions: 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0rc1, 1.0.0rc1
There are incompatible versions in the resolved dependencies:
  botocore<1.20.107,>=1.20.106 (from aiobotocore[boto3]==1.4.1->dvc[s3]==2.6.4->-r /var/folders/vb/qxkdpv5s5bvbrcs92zvq8kljdfzp9p/T/pipenvqs__kzo5requirements/pipenv-_6pq4hm2-constraints.txt (line 3))
  botocore<2.0a.0,>=1.12.36 (from s3transfer==0.5.0->awscli==1.20.34->-r /var/folders/vb/qxkdpv5s5bvbrcs92zvq8kljdfzp9p/T/pipenvqs__kzo5requirements/pipenv-_6pq4hm2-constraints.txt (line 2))
  botocore==1.21.34 (from awscli==1.20.34->-r /var/folders/vb/qxkdpv5s5bvbrcs92zvq8kljdfzp9p/T/pipenvqs__kzo5requirements/pipenv-_6pq4hm2-constraints.txt (line 2))

However, the dependencies are resolved successfully using pip:

$ python -m virtualenv venv
created virtual environment CPython3.8.9.final.0-64 in 403ms
$ source venv/bin/activate
$ pip --version
pip 21.1.3
$ pip install "dvc[s3]" awscli
Collecting dvc[s3]
  Using cached dvc-2.6.4-py3-none-any.whl (644 kB)
Collecting awscli
[...]
Successfully installed [...] awscli-1.20.34 [...] botocore-1.21.34 [...] dvc-2.1.0 [...]

The dependency resolution is complicated by the fact that dvc[s3] is dependant on aiobotocore, which is in turn dependant on a very specific version of botocore. Meanwhile, awscli versions are also dependent on very specific versions of botocore. However, pip is able to resolve a valid set of dependencies, so the task is not an impossible one.

$ pipenv --support

Pipenv version: '2021.5.29'

Pipenv location: '/Users/lnethert/.pyenv/versions/3.8.9/lib/python3.8/site-packages/pipenv'

Python location: '/Users/lnethert/.pyenv/versions/3.8.9/bin/python3.8'

Python installations found:

3.9.6: /usr/local/bin/python3
3.9.6: /usr/local/bin/python3.9
3.9.1: /Users/lnethert/.pyenv/versions/3.9.1/bin/python3
3.8.9: /Users/lnethert/.pyenv/versions/3.8.9/bin/python3
3.8.9: /Users/lnethert/.pyenv/versions/3.8.9/bin/python3
3.8.9: /Users/lnethert/.pyenv/versions/3.8.9/bin/python3.8
3.8.3: /Users/lnethert/.pyenv/versions/3.8.3/bin/python3
3.7.3: /usr/bin/python3
2.7.16: /usr/bin/python2
2.7.16: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.9',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '19.2.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 19.2.0: Sat Nov  9 03:47:04 PST '
                     '2019; root:xnu-6153.61.1~20/RELEASE_X86_64',
 'python_full_version': '3.8.9',
 'python_version': '3.8',
 'sys_platform': 'darwin'}

System environment variables:

TERM_PROGRAM
PYENV_ROOT
TERM
SHELL
CLICOLOR
HISTSIZE
TMPDIR
TERM_PROGRAM_VERSION
TF_SHELL
TERM_SESSION_ID
PYENV_VERSION
TF_ALIAS
ZSH
TF_SHELL_ALIASES
PYTHONIOENCODING
HISTFILESIZE
USER
SSH_AUTH_SOCK
PYENV_DIR
__CF_USER_TEXT_ENCODING
PAGER
LSCOLORS
PATH
PWD
EDITOR
DEFAULT_USER
LANG
ITERM_PROFILE
PYENV_HOOK_PATH
XPC_FLAGS
HISTCONTROL
XPC_SERVICE_NAME
PYENV_SHELL
SHLVL
HOME
COLORFGBG
LC_TERMINAL_VERSION
ITERM_SESSION_ID
LESS
LOGNAME
HH_CONFIG
LC_TERMINAL
HISTTIMEFORMAT
COLORTERM
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /Users/lnethert/.pyenv/versions/3.8.9/bin:/usr/local/Cellar/pyenv/2.0.3/libexec:/usr/local/Cellar/pyenv/2.0.3/plugins/python-build/bin:/Users/lnethert/.bin/:/Users/lnethert/Library/Python/3.7/bin:/Users/lnethert/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
SHELL: /bin/zsh
EDITOR: vim
LANG: en_GB.UTF-8
PWD: /Users/lnethert/test

Contents of Pipfile ('/Users/lnethert/workspace/aig_app_science/sonar_nlp/test/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
dvc = {extras = ["s3"], version = "*"}
awscli = "*"

[dev-packages]

[requires]
python_version = "3.8"

The text was updated successfully, but these errors were encountered:

franzhaas · 2021-09-06T10:54:47Z

I had a similar problem with executable book. I use pip as a resolution engine for pipenv now.

Not only because it worls, but also to be compatible with my colleagues...

Maybe you want to have a look on the suggestion

#4777

matteius · 2021-12-24T09:43:16Z

@ltn100 Well you seem to have created a Pipfile that really never finishes locking on the newest version 2021.11.23 and latest pip 21.3.1 ... not sure what to make of this one.

matteius · 2022-03-13T04:24:14Z

Well in one of the upcoming pipenv releases that will switch to pip 22.x from 21.x that branch it fails fairly quick:

matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-4781$ pipenv install
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

ERROR:pip.subprocessor:[present-diagnostic] python setup.py egg_info exited with 1
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 743, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 704, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 685, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 1376, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 1105, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 883, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: metadata generation failed

The --pre flag also fails for your example. Going to play around here based on what I know that pinning some things to "*" causes the pip resolver to take much longer -- plus I am not familiar with the syntax for dvc = {extras = ["s3"], version = "*"} Specifically the extras part.

matteius · 2022-03-13T04:35:22Z

I think you already figured it out because you said

The dependency resolution is complicated by the fact that dvc[s3] is dependant on aiobotocore, which is in turn dependant on a very specific version of botocore. Meanwhile, awscli versions are also dependent on very specific versions of botocore. However, pip is able to resolve a valid set of dependencies, so the task is not an impossible one.

Can you specify what version of pip you are using? I think the matter is that when I run it in either order, both dependencies get installed, but locking fails because locking has to be deterministic and dependencies can't violate the constraints in the way you mention.

matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-4781$ pipenv install awscli
Creating a virtualenv for this project...
Pipfile: /home/matteius/shared-projects/pipenv-triage/pipenv-4781/Pipfile
Using /home/matteius/.pyenv/versions/3.10.2/bin/python3 (3.10.2) to create virtualenv...
⠇ Creating virtual environment...created virtual environment CPython3.10.2.final.0-64 in 268ms
  creator CPython3Posix(dest=/home/matteius/.virtualenvs/pipenv-4781-BgvsIKiN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/matteius/.local/share/virtualenv)
    added seed packages: pip==22.0.3, setuptools==60.9.3, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/matteius/.virtualenvs/pipenv-4781-BgvsIKiN
Creating a Pipfile for this project...
Installing awscli...
Adding awscli to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (459756)!
Installing dependencies from Pipfile.lock (459756)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-4781$ pipenv install install dvc[s3]
Installing install...
Adding install to Pipfile's [packages]...
✔ Installation Succeeded 
Installing dvc[s3]...
Adding dvc to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (459756) out of date, updating to (f562f5)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

ERROR:pip.subprocessor:[present-diagnostic] python setup.py egg_info exited with 1
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 743, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 704, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 685, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 1376, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 1105, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 883, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: metadata generation failed

However reversing the order of that, I can start fresh and install dvc[s3]

matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-4781$ pipenv install install dvc[s3]
Creating a virtualenv for this project...
Pipfile: /home/matteius/shared-projects/pipenv-triage/pipenv-4781/Pipfile
Using /home/matteius/.pyenv/versions/3.10.2/bin/python3 (3.10.2) to create virtualenv...
⠇ Creating virtual environment...created virtual environment CPython3.10.2.final.0-64 in 254ms
  creator CPython3Posix(dest=/home/matteius/.virtualenvs/pipenv-4781-BgvsIKiN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/matteius/.local/share/virtualenv)
    added seed packages: pip==22.0.3, setuptools==60.9.3, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/matteius/.virtualenvs/pipenv-4781-BgvsIKiN
Creating a Pipfile for this project...
Installing install...
Adding install to Pipfile's [packages]...
✔ Installation Succeeded 
Installing dvc[s3]...
Adding dvc to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (855571)!
Installing dependencies from Pipfile.lock (855571)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-4781$ pipenv install awscli
Installing awscli...
Adding awscli to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (855571) out of date, updating to (f562f5)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

ERROR:pip.subprocessor:[present-diagnostic] python setup.py egg_info exited with 1
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 743, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 704, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/resolver.py", line 685, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 1376, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 1105, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/home/matteius/pipenv/pipenv/utils.py", line 883, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: metadata generation failed

matteius · 2022-03-13T04:44:11Z

@ltn100 I wonder which version of botocore you endup with when you install using pip. Maybe depends on the order in which you install? I suspect both of those packages are capable of working with multiple versions of botocore is what you find, but their stringent setup.py requirements are what is causing locking to fail, and rightfully so. Basically awscli requires botocore==1.21.34 and aiobotocore requires botocore==1.20.106 -- that is impossible to resolve.

matteius · 2022-03-13T04:45:19Z

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipenv's dependency resolution fails to resolve botocore version, while pip succeeds #4781

Pipenv's dependency resolution fails to resolve botocore version, while pip succeeds #4781

ltn100 commented Sep 2, 2021 •

edited

Loading

franzhaas commented Sep 6, 2021 •

edited

Loading

matteius commented Dec 24, 2021

matteius commented Mar 13, 2022

matteius commented Mar 13, 2022

matteius commented Mar 13, 2022 •

edited

Loading

matteius commented Mar 13, 2022

Pipenv's dependency resolution fails to resolve botocore version, while pip succeeds #4781

Pipenv's dependency resolution fails to resolve botocore version, while pip succeeds #4781

Comments

ltn100 commented Sep 2, 2021 • edited Loading

Issue description

franzhaas commented Sep 6, 2021 • edited Loading

matteius commented Dec 24, 2021

matteius commented Mar 13, 2022

matteius commented Mar 13, 2022

matteius commented Mar 13, 2022 • edited Loading

matteius commented Mar 13, 2022

ltn100 commented Sep 2, 2021 •

edited

Loading

franzhaas commented Sep 6, 2021 •

edited

Loading

matteius commented Mar 13, 2022 •

edited

Loading