Unexpected Dependancy Graph Issue

[JBKahn]

Issue description

I had a requirement pinned to "<4.0.0" and a sub-dependancy that required it to be "==3.3.0". I checked the graph and as expected, that was the only dependancy that required the protobuf package.

That gave me a dependancy resolution error of:
Could not find a version that matches protobuf<4.0.0,==3.3.0,==3.6.0

Expected result

I expected it to resolve to "==3.3.0"

Actual result

I was able to get it to be happy by changing my local "<4.0.0" to "==3.3.0".

Steps to replicate

I will fill this is later today, but wanted to spell out the issue just in case it is known.

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

If you're on macOS, run the following:

$ pipenv --support | pbcopy

If you're on Windows, run the following:

> pipenv --support | clip

[erinxocon]

Hey @JBKahn Any update on this?

[JBKahn]

I had been slammed at work, it seems to be possibly related to my build environment as this didn't happen on my machine but inside an alpine python docker container. Was going to dig in before I included all the details.

[teeberg]

I think I'm experiencing the same issue with a different package:

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches botocore<1.11.0,==1.10.26,>=1.10.26,>=1.10.64 (from -r /var/folders/h4/n2jm49b97dx5d89gkkt4yxkm0000gn/T/pipenv-KF21xr-requirements/pipenv-Vk6UhE-constraints.txt (line 34))

and it's true, ==1.10.26 and >=1.10.64 contradict themselves, but no package seems to be pulling in >=1.10.64?!

After running the suggested pipenv install --skip-lock and inspecting the output of pipenv graph, here are botocore's dependency specs:

> pipenv graph | grep -i botocore
  - botocore [required: ==1.10.26, installed: 1.10.26]
    - botocore [required: >=1.3.0,<2.0.0, installed: 1.10.26]
  - botocore [required: >=1.0.1, installed: 1.10.26]
    - botocore [required: >=1.10.26,<1.11.0, installed: 1.10.26]
      - botocore [required: >=1.3.0,<2.0.0, installed: 1.10.26]

I'm just investigating how much work it is to switch over to using pipenv from a workflow using pip-compile, so I have another virtual env set up with exactly these requirements as specified, and using pipdeptree in there doesn't show anything specifying >=1.10.64 either:

> pipdeptree -r -p botocore | grep "requires: botocore"
  - awscli==1.15.26 [requires: botocore==1.10.26]
  - awsebcli==3.14.3 [requires: botocore>=1.0.1]
  - boto3==1.7.26 [requires: botocore>=1.10.26,<1.11.0]
  - s3transfer==0.1.12 [requires: botocore>=1.3.0,<2.0.0]

So, maybe this comes down to, how do I find out where that bogus version spec comes from?

[JBKahn]

yeah, the real difficulty is that when there is a collision, the output isn't showing where the different requirements come from. I've been super busy these past few days but I'll try to get to this ASAP.

[mr-bo-jangles]

yeah, the real difficulty is that when there is a collision, the output isn't showing where the different requirements come from

Literally dealing with this right now, it's impossible to track down which package is causing issues

[mr-bo-jangles]

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/home/josh/.virtualenvs/savings-champion/lib/python3.6/site-packages/pipenv'

Python location: '/home/josh/.virtualenvs/savings-champion/bin/python'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/bin/python3.6m

• 3.6: /home/josh/.virtualenvs/savings-champion/bin/python3.6

• 3.6: /usr/bin/python3.6

• 3.6.6: /home/josh/.virtualenvs/savings-champion/bin/python

• 3.6.6: /usr/bin/python

• 2.7.15: /usr/bin/python2

• 3.6.6: /home/josh/.virtualenvs/savings-champion/bin/python3

• 3.6.6: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.17.11-arch1',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Sun Jul 29 10:11:16 UTC 2018',
 'python_full_version': '3.6.6',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• PATH
• XAUTHORITY
• GDMSESSION
• XDG_DATA_DIRS
• ZDOTDIR
• KDE_SESSION_UID
• TERM
• DBUS_SESSION_BUS_ADDRESS
• LANG
• XDG_SESSION_TYPE
• XDG_SESSION_ID
• XDG_CURRENT_DESKTOP
• QT_AUTO_SCREEN_SCALE_FACTOR
• DISPLAY
• MAIL
• MOZ_PLUGIN_PATH
• SESSION_MANAGER
• USERNAME
• LOGNAME
• PWD
• _
• XCURSOR_THEME
• LANGUAGE
• KDE_SESSION_VERSION
• SHELL
• GDM_LANG
• XCURSOR_SIZE
• DESKTOP_SESSION
• OLDPWD
• USER
• DESKTOP_STARTUP_ID
• KDE_FULL_SESSION
• WINDOWPATH
• XDG_SEAT
• GTK_MODULES
• GS_LIB
• LC_ALL
• KWIN_TRIPLE_BUFFER
• XDG_RUNTIME_DIR
• XDG_SESSION_DESKTOP
• XDG_VTNR
• SHLVL
• HOME
• EDITOR
• VISUAL
• PAGER
• LESS
• LESS_TERMCAP_mb
• LESS_TERMCAP_md
• LESS_TERMCAP_me
• LESS_TERMCAP_se
• LESS_TERMCAP_so
• LESS_TERMCAP_ue
• LESS_TERMCAP_us
• LS_COLORS
• GREP_COLOR
• GREP_COLORS
• VIRTUAL_ENV
• PS1
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/josh/.virtualenvs/savings-champion/bin:/usr/local/bin:/usr/local/sbin:/home/josh/go/bin:/home/josh/.yarn/bin:/usr/bin:/opt/cuda/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
• SHELL: /usr/bin/zsh
• EDITOR: nano
• LANG: en_GB.UTF-8
• PWD: /home/josh/PycharmProjects/savings-champion
• VIRTUAL_ENV: /home/josh/.virtualenvs/savings-champion

---

Contents of Pipfile ('/home/josh/PycharmProjects/savings-champion/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
lxml = "*"
nose = "*"
django-extensions = "*"
django-tinymce = "*"
sorl-thumbnail = "*"
"suds-py3" = "*"
feedparser = "*"
python-dateutil = "*"
python-memcached = "*"
django-pipeline = "*"
requests = "*"
django-pipeline-forgiving = "*"
django-ckeditor = "*"
django-filebrowser = "*"
raven = "*"
celery = {version = "*", extras = ["redis"]}
"psycopg2-binary" = "*"
django-crispy-forms = "*"
pyotp = "*"
newrelic = "*"
djangorestframework = "*"
djangorestframework-xml = "*"
djangorestframework-jwt = "*"
django-filter = "*"
parsedatetime = "*"
simple-salesforce = "*"
django-allauth = "*"
apache-libcloud = "*"
django-storages = "*"
"boto3" = "*"
sortedcontainers = "*"
redis = "*"
hiredis = "*"
msgpack-python = "*"
django-waffle = "*"
django-report-builder = "*"
django-markupfield = "*"
django-cors-headers = "*"
arrow = "*"
django-redis = "*"
"mailchimp3" = "*"
dotmailer = {editable = true, git = "https://github.com/savingschampion/dotmailer.git"}
simplejson = "*"
zeep = "*"
djangorestframework-jsonp = "*"
requests-oauthlib = "*"
django-ipware = "*"
hypothesis = {version = "*", extras = ["pytz"]}
drf-yasg = {version = "*", extras = ["validation"]}
Django = ">=2.0"
Pillow = "*"
uWSGI = "*"
"Django-Select2" = "*"
Collectfast = "*"
Markdown = "*"
Faker = "*"
rest_condition = "*"
django-rest-knox = "*"
django-simple-history = "*"
wagtail = "*"
wagtail-metadata-mixin = "*"

[dev-packages]

[requires]
python_version = "3.6"

Locking [dev-packages] dependencies...
Locking [packages] dependencies...

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches Django<2.1,==2.1,>=1.11,>=1.11.7,>=1.3,>=1.8,>=2.0 (from -r /tmp/pipenv-0s0uam0c-requirements/pipenv-7uweinlt-constraints.txt (line 51))
Tried: 1.1.3, 1.1.4, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.5, 1.5.1, 1.5.2, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.5.12, 1.6, 1.6, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.6.5, 1.6.5, 1.6.6, 1.6.6, 1.6.7, 1.6.7, 1.6.8, 1.6.8, 1.6.9, 1.6.9, 1.6.10, 1.6.10, 1.6.11, 1.6.11, 1.7, 1.7, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.7.4, 1.7.4, 1.7.5, 1.7.5, 1.7.6, 1.7.6, 1.7.7, 1.7.7, 1.7.8, 1.7.8, 1.7.9, 1.7.9, 1.7.10, 1.7.10, 1.7.11, 1.7.11, 1.8, 1.8, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.8.3, 1.8.3, 1.8.4, 1.8.4, 1.8.5, 1.8.5, 1.8.6, 1.8.6, 1.8.7, 1.8.7, 1.8.8, 1.8.8, 1.8.9, 1.8.9, 1.8.10, 1.8.10, 1.8.11, 1.8.11, 1.8.12, 1.8.12, 1.8.13, 1.8.13, 1.8.14, 1.8.14, 1.8.15, 1.8.15, 1.8.16, 1.8.16, 1.8.17, 1.8.17, 1.8.18, 1.8.18, 1.8.19, 1.8.19, 1.9, 1.9, 1.9.1, 1.9.1, 1.9.2, 1.9.2, 1.9.3, 1.9.3, 1.9.4, 1.9.4, 1.9.5, 1.9.5, 1.9.6, 1.9.6, 1.9.7, 1.9.7, 1.9.8, 1.9.8, 1.9.9, 1.9.9, 1.9.10, 1.9.10, 1.9.11, 1.9.11, 1.9.12, 1.9.12, 1.9.13, 1.9.13, 1.10, 1.10, 1.10.1, 1.10.1, 1.10.2, 1.10.2, 1.10.3, 1.10.3, 1.10.4, 1.10.4, 1.10.5, 1.10.5, 1.10.6, 1.10.6, 1.10.7, 1.10.7, 1.10.8, 1.10.8, 1.11, 1.11, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.11.4, 1.11.4, 1.11.5, 1.11.5, 1.11.6, 1.11.6, 1.11.7, 1.11.7, 1.11.8, 1.11.8, 1.11.9, 1.11.9, 1.11.10, 1.11.10, 1.11.11, 1.11.11, 1.11.12, 1.11.12, 1.11.13, 1.11.13, 1.11.14, 1.11.14, 1.11.15, 1.11.15, 2.0, 2.0, 2.0.1, 2.0.1, 2.0.2, 2.0.2, 2.0.3, 2.0.3, 2.0.4, 2.0.4, 2.0.5, 2.0.5, 2.0.6, 2.0.6, 2.0.7, 2.0.7, 2.0.8, 2.0.8, 2.1, 2.1
Skipped pre-versions: 1.8a1, 1.8b1, 1.8b2, 1.8rc1, 1.9a1, 1.9b1, 1.9rc1, 1.9rc2, 1.10a1, 1.10a1, 1.10b1, 1.10b1, 1.10rc1, 1.10rc1, 1.11a1, 1.11b1, 1.11rc1, 1.11rc1, 2.0a1, 2.0b1, 2.0rc1, 2.1a1, 2.1b1, 2.1rc1
There are incompatible versions in the resolved dependencies.

I have no idea where the ==2.1 comes from, listing of all requirements from graph with Django in the name

  - Django [required: >=1.8, installed: 2.0.7]
  - django-storages [required: >=1.6, installed: 1.6.6]
django-allauth==0.36.0
  - Django [required: >=1.11, installed: 2.0.7]
django-ckeditor==5.6.1
  - django-js-asset [required: Any, installed: 1.1.0]
django-cors-headers==2.4.0
django-crispy-forms==1.7.2
django-extensions==2.1.0
django-filebrowser==3.10.1
  - django-grappelli [required: >=2.11,<2.12, installed: 2.11.1]
django-filter==2.0.0
  - Django [required: >=1.11, installed: 2.0.7]
django-ipware==2.1.0
django-markupfield==1.4.3
django-pipeline-forgiving==1.0.0
  - django-pipeline [required: Any, installed: 1.6.14]
django-redis==4.9.0
  - Django [required: >=1.11, installed: 2.0.7]
django-report-builder==6.0.2
  - djangorestframework [required: >=3.1.0, installed: 3.8.2]
django-rest-knox==3.1.5
  - django [required: Any, installed: 2.0.7]
  - djangorestframework [required: Any, installed: 3.8.2]
django-select2==6.1.1
  - django-appconf [required: >=0.6.0, installed: 1.0.2]
django-simple-history==2.3.0
django-tinymce==2.7.0
django-url-filter==0.3.5
  - Django [required: >=1.8, installed: 2.0.7]
django-waffle==0.14.0
djangorestframework-jsonp==1.0.2
djangorestframework-jwt==1.11.0
djangorestframework-xml==1.3.0
  - Django [required: >=1.11.7, installed: 2.0.7]
  - djangorestframework [required: >=3.7.7, installed: 3.8.2]
  - django [required: >=1.3, installed: 2.0.7]
  - djangorestframework [required: Any, installed: 3.8.2]
  - django-meta [required: >=1.0, installed: 1.4.1]
    - Django [required: >=1.11,<2.1, installed: 2.0.7]
    - django-modelcluster [required: >=4.0,<5.0, installed: 4.1]
    - django-taggit [required: >=0.22.2,<1.0, installed: 0.22.2]
    - django-treebeard [required: >=4.2.0,<5.0, installed: 4.3]
      - Django [required: >=1.8, installed: 2.0.7]
    - djangorestframework [required: >=3.7.4,<4.0, installed: 3.8.2]

[uranusjr]

It’s from Wagtail, you need to pin django<2.1. (I know this because we are working on this right now and this can be pulled out from the half-down code.)

And yes, I know the resolver should be able to pin this automatically, but right now it’s what it is.

[JBKahn]

Is it easy to write tests for the resolver. I haven't taken a look at the code but I'd help out if it isn't too bad to get started.

[OliverHofkens]

I'm running into the same issue with boto3:

$ pipenv lock
Could not find a version that matches botocore<1.11.0,<1.12.0,>=1.10.84,>=1.11.3

Running pipenv install --skip-lock and inspecting the graph shows that no package seems to require <1.11.0

$ pipenv graph | grep botocore
    - botocore [required: >=1.11.3,<1.12.0, installed: 1.11.3]
      - botocore [required: >=1.3.0,<2.0.0, installed: 1.11.3]
        - botocore [required: >=1.11.3,<1.12.0, installed: 1.11.3]
          - botocore [required: >=1.3.0,<2.0.0, installed: 1.11.3]

[mjdunn]

@OliverHofkens see #2596. Running pipenv lock --clear helped me.

[OliverHofkens]

@mjdunn Great, thanks!

[teeberg]

still seeing this, but with a different package now:

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches html5lib==1.0b10,>=1.0
Tried: 1.0-reupload, 0.2, 0.9, 0.10, 0.11, 0.11.1, 0.90, 0.95, 0.99, 0.999, 0.9999, 0.99999, 0.999999, 0.9999999, 0.99999999, 0.999999999, 0.999999999, 1.0.1, 1.0.1
Skipped pre-versions: 1.0b1, 1.0b2, 1.0b3, 1.0b5, 1.0b6, 1.0b7, 1.0b8, 1.0b9, 1.0b10, 1.0b10
There are incompatible versions in the resolved dependencies.

but according to pipdeptree | grep -i html5lib, nothing pulls in >=1.0:

    - html5lib [required: ==1.0b10, installed: 1.0b10]
  - html5lib [required: Any, installed: 1.0b10]

[techalchemy]

With which version of pipenv

[teeberg]

sorry, missed that! in 2018.10.9. I think that's the latest, right?