Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't resolve dependencies for Gensim / smart-open / botocore #2867

Closed
whyboris opened this issue Sep 19, 2018 · 10 comments · Fixed by #2935
Closed

Can't resolve dependencies for Gensim / smart-open / botocore #2867

whyboris opened this issue Sep 19, 2018 · 10 comments · Fixed by #2935

Comments

@whyboris
Copy link

whyboris commented Sep 19, 2018

Issue description

pipenv install gensim fails to create Pipfile.lock because of incompatible required versions in smart-open of botocore.

Expected result

It should install without conflict ... somehow

Actual result

pipenv graph gives this:

gensim==3.5.0
  - numpy [required: >=1.11.3, installed: 1.15.1]
  - scipy [required: >=0.18.1, installed: 1.1.0]
    - numpy [required: >=1.8.2, installed: 1.15.1]
  - six [required: >=1.5.0, installed: 1.11.0]
  - smart-open [required: >=1.2.1, installed: 1.7.0]
    - boto [required: >=2.32, installed: 2.49.0]
    - boto3 [required: <1.8.0, installed: 1.7.84]
      - botocore [required: >=1.10.84,<1.11.0, installed: 1.10.84]
        - docutils [required: >=0.10, installed: 0.14]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
          - six [required: >=1.5, installed: 1.11.0]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
        - botocore [required: >=1.3.0,<2.0.0, installed: 1.10.84]
          - docutils [required: >=0.10, installed: 0.14]
          - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
            - six [required: >=1.5, installed: 1.11.0]
    - botocore [required: <1.11.0, installed: 1.10.84]
      - docutils [required: >=0.10, installed: 0.14]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
        - six [required: >=1.5, installed: 1.11.0]
    - bz2file [required: Any, installed: 0.98]
    - requests [required: Any, installed: 2.19.1]
      - certifi [required: >=2017.4.17, installed: 2018.8.24]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.8, installed: 2.7]
      - urllib3 [required: >=1.21.1,<1.24, installed: 1.23]

Specifically, please note these three lines inside smart-open:

- botocore [required: >=1.10.84,<1.11.0,  installed: 1.10.84]
- botocore [required: >=1.3.0,<2.0.0,     installed: 1.10.84]
- botocore [required: <1.11.0,            installed: 1.10.84]

This may be a problem with smart-open rather than pipenv but I can't tell. I posted the issue on smart-open but I don't know enough to decide who needs to adjust what: piskvorky/smart_open#227

Steps to replicate

pipenv install gensim
Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

pipenv --support:

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python3.7'

Other Python installations in PATH:

  • 2.7: /usr/local/bin/python2.7

  • 2.7: /usr/local/bin/python2.7

  • 2.7: /usr/bin/python2.7

  • 3.6: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6m

  • 3.6: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6

  • 3.6: /usr/local/bin/python3.6

  • 3.7: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7m

  • 3.7: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7

  • 3.7: /usr/local/bin/python3.7

  • 2.7.15: /usr/local/bin/python

  • 2.7.10: /usr/bin/python

  • 2.7.15: /usr/local/bin/python2

  • 3.7.0: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3

  • 3.6.4: /Library/Frameworks/Python.framework/Versions/3.6/bin/python3

  • 3.7.0: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT '
                     '2018; root:xnu-4570.71.2~1/RELEASE_X86_64',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

  • PATH
  • TERM_PROGRAM
  • NVM_CD_FLAGS
  • TERM
  • SHELL
  • TMPDIR
  • Apple_PubSub_Socket_Render
  • TERM_PROGRAM_VERSION
  • ZSH
  • NVM_DIR
  • USER
  • SSH_AUTH_SOCK
  • __CF_USER_TEXT_ENCODING
  • PAGER
  • LSCOLORS
  • _
  • PWD
  • LANG
  • XPC_FLAGS
  • XPC_SERVICE_NAME
  • SHLVL
  • HOME
  • LESS
  • LOGNAME
  • LC_CTYPE
  • NVM_BIN
  • DISPLAY
  • COLORTERM
  • PYTHONDONTWRITEBYTECODE
  • PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/local/Cellar/pipenv/2018.7.1/libexec/tools:/usr/local/opt/opencv3/bin:/Users/byakubchik/.nvm/versions/node/v8.11.2/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:~/.composer/vendor/bin:/Library/TeX/texbin:/usr/local/go/bin:/opt/X11/bin
  • SHELL: /bin/zsh
  • LANG: en_US.UTF-8
  • PWD: /Users/byakubchik/Desktop/temp

Contents of Pipfile ('/Users/byakubchik/Desktop/temp/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
gensim = "*"

[dev-packages]

[requires]
python_version = "3.7"
@whyboris
Copy link
Author

It looks like pipenv is doing its best, but the versions are simply incompatible -- two dependencies want two different versions of sub-dependencies:

smart_open wants

        'boto3 < 1.8.0',
        'botocore < 1.11.0'

https://github.com/RaRe-Technologies/smart_open/blob/master/setup.py#L54

While boto3 wants

    'botocore>=1.12.6,<1.13.0',

https://github.com/boto/boto3/blob/develop/setup.py#L17

@whyboris
Copy link
Author

Looks like the issue will be resolved on the smart-open end: piskvorky/smart_open#227 (comment)

I believe that npm and yarn manage to handle different version requirements somehow (e.g. package A wants sub-dependency C at version 1.0 while package B wants sub-dependency C at version 2.0 -- and they make it work in the same project). But this is a tall order, possibly enormously difficult to have pipenv install two different versions and hook up them up to corresponding packages (don't know enough inner workings here). 😅

@petergaultney
Copy link

@whyboris - what version of Python are you requiring in your Pipfile?

I seem to be seeing this issue only with a Python 3.6 venv, whereas in a Python 3.7 venv, everything works fine. I'm curious if this is reproducible.

pipenv install gensim --python 3.6 (in an empty directory) fails, whereas pipenv install gensim --python 3.7 (in an empty directory) succeeds.

For my use case, I have to have 3.6 for AWS, but it's very interesting that it seems to work in 3.7.

@whyboris
Copy link
Author

The issue was fixed very swiftly at smart_open with a new release that unpins the required dependency.

piskvorky/smart_open#227 (comment)

I believe the issue I raised is an instance of an inevitable problem when package A wants sub-dependency C at version 1.0 while package B wants sub-dependency C at version 2.0 -- I'm rather surprised you didn't run into a problem.

@petergaultney - I was unable to replicate the error now that the package has been updated, but I too am curious why you experienced different behavior between the two versions of Python.

@petergaultney
Copy link

i can still replicate it with pipenv install smart_open==1.7.0 gensim --python 3.6 in an empty directory.

@whyboris
Copy link
Author

whyboris commented Sep 19, 2018

@petergaultney -- thanks for that command -- I was able to replicate the problem with
pipenv install smart_open==1.7.0 gensim --python 3.6
but the problem occurred with python 3.7 too in an empty directory:
pipenv install smart_open==1.7.0 gensim --python 3.7
I suspect what you experienced before was a fluke 🤷‍♂️ ?

@petergaultney
Copy link

Thanks for trying. It doesn't make any sense to me, but weirdly enough I can still get pipenv install smart_open==1.7.0 gensim --python 3.7 to pass cleanly, simply running in an empty directory. A ghost in the machine, I suppose? It doesn't enhance my confidence in the Python packaging ecosystem, unfortunately...

screenshot 2018-09-19 12 02 51

@uranusjr
Copy link
Member

Try pipenv lock —clear to purge the resolution cache.

@petergaultney
Copy link

that does indeed cause the error as expected. Are there different caches per Python version?

@uranusjr
Copy link
Member

Yes, the cache is partitioned by version.

techalchemy added a commit that referenced this issue Oct 7, 2018
- Fixes #2499
- Fixes #2529
- Fixes #2589
- Fixes #2666
- Fixes #2767
- Fixes #2785
- Fixes #2795
- Fixes #2801
- Fixes #2802
- Fixes #2824
- Fixes #2862
- Fixes #2867
- Fixes #2879
- Fixes #2880
- Fixes #2894
- Fixes #2902
- Fixes #2924

Signed-off-by: Dan Ryan <dan@danryan.co>
techalchemy added a commit that referenced this issue Oct 7, 2018
- Fixes #2499
- Fixes #2529
- Fixes #2589
- Fixes #2666
- Fixes #2767
- Fixes #2785
- Fixes #2795
- Fixes #2801
- Fixes #2802
- Fixes #2824
- Fixes #2862
- Fixes #2867
- Fixes #2879
- Fixes #2880
- Fixes #2894
- Fixes #2902
- Fixes #2924

Signed-off-by: Dan Ryan <dan@danryan.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants