-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipenv installs incorrect version of package #2088
Comments
This actually should be pretty simple--
That's what our own tests do. Our travis file looks like this: sudo: false
dist: trusty
language: python
python:
- "3.6"
- "2.7"
env:
global:
- PYPI_VENDOR_DIR='./tests/pypi/'
- GIT_ASK_YESNO='false'
- PYTHONIOENCODING='utf-8'
matrix:
- TEST_SUITE='not install'
- TEST_SUITE='install'
PYTEST_ADDOPTS='--cache-clear'
install:
- "pip install --upgrade pip"
- "pip install -e . --upgrade --upgrade-strategy=only-if-needed"
- "pipenv install --dev"
- 'pip install -e "$(pwd)" --upgrade'
- 'pipenv install --system --dev'
script:
- 'pipenv run time pytest -v -n 4 -m "$TEST_SUITE" tests'
sudo: false
dist: trusty
language: python
python:
- "3.6"
- "2.7"
env:
global:
- PYPI_VENDOR_DIR='./tests/pypi/'
- GIT_ASK_YESNO='false'
- PYTHONIOENCODING='utf-8'
matrix:
- TEST_SUITE='not install'
- TEST_SUITE='install'
PYTEST_ADDOPTS='--cache-clear'
install:
- "pip install --upgrade pip"
- "pip install -e . --upgrade --upgrade-strategy=only-if-needed"
- "pipenv install --dev"
- 'pip install -e "$(pwd)" --upgrade'
- 'pipenv install --system --dev'
script:
- 'pipenv run time pytest -v -n 4 -m "$TEST_SUITE" tests' Let me know if this helps! |
Thanks for responding so quickly. I tried adding the suggested env vars and explicitly running python, but am still getting the same result: https://travis-ci.org/dancerfly/django-brambling/builds/372783297 In case it's relevant, it doesn't seem like either suggested env var is used in your |
I can also now confirm that if I limit the dependencies to just django, the installation works fine https://travis-ci.org/dancerfly/django-brambling/builds/372787360 |
I get the correct version of django as long as I don't install either of the git dependencies. |
Minimal pipfile to get this error case on travis:
Installs django 1.11.12 |
I see what's going on here, it's prioritizing the resolution of the VCS dependencies (we don't typically recommend listing non-top-level dependencies like django unless it's absolutely necessary) -- in your case I think you should take these out of editable mode -- I can see that the second dependency needs Basically this will stop the resolver from attempting to find the dependencies of the git-referenced packages and putting them in the lockfile. The other option is to just put these in Let me know if that works |
FYI the most idiomatically correct way to handle this is to put your actual top level requirements in your python_requires="<3.0",
install_requires=[
'bleach==1.4.2',
'django~=1.8.11',
'dj-database-url==0.4.2',
'django-countries==3.4.1',
'django-daguerre==2.1.2',
'django-filter==0.12.0',
'django-grappelli==2.7.3',
'django-libsass==0.4',
'django-localflavor==1.1',
'djangorestframework==3.3.0',
'dwolla==2.0.7',
'factory-boy==2.4.1',
'gunicorn==19.7.1',
'Markdown==2.6.5',
'openpyxl==2.2.6',
'psycopg2==2.7.4',
'sendgrid-django==2.0.0',
'stripe==1.35.0',
'unicode-slugify==0.1.3',
'unicodecsv==0.14.1',
'vcrpy==1.7.4'
] And then you can put in your pipfile: [[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
django-bootstrap = {git = "https://github.com/littleweaver/django-bootstrap.git", editable = true, ref = "bootstrap3"}
django-zenaida = {git = "git://github.com/littleweaver/django-zenaida.git", editable = true, ref = "master"}
django-brambling = {path = ".", editable = true}
[dev-packages]
tox = "==1.8.1"
django-debug-toolbar-template-timings = "==0.6.4"
django-debug-toolbar = "*"
[requires]
python_version = "2.7" Note that whatever you put last gets resolved last and therefore 'wins out' in that section if there is a conflict. |
@techalchemy Thanks again for your quick response! :-) The django-brambling repository is a project repository, not a reusable package. We use it to deploy our servers, so it's imperative that the Pipfile.lock is honored, even if we have git dependencies. In this case, our pipfile.lock correctly listed django 1.8.11 as the locked version, but django 1.11.12 was installed despite that. More generally, I would also expect that explicitly stating a requirement for a particular version of a package would have precedence over all subdependencies of packages I might list, no matter what order things are listed in. I'm not quite clear from your responses in this thread whether these are acknowledged as bugs or if they're seen as user error. It's pretty clear in my mind that they are bugs, to the extent that I just recommended against switching to pipenv at work for these reasons, but I would love to get confirmation one way or another as to where you're at? Regarding the structuring suggestions: My understanding of The reason that I have tox and django-debug-toolbar in |
I tried moving things to install_requires, as suggested, and it resulted in none of those dependencies getting installed by pipenv, so that seems like it unfortunately is not the solution (unless I'm doing something wrong). |
Confirmed that I can reproduce this behavior locally if pipenv is running in python 2.7 instead of python 3. It only shows up on the first install; subsequent installs seem to re-install the correct version. |
After further investigation, this seems to be (partially) a race condition issue. If I introduce a delay before installing the VCS dependencies, or use The other half of this is that VCS dependencies (unlike any other dependency) do not use So yeah - workaround is: use |
Reusability and being a package don't have anything to do with one another, being reusable doesn't require being a package, and being a package doesn't imply that a thing is reusable. Many teams use local packaging environments to install their own software in production or testing, or to insulate individual dependencies.
I agree. That is a desirable outcome.
I actually disagree with this, lets use an example. Say you install In this case your version ( This is a resolution order issue and I agree it should be handled correctly. While I understand this may seem simple, in fact it isn't, or it would be working correctly :) That's why we try to let the resolver handle this instead of using strict pins.
They are most certainly bugs, that's why we have open issues addressing them and flattening the dependency graph is not something you will get out of pip either, which is also order-sensitive when installing
The primary advantage of this is that it makes pipenv treat your project as the only top level dependency and resolves everything relative to it.
No, pipenv has nothing to do with
On what version of pipenv? Dev packages should never win in resolution against packages, I'd be curious to see a pipfile that shows this happening |
Thanks for your detailed reply. :-) It sounds like we disagree on some philosophical points and are maybe talking past each other in a couple spots; I'm not going to respond fully to your comments because I don't want to distract from the point of the ticket.
Based on my investigation, this is not a resolution order issue, it's a race condition during parallel installation that's triggered because VCS repo dependencies install their dependencies, which can override the dependencies specified in Pipfile.lock. |
Hmm, in this case, I wonder what would happen if you specify the |
@uranusjr The issue isn't that pip upgrades. The problem is:
If I then re-run pipenv install I correctly end up with Django 1.8, or if I run FWIW my preferred solution would be to install VCS repos without dependencies, like any other package. |
I see. The reason I raised this is that if the VCS dependencies are installed after 1.8, it would actually be fine because pip won’t upgrade installed dependencies. Maybe this is what we should do here (maybe not, just raising possibilities). For the record, judging from past issues, not installing dependencies obtained from VCS repos is likely to cause complaints from the other side. Many would want them to be installed. |
I think you might be misunderstanding. VCS Dependencies MUST EXECUTE the We can't not install dependencies here, so we can just leave that one alone.
No, they don't override the lockfile if you include the editable flag so that their dependencies land in the lockfile. That's why we have a warning about this. If you leave this off, then yeah, the dependencies are getting installed either way, so I guess you're rolling the dice.
So it's not an issue with the order in which things are resolved => installed, it's just a race condition related to the order in which things are resolved and installed. |
@techalchemy The VCS repos in question have the editable flag set to true already. That's not the problem. When you talk about "resolution order", do you mean that Pipenv is keeping track of dependencies during install so that a VCS repo that depends on django should never start installing until after the django version specified in the lockfile has finished installing? If so then I agree with you that this is a problem with resolution order. (I previously thought you were talking about the order in which dependencies were resolved in order to generate the lockfile.) |
@melinath exactly. VCS installs are supposed to happen sequentially and should never start until after the rest of the installs finish. This has me wondering if this is causing more issues and if we regressed that sequential install somehow |
- Fixes #2088, #2234, #1901 - Fully leverage piptools' compile functionality by using constraints in the same `RequirementSet` during resolution - Use `PIP_PYTHON_PATH` for compatibility check to filter out `requires_python` markers - Fix vcs resolution - Update JSON API endpoints - Enhance resolution for editable dependencies - Minor fix for adding packages to pipfiles Signed-off-by: Dan Ryan <dan@danryan.co>
- Fixes #2088, #2234, #1901 - Fully leverage piptools' compile functionality by using constraints in the same `RequirementSet` during resolution - Use `PIP_PYTHON_PATH` for compatibility check to filter out `requires_python` markers - Fix vcs resolution - Update JSON API endpoints - Enhance resolution for editable dependencies - Minor fix for adding packages to pipfiles Signed-off-by: Dan Ryan <dan@danryan.co>
- Fixes #2088, #2234, #1901 - Fully leverage piptools' compile functionality by using constraints in the same `RequirementSet` during resolution - Use `PIP_PYTHON_PATH` for compatibility check to filter out `requires_python` markers - Fix vcs resolution - Update JSON API endpoints - Enhance resolution for editable dependencies - Minor fix for adding packages to pipfiles Signed-off-by: Dan Ryan <dan@danryan.co>
Still experiencing this problem and had to use |
My pipfile and pipfile.lock both specify an explicit version of Django - 1.8.11. Locally, using python3 on a Mac, this install works fine. On Travis, using python2 on linux, Django 1.11.12 is installed instead.
Travis: $ python -m pipenv.help output
Pipenv version:
'11.10.1'
Pipenv location:
'/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pipenv'
Python location:
'/home/travis/virtualenv/python2.7.14/bin/python'
Other Python installations in
PATH
:2.7
:/home/travis/virtualenv/python2.7.14/bin/python2.7
2.7
:/home/travis/virtualenv/python2.7.14/bin/python2.7
2.7
:/opt/pyenv/shims/python2.7
2.7
:/usr/bin/python2.7
3.4
:/usr/bin/python3.4m
3.4
:/usr/bin/python3.4
3.6
:/opt/pyenv/shims/python3.6
2.7.14
:/home/travis/virtualenv/python2.7.14/bin/python
2.7.14
:/opt/pyenv/shims/python
2.7.6
:/usr/bin/python
2.7.14
:/home/travis/virtualenv/python2.7.14/bin/python2
2.7.14
:/opt/pyenv/shims/python2
2.7.6
:/usr/bin/python2
3.4.3
:/opt/pyenv/shims/python3
3.4.3
:/usr/bin/python3
PEP 508 Information:
System environment variables:
TRAVIS_UID
rvm_version
PYTHONDONTWRITEBYTECODE
TRAVIS
_JAVA_OPTIONS
PIPENV_IGNORE_VIRTUALENVS
TRAVIS_STACK_LANGUAGES
JRUBY_OPTS
VIRTUAL_ENV
SHELL
TRAVIS_PULL_REQUEST_BRANCH
PYENV_SHELL
TRAVIS_BRANCH
TRAVIS_PULL_REQUEST_SLUG
HISTSIZE
NVM_BIN
RBENV_SHELL
MANPATH
JAVA_HOME
_system_type
TRAVIS_SECURE_ENV_VARS
MY_RUBY_HOME
STRIPE_TEST_ORGANIZATION_PUBLISHABLE_KEY
RUBY_VERSION
PIP_DISABLE_PIP_VERSION_CHECK
HOSTNAME
DATABASE_URL
_system_version
TRAVIS_COMMIT_RANGE
GOPATH
CONTINUOUS_INTEGRATION
GOROOT
rvm_path
TRAVIS_REPO_SLUG
USER
PS1
PS4
container
STRIPE_TEST_APPLICATION_ID
SHLVL
TRAVIS_PULL_REQUEST_SHA
MERB_ENV
JDK_SWITCHER_DEFAULT
STRIPE_TEST_PUBLISHABLE_KEY
GIT_ASKPASS
TRAVIS_FILTERED
GEM_PATH
HAS_ANTARES_THREE_LITTLE_FRONZIES_BADGE
TRAVIS_EVENT_TYPE
DWOLLA_TEST_USER_ACCESS_TOKEN
PIP_PYTHON_PATH
DWOLLA_TEST_USER_USER_ID
DWOLLA_TEST_ORGANIZATION_ACCESS_TOKEN
TRAVIS_TAG
DWOLLA_TEST_APPLICATION_KEY
TRAVIS_BUILD_NUMBER
PYENV_ROOT
TRAVIS_STACK_FEATURES
_system_name
DWOLLA_TEST_ORGANIZATION_REFRESH_TOKEN
TRAVIS_SUDO
MIX_ARCHIVES
TRAVIS_BUILD_ID
NVM_DIR
TRAVIS_STACK_NAME
HOME
TRAVIS_PULL_REQUEST
LANG
TRAVIS_COMMIT
_
TRAVIS_STACK_JOB_BOARD_REGISTER
_system_arch
MYSQL_UNIX_PORT
CI
rvm_prefix
DEBIAN_FRONTEND
TRAVIS_PRE_CHEF_BOOTSTRAP_TIME
TRAVIS_COMMIT_MESSAGE
IRBRC
TRAVIS_STACK_TIMESTAMP
CASHER_DIR
STRIPE_TEST_ORGANIZATION_USER_ID
TRAVIS_STACK_NODE_ATTRIBUTES
RACK_ENV
PERLBREW_HOME
GEM_HOME
HAS_JOSH_K_SEAL_OF_APPROVAL
PYTHON_CFLAGS
COMPOSER_NO_INTERACTION
DWOLLA_TEST_USER_REFRESH_TOKEN
NVM_CD_FLAGS
TRAVIS_BUILD_STAGE_NAME
PERLBREW_BASHRC_VERSION
PATH
DWOLLA_TEST_USER_PIN
TRAVIS_ALLOW_FAILURE
TERM
TZ
STRIPE_TEST_ORGANIZATION_ACCESS_TOKEN
MALLOC_ARENA_MAX
HISTFILESIZE
TRAVIS_OSX_IMAGE
rvm_bin_path
STRIPE_TEST_SECRET_KEY
RAILS_ENV
PERLBREW_ROOT
TRAVIS_JOB_NUMBER
PYTHON_CONFIGURE_OPTS
DWOLLA_TEST_APPLICATION_SECRET
DWOLLA_TEST_ORGANIZATION_USER_ID
DWOLLA_TEST_ORGANIZATION_PIN
LC_ALL
TRAVIS_JOB_ID
TRAVIS_PYTHON_VERSION
PAGER
OLDPWD
TRAVIS_LANGUAGE
TRAVIS_BUILD_DIR
HISTCONTROL
PWD
TRAVIS_OS_NAME
ELIXIR_VERSION
rvm_pretty_print_flag
Traceback (most recent call last):
File "/opt/python/2.7.14/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/opt/python/2.7.14/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pipenv/help.py", line 89, in
main()
File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pipenv/help.py", line 48, in main
print(u'Pipenv–specific environment variables:')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 6: ordinal not in range(128)
The command "python -m pipenv.help" failed and exited with 1 during .
Local: $ python -m pipenv.help output
Pipenv version:
'11.10.1'
Pipenv location:
'/usr/local/lib/python3.6/site-packages/pipenv'
Python location:
'/usr/local/opt/python3/bin/python3.6'
Other Python installations in
PATH
:2.7
:/usr/bin/python2.7
2.7
:/usr/bin/python2.7
3.6
:/usr/local/bin/python3.6m
3.6
:/usr/local/bin/python3.6
2.7.10
:/usr/bin/python
3.6.3
:/usr/local/bin/python3
PEP 508 Information:
System environment variables:
TERM_PROGRAM
TERM
SHELL
TMPDIR
Apple_PubSub_Socket_Render
TERM_PROGRAM_VERSION
TERM_SESSION_ID
USER
SSH_AUTH_SOCK
PATH
PWD
LANG
XPC_FLAGS
XPC_SERVICE_NAME
HOME
SHLVL
LOGNAME
SECURITYSESSIONID
_
__CF_USER_TEXT_ENCODING
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH
Pipenv–specific environment variables:
Debug–specific environment variables:
PATH
:/usr/local/Cellar/node/9.6.1/bin/:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Postgres.app/Contents/Versions/latest/bin
SHELL
:/bin/bash
LANG
:en_US.UTF-8
PWD
:/Users/******/projects/django-brambling
Contents of
Pipfile
('/Users/******/projects/django-brambling/Pipfile'):Contents of
Pipfile.lock
('/Users/******/projects/django-brambling/Pipfile.lock'):Expected result
The version of django specified & locked is installed.
Actual result
A different version is installed. I can't provide verbose logs because they're over 4MB large and travis won't even save the whole thing.
Steps to replicate
I'm not sure how to replicate this, since it works fine locally. Here's the repository I'm working on at a commit where this is breaking on travis. Here's the related travis build that's failing.
The commands that travis is running (some purely for debug information) are:
Please note that
pipenv graph
, which runs immediately afterpipenv install
already shows the incorrect version of django installed.The text was updated successfully, but these errors were encountered: