Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resolver dependency contamination #2785

Closed
gsemet opened this issue Aug 28, 2018 · 9 comments · Fixed by #2935
Closed

resolver dependency contamination #2785

gsemet opened this issue Aug 28, 2018 · 9 comments · Fixed by #2935
Labels
Category: Dependency Resolution Issue relates to dependency resolution.

Comments

@gsemet
Copy link
Contributor

gsemet commented Aug 28, 2018

Hello

I see a weird behavior on the version resolution on pipenv update, for dependencies where the latest version is to be avoided by another dependency.

Given a Pipfile with no special restriction:

[dev-packages]
"flake8" = "*"
pycodestyle = "*"
pyflakes = "*"
pylint = "*"
yapf = "*"
cryptography = "*"
isort = "*"
PyYAML = "*"
pytest = "*"
tox = "*"
"flake8-comprehensions" = "*"
asynctest = "*"
rope = "*"
pytest-sugar = "*"
pytest-mock = "*"
pytest-watch = "*"
mock = "*"
"autopep8" = "*"
pytest-cov = "*"
bandit = "*"

[packages]

For example, pycodestyle latest version is 2.4.0. Some dependencies such as flake8 adds the restriction "<2.4.0". Another one ask for "pycodestyle<1.7.0". But the resolver fails at the end with the message

Could not find a version that matches pyflakes<1.7.0,==2.0.0,>=1.5.0

While reasonably the "==2.0.0" have nothing to do here since no one ever restrict or freeze to this version. This looks like the latest found version on pypi in the previous resolver stage is taken are a hard restriction (with ==), so any later resolution stager should follow it, or fail.

This appears to me as a bug since later stages can only add new dependencies, not restrict the version of a package already resolved.

To me, each resolver stage should keep the resolution context (for pycodestyle, keep the "require=*" information instead of "require==2.4.0").

We can see this behavior in the logs:

$ pipenv update --clear --verbose
...
pycodestyle (from -r /tmp/pipenv-bvm1pyac-requirements/pipenv-jtt7r2p0-constraints.txt (line 4))
...

Finding the best candidates
...
  found candidate pycodestyle==2.4.0 (constraint was <any>)
...

Finding secondary dependencies:
..
  flake8-comprehensions==1.4.1 requires flake8!=3.2.0, flake8-comprehensions==1.4.1, mccabe<0.7.0,>=0.6.0, pycodestyle<2.4.0,>=2.0.0, pyflakes<1.7.0,>=1.5.0
...
pycodestyle==2.4.0        requires pycodestyle==2.4.0
  ^=== This is were the issue happen, the ==2.4.0 should not be, it should be something like pycodestyle==2.4.0 (restriction="*"), to keep the information
...
  flake8==3.5.0             requires flake8==3.5.0, mccabe<0.7.0,>=0.6.0, pycodestyle<2.4.0,>=2.0.0, pyflakes<1.7.0,>=1.5.0
...

New dependencies found in this round:
...
  adding ['pycodestyle', '<2.4.0,==2.4.0,>=2.0.0,>=2.3', '[]']

From now on, the resolution couldn't succeed since "<2.4.0" and =="2.4.0" are obviously incompatible.

Workaround: recopy the restriction "<2.4.0" in my Pipfile. From now, the resolver will take the latest version that respect <2.4.0 and everybody will be happy later on.

Happens with pycodestyle and pyflakes (restricted to "<1.7.0" by flake8 3.5.0)

@frostming
Copy link
Contributor

frostming commented Aug 28, 2018

Don't repeat dependencies in your Pipfile, pycodestyle and pyflakes are dependencies of flake8, please remove them. A good practice is to list top level dependencies only.

This will solve most issues since Pipenv's resolver is not smart enough to handle cases like this.

@gsemet
Copy link
Contributor Author

gsemet commented Aug 28, 2018

I understand, but this is just an example on how to easily reproduce it. The first round of dependencies résolution kind of freeze the version of what has been found while it should not.

@techalchemy techalchemy added the Category: Dependency Resolution Issue relates to dependency resolution. label Sep 2, 2018
techalchemy added a commit that referenced this issue Oct 7, 2018
- Fixes #2499
- Fixes #2529
- Fixes #2589
- Fixes #2666
- Fixes #2767
- Fixes #2785
- Fixes #2795
- Fixes #2801
- Fixes #2802
- Fixes #2824
- Fixes #2862
- Fixes #2867
- Fixes #2879
- Fixes #2880
- Fixes #2894
- Fixes #2902
- Fixes #2924

Signed-off-by: Dan Ryan <dan@danryan.co>
techalchemy added a commit that referenced this issue Oct 7, 2018
- Fixes #2499
- Fixes #2529
- Fixes #2589
- Fixes #2666
- Fixes #2767
- Fixes #2785
- Fixes #2795
- Fixes #2801
- Fixes #2802
- Fixes #2824
- Fixes #2862
- Fixes #2867
- Fixes #2879
- Fixes #2880
- Fixes #2894
- Fixes #2902
- Fixes #2924

Signed-off-by: Dan Ryan <dan@danryan.co>
@gsemet
Copy link
Contributor Author

gsemet commented Oct 8, 2018

great news! looking forward to testing that ! thanks for the amazing work on pipenv :)

@gsemet
Copy link
Contributor Author

gsemet commented Oct 19, 2018

Saldy, I still have this issue.

@idchlife
Copy link

Suddenly project started showing failed dependency of

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies. First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again. Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation. Hint: try $ pipenv lock --pre if it is a pre-release dependency. Could not find a version that matches pycodestyle<2.4.0,>=2.0.0,>=2.4.0 Tried: 2.0.0, 2.0.0, 2.1.0, 2.1.0, 2.2.0, 2.2.0, 2.3.0, 2.3.0, 2.3.1, 2.3.1, 2.4.0, 2.4.0 Skipped pre-versions: 1.8.0.dev0, 1.8.0.dev0, 2.0.0a1, 2.0.0a1 There are incompatible versions in the resolved dependencies.

But I don't have this dependency inside Pipfile D:

Project was working month ago, not suddenly stopped. Could it be my upgrade to Python 3.7?

@gsemet
Copy link
Contributor Author

gsemet commented Oct 30, 2018

not, it is still happening. What I called "contamination" still happens, because autopep8 depends on a version of flake8 that requires pycodestyle<2.4.0. Your solution is to update flake8, or prevent update of autopep8 with autopep8<1.4.1.

The resolver algorithm is pretty hard to debug, probably needs to be completely rewritten. I actually do not understand why they do not delegate completely this to pip which is able to find the right version if you pip install all direct dependencies on a single line.

@techalchemy
Copy link
Member

Because pip isn’t a resolver and doesn’t have one to resolve things. That’s why we don’t delegate it. If you want pip to instal whatever it feels like you can use pip. Pipenv is backed by dependency resolution which doesn’t just do randomly ordered installation of whatever it encounters.

We did rewrite the entire thing from scratch btw, https://github.com/sarugaku/passa — feel free to try it out

@gsemet
Copy link
Contributor Author

gsemet commented Oct 31, 2018

Good to know! I hope it will fix this issue. Because autopep8 releases a new version 1.4.1 which may introduce an incompatibility in resolution, the resolver should be able to find it needs to use 1.4.0, even if it involves several round and retries. I understand this complicates a lot, but would greatly improve the adoption of pipenv for python app.

@idchlife
Copy link

@gsemet yes, exactly! Learned about dependency which breaks the installation by removing-returning parts of Pipfile and locating dependency.

It will help much if pipenv would print like parent dependency, so developer can learn right now which one should be updated or with frozen version.

Anyway, except this pipenv worked flawlessly. Hope my way of resolving issue would help newcomers (as myself) into pipenv.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Dependency Resolution Issue relates to dependency resolution.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants