pip needs a dependency resolver

[cboylan]

pip's dependency resolution algorithm is not a complete resolver. The current resolution logic has the following characteristics:

• Only one top level specification of a requirement (otherwise pip raises a "double requirement" exception)
• "first found wins" behavior for dependency requirements/constraints, which results in not respecting all constraints. pip 10+ print a warning when this occurs.

NOTE: In cases where the first found dependency is not sufficient, specifying the constraints for the dependency on the top level can be used to make it work.

pip install project "dependency>=1.5,<2.0"

---

(2019-06-23)

This is being worked on by @pradyunsg, in continuation of his GSoC 2017 project. A substantial amount of code cleanup has been done, and is ongoing, to make it tractable replace the current resolver, in a reasonable manner. This work enabled pip >= 10 to warn when it is going to make an installation that breaks the dependency graph. (The installations are not aborted in such scenarios, for backwards compatibility.)

---

(2019-11-29)

A status update regarding this is available here.

---

(2022-12-16)

See the closing note for details.

[dstufft]

I added this to 1.5 because I think it's a pretty big wart that should be sorted out.

[qwcode]

for now, the "pip solution" is to declare in your requirements file (or as an install argument directly) what specifier you want, and it overrides what happens down in the dependency tree.

i.e. in this case, put "foo>=1.0,<=2.0" in your requirements or add it as an install argument.

also, pip compiles everything it's going to install into a set first, then installs.
it's not installing everything as it's discovered, and it's discovery order is not depth first.
the order of discovery isn't the issue, but rather the lack of doing any resolution as it's compiling the set.
Currently, first found wins.

[qwcode]

I keep meaning to add a "conflict resolution" section to the Internal logic section to the docs that covers this.
practically speaking, pip's override logic is pretty nice IMO, and is all most people need most of the time.

[emonty]

I believe, based on discussions with @dstufft, that this may be related to the problems people will have if they try to upgrade from old distribute to new setuptools in the same set of software as other things. Or, more specifically, if you depend on a piece of software that depends on distribute with an unbounded high end, and you are currently running on a pre-merge distribute, what will happen is that the dependency discovery will put distribute into the list along with the rest of the things that software A depends on, then it will add distribute's dependency, setuptools, to the list, later. THEN, when it's installing, distribute will be upgraded, which removes the distribute provided setuptools code, then the next thing in the list is installed, which is not setuptools, and which fails because setuptools is not installed yet.

[qwcode]

@emonty the distribute to setuptools upgrade problem you speak of is described in #1064, but also here http://www.pip-installer.org/en/latest/cookbook.html#importerror-no-module-named-setuptools

but this is not related to pip's conflict resolution shortcoming issues described in this issue.

[dracos]

Just had this issue installing a project - on a system that already had python-dateutil 1.4 installed - that depended upon python-dateutil (no specific version) and django-tastypie 0.9.16. Even though django-tastypie has a python-dateutil dependency of >=1.5 (and not 2.0), and the parent has no specific version dependency on python-dateutil, python-dateutil remained at 1.4. This seems pretty fundamentally against what a package installer should be doing in such a circumstance :)

[dstufft]

FWIW I've been experimenting with making a real resolver for pip.

[qwcode]

@dracos there is a "solution" right now (short of a new resolver for pip; although that would be nice). Specifically declare what you want the python-dateutil requirement to be in a requirements file, or as a top-level pip install argument, and that will be honored.

http://www.pip-installer.org/en/latest/cookbook.html#requirements-files
http://www.pip-installer.org/en/latest/logic.html#requirement-specifiers

pip install myproject datetutil>=1.5,<2.0

[dracos]

@qwcode Thanks, I know that and will have to do so - but this means anyone working on the project will have to manually check all dependencies. Say someone upgrades django-tastypie and it now requires a later version of python-dateutil, there's no way to detect this besides manual checking of each dependency upgrade/install. Oh well.

[qwcode]

@dracos understood. it's a pain point. but just want others who find this, to at least know, there is some kind of solution.

[qwcode]

@dstufft as you work on a new resolver, keep in mind that top level requirements (i.e. pip install arguments are requirements file entries) will still have to be considered overrides (or dominant). that's a pip feature. Also, to state the obvious, this change will be "backwards incompatible" in the sense that many installations will turn out differently.

[dracos]

I've made a script (that is basically a patch to pip's prepare_files(), but I didn't want to patch pip, I can't control pip everywhere I use it for example) - available at https://github.com/dracos/check-pip-dependencies - that notes multiple dependency requests for the same package and outputs a list of conflicts, that you can then manually resolve using the methods suggested above.

[benoitbryon]

This ticket looks like #174.

[ghost]

The pydata people at continuum analytics have built a packaging toolchain
more suited to scientific packaging (That's good, it can get funky).
Facing the same issues, they've implemented a dependency resolver using a SAT solver,
see here. The paper mentioned there is readable and they've open sourced the wrapper package
around the (open as well) SAT solver engine.

Basically, it's all there.

[Ivoz]

@y-p the only problem with that is that it relies on C code, rather than pure python. I presume that it would notionally be unacceptable to expect that everyone installing pip have a C compiler easily available on their system; thus you would have to provide a compiled pip for every system Python hopes to run on. That is a major ask (anyone compiling and distributing for python arm?).

It might be "all there", but it's in an unusable form for pip's general audience.

[merwok]

MIT-licensed pure Python: https://github.com/ActiveState/depgraph
(I just know of it, never tested it on the field)

[Ivoz]

0install used to be based on python (now on OCaml), and also wrote a pure python solver, which can be found in their 1.16 release (also in sat.py); afaik it was py23 compatible.

Also some good notes on sat solving on their page, and an awesome email.

minisat paper; GRASP paper

[Wilfred]

pkglib has recently released a pure Python dependency solver, FWIW: https://github.com/ahlmss/pkglib/blob/master/pkglib/pkglib/setuptools/dependency.py#L596

[dstufft]

Thanks, I'll take a look at it when I take a look at the enthought one as well.

[pfmoore]

I've been reading some of the references. One thing I don't see in any of them is how version dependencies are managed - specifically, if package A depends on B (version >=2.0) how is that encoded. Is each version of a package treated as an entirely separate object for resolution purposes? Does anyone have any pointers to references on how this is handled?

[brainwane]

Per #8511 we have now released pip 20.2. This release includes the beta of the next-generation dependency resolver. It is significantly stricter and more consistent when it receives incompatible instructions, and reduces support for certain kinds of constraints files, so some workarounds and workflows may break. Please test it with the --use-feature=2020-resolver flag. Please see our guide on how to test and migrate, and how to report issues. Please report bugs using this survey or by opening a new GitHub issue, not commenting on this one.

The new dependency resolver is off by default because it is not yet ready for everyday use.

We plan to make pip's next quarterly release, 20.3, in October 2020. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3.

Please spread the word by pointing to this blog post -- spread the word on Hacker News, Reddit, Twitter, Facebook, Dev.to, Telegram, relevant Stack Overflow answers, your favorite Slacks and Discords, etc. Most of the people this will affect do not keep up with Python-specific developer news. Help them get the heads-up before October, and help us get their bug reports.

[thomasf]

Please consider adding a minimal version selection to pip argument for this resolver, that means only consider the minimal versions specified by each dependency version spec. Go modules uses this and it's the least intrusive version spec system I have seen.

It enables having one pip requirements.txt file without lock files and always results in a reproducible version dependency tree.

If you want to force an upgrade for a sub dependency, just add it to your requirements file or I guess a constraints file. If all package in the resolver has no minimal version set for one specific dependency maybe that could generate a warning because you probably want to add a constraint if that happens.

I guess exceeding a maximum version should still generate an error.

It's a little bit more effort for the project using a feature like this but it also brings repeatable installs without additional lock files and more programs just to manage than lock file.

It would be really nice to move off massively complex tools like pipenv and poetry and be able to skip the concept of a lock file and still get reproducible builds. I'm never interested in the latest possible version of a dependency, I'm always interested on the one I have tested my software with.

After a brief look this is probably best implemented by writing a separate resolver for resolvelib, right?

[brainwane]

Thanks to everyone who tested pip 20.2 and provided bug reports and feedback, or who spread the word! We also made a video you can share.

We are aiming to release pip 20.3 about a week from now, on Wednesday or Thursday, Oct 28 or 29. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3.

For more on the rollout and how you can help, see #6536 (comment) -- starting tomorrow, @di is gathering a volunteer first-response team to help reply to confused users.

[brainwane]

As I discussed in a comment elsewhere we decided to delay the release slightly, because of some CI problems cropping up and because of some external factors. pip 20.3b1 is available in case you want to try that out.

In today's team meeting we agreed that the 20.3 release will likely be tomorrow or Friday. You can follow #8936 for more.

We've also substantially improved the "what's changing" user guide so please take a fresh look and circulate it!

And the new resolver is already solving some people's issues, which is great!

[brainwane]

We have now resolved a finicky Mac OS Big Sur support issue and a headache-inducing infinite resolution issue #9011, which were stopping us from releasing. Per #8936 (comment) the pip 20.3 release, in which the new pip resolver will be the default, will very very likely be tomorrow, Monday, 30 November.

[pradyunsg]

pip 20.3 has been released, and it has the new resolver by default! Here's our release announcement on the PSF blog: https://blog.python.org/2020/11/pip-20-3-release-new-resolver.html

[pradyunsg]

That felt goooood. :)

[WhyNotHugo]

Thanks for all the hard work! <3

[flying-sheep]

that means exactly what it says: it’s impossible to resolve this without contradictions.
figure out which package could loosen its restrictions and bug them in their issuetracker about it.

also I think this is a good place to lock this conversion, people will continue to come in with stuff like this.