New resolver downloads hundreds of different package versions, without giving reason #9215
Comments
|
Hello and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us and including some thoughts on how to address it -- I agree that a more specific error message would be better. I am going to defer to the resolver developers on whether this is an issue they are already addressing in a different issue or whether it's distinct. You probably saw, in that output, the line:
but maybe we should also link to more info there, or have it show up again later. Just in case it's useful, I'll mention here some useful troubleshooting and workaround tips from the documentation:
Do any of those tips help? (If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the pip resolver beta period.) |
|
Thank you very much! The information you provided is very helpful. I think I will use the legacy-resolver as long as I don't know the root cause for the issue. Regarding your last question: I tested this also during the beta-phase and have seen the same issue. I reported the issue over https://pip.pypa.io/surveys/backtracking But it is also my fault, I did not spend so much time in understanding the root cause then. I was thinking that this is an code error in pip rather than a dependency error, because I have not seen a clear dependency conflict statement and could not believe that this is a "normal" behavior in such cases. Thus I just filled the survey and forgot it until today. |
|
One further comment from my side: When I read the documentation the overall statement is "be more strict" to avoid those issues above. Wouldn't it be better to say "be as open as possible and as strict as necessary" in order to guarantee, that most libraries will also in future work together? |
|
I just hit the same: I have the habit of being as strict as possible in my |
|
@lelit could you make a Github Gist with reproduction instructions for this? The resolver should be considering the constraints you pass it and restrict the versions it explores based on it. |
|
Sure, I'll try to distill a recipe. |
|
Ok, got it! |
|
Hurray! Thanks for sharing. I'm AFK at the moment, but if you could you try that with the current master branch, that'd be awesome. There's one fairly major fix merged since the release that'd make things more... efficient. It can be installed with |
|
@pradyunsg: using current master the "backtrace" moves to another package ( |
|
I had a related problem, but it was somewhat worse, because seemingly it not only downloads them, but also verifies them. And there was one version of this package from 4 years ago that was mispackaged, and pip immediately aborted mission. See spyder-ide/spyder#14365 And there is more, because after introspecting the dependency tree of Spyder, all dependencies requiring the package Based on this I have a few recommendations:
|
This is what the resolver does.
This behaviour is under discussion at #9203
In this case, yes indeed. That's what the behavior of the resolver would be. |
|
@hwalinga |
|
Given #9232 it seems more people are seeing unnecessary backtracking. |
This needs to remain a feature in one way or another (maybe a different flag?) when 21.0 releases. The new resolver has made testing several of our packages in Tox, particularly those with many ML-centric dependencies virtually impossible. |
Is that because you don't expect to have addressed the issue before the 21.0 release? If that's the case, do you have any feel yet for how long it will take before you'll be ready to switch to the new resolver? We won't retain the old resolver indefinitely, but it's certainly an option to delay removing it a little longer than 21.0, if there's sufficient reason. (But there's always the option of pinning to an older version of pip, if you expect it to take a really long time to address the problems). |
If 21.0 is released next month, well yes, maybe there is not enough time to fix everything. I suspect this is the most problematic issue currently #9203 (comment) (at least stuff broke for me) and I wonder if the statement that this is uncommon is true (it wasn't the oauthlib in my case, so that makes at least 2). Given by the statement that the problems are particular ML-centric packages, I suspect old mispackaged packages are the problem as well. (ML experts aren't known to be experts in software packaging.) As long this is in Pip, I wouldn't abandon the old resolver just yet: pip/src/pip/_internal/resolution/resolvelib/candidates.py Lines 207 to 208 in 30eeb9c |
|
It's not that we don't expect to have addressed the issue before 21.0, it's that right now the new resolver without any flexibility will probably by-design persist the issue. Here's a walkthrough of the issue: The reason why it is rough on Tox is that Tox functions by re-instantiating a new virtualenv and reinstalling a package's dependencies fresh when it runs. We also dockerize this process so we can ensure we can test for consistency regardless of platform. When pip does the fresh reinstallation, by running through the entire dependency resolution each time, it slows down this process dramatically. And on packages with heavier dependencies with wide ranges of version possibilities (we have even encountered 19 hour hangups trying to install Even if it finishes, any time in the future we need to test again, this process has to start all over. Our projects use Poetry and Pipenv, which lock dependency resolutions so that they do not need to be repeated until dependency requirements explicitly change. Right now, pip 20.3 does not seem to have this option to lock. We do currently pin our pip version, and we've pinned for now at 20.2.4, but not having the option to get any new features in later pip versions because we want the option to not always use dependency resolution (even Pipenv gives you the option, when debugging, to install dependencies without resolution and locking) feels like a harsh tradeoff. |
brainwane
added
the
state: needs eyes
|
I've encountered this issue as well in this build. It doesn't happen on my local macOS workstation nor on an Ubuntu Focal docker image nor on the Python 3.8 build in GHA, but on Python 3.6 for macOS and Linux as found in GHA, the build fails when it encounters the dependency What's particularly interesting about pytimeparse is it has no dependencies and it's only required by one (transitive) dependency in the project (recapturedocs -> jaraco.mongodb -> pytimeparse). That is, I can't think of any reason why pip would be inclined to spend any time downloading anything but the most recent version of pytimeparse and should never trigger backtracking. Also, when it attempts to download every version of pytimeparse, it fails when it gets to 1.0.0, which had a bug 6 years ago. |
|
@jaraco It is clear in reptrospect that pytimeparse won’t make a difference since it has no dependencies, but the resolver can’t know that unless it downloads and tries the distribution since Python package distributions store dependency information in the package. |
|
@uranusjr That is completely correct, if pytimeparse indeed introduced conflicting dependencies. However after downloading the most recent version of pytimeparse, it should conclude that that version can be installed without error, and proceed to install the most recent version, without downloading all other versions of pytimeparse. I have suggested this before, but pip should when it is backtracking check from most recent to older versions of a package and then whenever it finds a version that satisfies the requirements go ahead and install that, without going through all possible versions. Likely in almost all cases, it will conclude the most recent version suffices and install that. However, that remark got dismissed because pip should already do that (#9215 (comment)). But given that pip went ahead and seemingly digged into 6 years of the history of a simple package with no dependencies on itself, being a dependent of only one other package, I have a hard time believing that pip really does this correctly. |
|
After some further investigation on the recapturedocs issue, I found I was able to replicate the issue locally by building on Python 3.6 (not sure why I didn't consider that earlier). I then attempted to build on Python 3.7 and got the error
|
One reason pip cannot use this information might be that there's no capability in packaging's specifier implementation to determine that a set of specifiers can never match. For example Having said this, we shouldn't technically need to download sdists or wheels just to check if their version matches a specifier - we assume that the version in the filename is accurate (which it should according to the specs, and we do later raise an error if the metadata doesn't agree). But we only do this in practice in the finder (which may only know a partial set of specifiers) - we "prepare" the candidate (which includes downloading) as part of constructing it, even if we don't need anything other than name and version yet. Maybe we should reconsider this, and lazily prepare candidates, only doing so when necessary (to build, or when we need metadata other than name/version). IIRC, we did this originally, but abandoned it when it became complex, as it seemed like a premature optimisation, and eagerly preparing allowed better error reporting. It might be worth reconsidering this decision. |
|
Does the following patch help with this issue? Sorry, I don't have a Python 3.7 install to hand to test this right now. It's not a complete fix (the test suite fails because inconsistent metadata is reported incorrectly) but if it does address this issue, then that will demonstrate that there might be value in lazily preparing candidates. |
I created a git tag by patching 21.2.3: notatallshaw@fd33b5a and installing: I could not find any improvement in the performance of either this reproducible example or the reproducible example in #10201. Let me know if there's anything you would like me to try. |
I started down this route and filing a bug in packaging describing the feature request when I realized that maybe this feature isn't what's needed.
Right. I'm also thinking it shouldn't be necessary to determine if a specifier set is invalid, only that no packages in the index match that specifier set. So when the current candidate set of packages is And that approach is even more relevant, because the specifier might include valid specifiers that still match no packages in the index.
I agree. And I've pondered this for scores of minutes and haven't imagined an algorithm that's obviously superior. I appreciate the effort and only seek to add examples and suggestions that may prove helpful. |
|
One suggestion: Please make pip mention why it has to go for backtracking. In one of my use cases I
with no justification, pip just starts downloading every scipy version under the sun. Let it please tell us
|
|
is this bug related to this problem the python foundation is having with pypi? it looks like their server overload problems date back to about the time of the changes to the dep resolver |
|
from the link: " Investigating - PyPI's search backends are experiencing an outage causing the backends to timeout and fail, leading to degradation of service for the web app. Uploads and installs are currently unaffected but logged in actions and search via the web app and API access via XMLRPC are currently experiencing partial outages. |
No, this is an unrelated issue. |
Pip stalls trying to install cwlref-runner. It is related to the new dependency resolver: * pypa/pip#9215 * https://stackoverflow.com/questions/65122957/resolving-new-pip-backtracking-runtime-issue This changeset introduces the use of pipenv to manage the setup.
Summary: After D30550349 (5f44b20) and D30652610 (f1694ef), which introduced `neuralprophet` to Kats, a number of issues arose with the build, mostly stemming from the known issue that `neuralprophet` only supports older `torch` versions (ourownstory/neural_prophet#332). The dependency web was quite complex, as some issues arose from dependencies of dependencies of dependencies. All my solutions are detailed below: 1. The `pip install -r test_requirements.txt` failed as `pip` could not resolve the added dependency complexities from downgrading the required `torch` version in D30884622 (8d0b005). This is a known issue in newer pip versions (pypa/pip#9215), so this diff forces the Github test to use the legacy resolver. 2. `torch>=1.4.0` was still resolving to `torch==1.8.0`, so this diff forces a version downgrade to `torch<1.7.0` which is compatible with `neuralprophet`. 3. Subsequently, the `gpytorch` dependency was failing to install due to it requiring a more recent `torch` version, so this diff downgrades the required `gpytorch` version to `1.2.1` which will still accept `torch==1.6.0`. 4. The newest version of `ax-platform` has `botorch==0.5.1` as a requirement, and this version of `botorch` requires `gpytorch>=1.5.1`, which is incompatible with #3. As such, this diff forces `ax-platform==0.1.18`, which will install `botorch>=0.3.2`, which accepts `gpytorch>=1.2`. 5. Nonetheless, despite #4, sometimes `pip` would still install `botorch==0.5.1`, as it technically satisfies the `ax-platform==0.1.18` requirement of `botorch>=0.3.2`. As such, this diff explicitly adds `botorch==0.3.2` as a dependency and places it before the `ax-platform` installation, ensuring that the correct version of `botorch` is installed, thus allowing the dependencies to resolve. Reviewed By: michaelbrundage Differential Revision: D31044893 fbshipit-source-id: 9152fe04da199dd0061472ea60b302ed3945238f
Toil setup is incompatible with recent python versions. Moreover python is currently suffering of an issue in its depencency resolver. Errors fixed: * error in rdflib-jsonld setup command: use_2to3 is invalid See: https://stackoverflow.com/questions/69100275/error-while-downloading-the-requirements-using-pip-install-setup-command-use-2 * pip stalling in resolving toils dependencies See: pypa/pip#9215 Changes: * Fixed setuptools and other packages versions (using Pipfile) to overcome the "use2to3 error". * Removed toil role (it has one task only). Now toil is installed togheter with cwl using a common pipenv configuration.
This is the preferred way. Additionally it helps prevents this pip issue: pypa/pip#9215 To use pip-compile, pip-tools is required: https://github.com/jazzband/pip-tools/ Versions will be automatically updated by dependabot. Signed-off-by: Giulia Naponiello <gnaponie@redhat.com>
This is the preferred way. Additionally it helps prevents this pip issue: pypa/pip#9215 To use pip-compile, pip-tools is required: https://github.com/jazzband/pip-tools/ Versions will be automatically updated by dependabot. Signed-off-by: Giulia Naponiello <gnaponie@redhat.com>
This is the preferred way. Additionally it helps prevents this pip issue: pypa/pip#9215 To use pip-compile, pip-tools is required: https://github.com/jazzband/pip-tools/ Versions will be automatically updated by dependabot. Signed-off-by: Giulia Naponiello <gnaponie@redhat.com>
pradyunsg
added
C: dependency resolution
This is the preferred way. Additionally it helps prevents this pip issue: pypa/pip#9215 To use pip-compile, pip-tools is required: https://github.com/jazzband/pip-tools/ Versions will be automatically updated by dependabot. Signed-off-by: Giulia Naponiello <gnaponie@redhat.com>
|
I think #10481 might have fixed this issue, as suggested in #10201 (comment) |
|
Indeed! I think we just missed this one while triaging to close the rest. Thanks for the comment @Gallaecio! See #10201 (comment) for guidance, if you're still seeing this issue -- notably, please file a new issue with clear reproduction steps so that we can investigate the reported behaviour. |
Add workaround to pin odltools to the latest. Changes in the new dependency resolver in 21.3 is likely the casue of downloading multiple versions of packages. Issue-ID: LF-JIRA IT-23648 Ref: pypa/pip#9215 Change-Id: Ic40714d449d6c94ec9b8660afc6ad4135e778441 Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
pradyunsg
removed
the
state: needs eyes
We switched today from pip 20.2.2 to 20.3.1 and suddenly in a big project with a very long dependency list (over 60 packages), pip does not manage anymore to install the dependencies from the
requirements.txtfile. It always tries to install hundreds of different package versions from the same package. Sometimes, it just tries out 3 or 10 versions, which takes some time but works in the end, but sometimes, it downloads and installs all possible versions it can find.Here this is what happens with the package
requests:While I understand, that this is coming from the new resolver and a potential compatibility conflict with a certain package, I don't understand what is the exact issue here. We don't use
requestsdirectly, but it seems that this a dependency from a dependency we have. So we are not directly in control of this. And pip does not give me any help to understand, which packages have a version restriction, which lead to this behavior.My suggestion is here to provide a better debug output for developers. When pip has to try out different version, it should tell why it must do so. For example:
This would help me to understand where to begin with fixing those conflicts. At the moment I'm absolutely lost and have to switch back to pip 20.2.2.
The text was updated successfully, but these errors were encountered: