Multiple constraint dependency resolution is unexpected #9687
Comments
eickr
added
kind/bug
|
poetry is just finding the simplest solution that it can to the problem that you set it. if you want it to install a later version of numpy, then make that a requirement |
How? By increasing the lower bound on the second constraint? That would prevent anything using this project as a dependency from being able to use the previously valid lower versions of numpy. Neither of those options are desirable. If I could simply use
Is it, though? By running Wouldn't the simplest solution be the following?
I won't pretend to understand poetry's internals, so maybe I'm way off base and/or there's a specific reason that type of resolution is avoided. |
|
If you are satisfied that numpy 1.24.4 is a valid solution for all python versions then you have no reason to complain about poetry choosing it. For packages that you are intending to distribute, I mostly think of the lock file as a proof of the existence of a solution: it is not the solution that any consumer will necessarily get. Indeed you are under no obligation to install from the lock file yourself: |
I would have no complaints if my pyproject.toml file merely contained Perhaps I am not adequately conveying why the behavior described in this issue is a problem. Is there any use case where the current behavior is desired?
I initially intended to reply that the lock file is merely an artifact affected by this issue, not really related to the core issue here as they aren't required to distribute applications. However, further testing has revealed another unexpected difference in behavior depending on the lock file. See the three comments at the end of example 1 in the poetry runtime logs section. Regardless, I can only see the advice to use pip as an agreement that there's a problem with poetry. There's already feature requests regarding the ability to specify an arbitrary (valid) version. If the root cause of this issue is the same thing making that feature request difficult to implement, then this can eventually be wrapped up into that. In the meantime, I've found a workaround that I can consider as acceptable enough: python = ">=3.8,<3.12"
numpy = [
{ version = "^1.23.2 || >99999", python = ">=3.8,<3.9" },
{ version = "^1.23.2", python = ">=3.9,<3.12" },
] |
|
The docs that you quote do not support your expectation. It is clearly not the case that poetry will always lock everything to its latest version. (If only it could always be so easy!) In all sorts of circumstances: if you don't like the valid solution that poetry finds to your constraints - then you need to change your constraints. You are saying both that
I do not think it is likely that there will be a change to the solver here. |
Would you agree that
I am not saying that second bullet. The earlier versions down to the minimum bound are fine with the project. Numpy v1.24.4 for python v3.11 is a valid version given the constraints. The problem, as I see it, is the inconsistent solver behavior seemingly due to the (undocumented) merging of conditional environment markers during solving.
The docs say poetry supports PEP 508 markers. PEP 508 says "A marker expression evaluates to either True or False. When it evaluates to False, the dependency specification should be ignored." However, PEP 508 also says "It draws a border at the edge of describing a single dependency - the different sorts of dependencies and when they should be installed is a higher level problem.". So, technically, I don't think poetry is violating it by merging multiple specifications into a new non-equivalent specification. |
|
You are exploring non-essential behaviours of the poetry resolver and trying to rely on them. If poetry happened to do what you want already, and in that world someone showed up complaining that it failed to combine two obviously-combinable requirements - they would have just as much of a point as you do. Either behaviour seems plausible, the docs make no commitment either way, I see no good reason to change in either direction. fwiw both dependencies = [
"numpy>=1.23.2,<2.0.0 ; python_version >= '3.8' and python_version < '3.9'",
"numpy>=1.23.2,<2.0.0 ; python_version >= '3.9' and python_version < '3.12'",
]for both |
|
I agree that, given the reason the solver is doing what it's doing, there is probably no good argument to say one way would be definitively better than the other. Well, unless you wanted to use poetry to manage dependencies for automatic testing without needing multiple pyproject.toml files, but that's well outside the scope of this issue. In hindsight, I just wish a user didn't have to delve into the source code to find out that poetry will intentionally try to merge the conditionals of all entries for a dependency that have the same version constraint key. Or rather, to find out that the behavior I was seeking is not currently possible intentionally. If you think that there's not enough reason to update the documentation to clarify the current behavior and capabilities, then you can close this issue. Maybe this issue itself will be enough for anyone scratching their head to find the clarification they need without needing to understand the source code. |
|
I agree with dimbleby. It is an implementation detail and by cleverly changing the constraints, you influence Poetry according to your wishes. However, since it is an implementation detail it is pure luck and can change from one version to another. A sustainable solution might be a resolution strategy that does one resolution per Python minor version. The downside is that resolution will be much slower with such a strategy. If we ever implement alternative resolution strategies that might be a thing. |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Description
When using multiple constraints syntax to tell poetry that it should handle a dependency differently for different versions of python, if the dependency's version constraint is equivalent then poetry short circuits unexpectedly and does not resolve to the latest viable version for the current environment.
Given the example pyproject.toml file provided, I would expect poetry to install numpy v1.24.4 in a python v3.8 environment and to install numpy v1.26.4 in a python v3.11 environment.
Instead, poetry will select numpy v1.24.4 for both python v3.8 and v3.11 environments. If the venv is clean then it will install numpy v1.24.4 for both environments.
Interestingly, if numpy v1.26.4 is already installed in a python v3.11 environment, then despite selecting numpy v1.24.4, poetry will not downgrade to it.
One might argue that the constraint for python v3.8 restricted numpy should be capped below v1.25.0 to make it different, since that's the version that dropped v3.8 support. However, that knowledge is not always available. Pretend that numpy v1.24 wasn't even available when this example project was last touched. If the dependency resolution worked as expected then this example project wouldn't need to change the constraint and rebuild.
My best guess about what is happening here is that poetry might be merging the two multiple constraints together into a single
numpy = { version = "^1.23.4", python = ">=3.8,<3.12" }constraint. That behavior would make sense... if it resulted in poetry installing numpy v1.26.4 in a python v3.11 environment instead of installing numpy v1.24.4.Workarounds
If you differ the dependency's version constraint slightly then poetry is able to resolve the dependency to the latest viable version that you would expect.
For example, if you replaced the numpy constraint in the example pyproject.toml with the following then it will resolve as expected.
or
Either of those will result in poetry installing numpy v1.26.4 in a python v3.11 environment and installing numpy v1.24.4 in a python v3.8 environment.
Poetry Installation Method
pip
Operating System
Ubuntu 22.04
Poetry Version
Poetry (version 1.8.3)
Poetry Configuration
Python Sysconfig
No response
Example pyproject.toml
Poetry Runtime Logs
The text was updated successfully, but these errors were encountered: