-
Notifications
You must be signed in to change notification settings - Fork 803
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider discarding < 4
upper-bound on Requires-Python
#4022
Comments
It's awkward but perhaps a pragmatic decision to treat |
Thanks for opening a tracking issue. Here's some relevant discussion:
Probably more out there too. |
Seems pretty reasonable until Python 4 is like... a real thing that might happen. If they switch to CalVer <4 is going to be a problem anyway :) |
Haha. But if Python 4 is a thing, then users could set |
A broader thing you could do is ignore all upper bounds for Requires-Python. https://discuss.python.org/t/requires-python-upper-limits/12663 has much longer discussion but while standards say require python works for lower bounds, for upper bounds it's less clear. edit: The main issue is how will backtracking behave and older versions often do not have requires-python upper bound set even though it's unlikely older version actually does work better. |
Henry's comment there is interesting:
In Poetry, the published |
cc @henryiii :) |
That suggests we shouldn't use |
Requires-Python can't be used for upper bounds, it doesn't work properly. The field currently is used to back-solve, and backsolving is the wrong thing to do for an upper bounds. Libraries that have added an upper bound here (numba, numpy, etc) have had to remove it as it provides a worse experience to have the solver just permanently install a really old and broken package instead of the latest broken package. :) https://iscinumpy.dev/post/bound-version-constraints/ talks about it some, and I've proposed some solutions in https://discuss.python.org/t/requires-python-upper-limits/12663 - though it got a bit stuck because packaging.tags should have a way to compute unions on SpecifierSets to help libraries detect an upper bound reliably, and I don't think we fully settled on a preferred solution. Only a small handful of libraries actually have a true, known upper bound; libraries like numba and numpy - but those also have binaries, so After the next Python is released, old releases may actually have a computable upper bound, but you can't go back and modify old metadata. |
What would you suggest we do? We can compute unions on specifier sets. |
For now, ignoring <4 when making a lock file is safe. IMO ignoring upper bounds when making a lock file is safe, as you can't guarantee the upper solution is in fact compatible regardless (for example, https://pypi.org/project/rembg/#data says <3.13 but it's because a dependency states that, and poetry would force them to do that. But when the dependency updates, that fixes the package depending on it, a package should state what they support!), but it's not as surprising to respect it as it is to respect <4. If you want to have a lock file range, you can default to |
this has come up a few times in poetry, I think python-poetry/poetry#6884 contains the most useful discussion. fwiw I continue to hold the view that I expressed there: that it would be better to get the rules changed - presumably by an updated PEP - than to have solvers guess that folk meant something different than what they wrote. Incidentally re
While I think it's true that poetry's choice to default |
Poetry would force this though, and it's just the most recent thing I'd seen with the cap. Currently, no one knowns what Python 4 will be like, other than it will be "less disruptive" than Python 3, and that it will come after 3.33. So I think it's perfectly fine to ignore "<4" as it only purely incorrect for anyone to use it now. Certainly don't make it infectious. This Python limit is special, though, unless you can also solve for the Python version, which Poetry, PDM, etc. can't do. If you can't ever fix a solve by using this, what is the point of potentially breaking working solves? And in general, if any solver starts going back to older versions of a package with an upper cap (which they all do), they not going to arrive a better solve with an older package just because they find one without a cap. I feel most solvers go for mathematical correctness here when practically backsolving is only useful for lower bounds, and is at best expensive (for perfect metadata) and at worst wrong (for un-editable metadata). Do keep in mind, though, I work on a total of 0 solvers. :) |
sure, it's not that you're wrong about poetry - but you are wrong to assume that everything is poetry's fault! I agree that most solvers go for correctness, I agree that it's plausible that if the folk writing the spec had understood the consequences of that then they might have written it differently. But perhaps we disagree about the right resolution to that:
Maybe it would be interesting to have a solver take the other path and see what happens. |
What does this mean @henryiii? |
Sorry, didn't mean everything is Poetry's fault! I'll settle for "almost everything". :P Poetry and PDM solve for a range - that's why this is infectious. Say you have one dependency: numpy. But this is during the few weeks where they had
What would you suggest fixing? I have three options, I don't think we ever picked one though. And IMO it's still pretty squarely on the solvers to fix; an official statement that the upper bound is ignored or disallowed would help, of course. If I don't wan't to put a limit on my usage of Python, a solver should not force me to because my dependency does. I'm fine if it forces me to acknowledge that with
Just that Conda lockfiles have a pinned version of Python. Poetry, PDM, etc. have a range of Pythons. (TL;DR: the solution here may be affected by uv's Python installation plans ;) ) |
Regarding updating I'm also quite intrigued by the idea of "solving" for Python versions but haven't fully grasped what that would look like in practice. |
depends what exactly you think is broken! eg I think I have seen you assert that requires-python was never intended / should not be used for upper bounds at all (apologies if this is misrepresenting you, or you now think something different). In that case, one might propose something like: introduce
While related, I think this is a separate question from whether or not upper bounds and consequent backsolving were a good idea. eg per zanieb the published range and the locked range do not have to be identical, this is something that could - at least in principle - be addressed today without changes to specs. |
Yes, I have said both, though the former is not correct, the designers assumed it could be used for anything, it just turned out that it doesn't "just work" when you design solvers. Adding a cap (remember when it was designed nothing had it, so it was always "added") just causes backsolving to before the cap existed.
|
But anyway, I am curious to see what's chosen here, and eventually will try to restart the Python discussions. |
I think there's something to this but it doesn't quite work as-is. We include your own project in the lockfile, so that sets an upper limit on the
My guess right now is that we'll ignore upper bounds on declared |
This I take to mean treat python as another dependency and install it. uv pip install python==3.11 for example. And that user could specify,
and then treat requires-python same as adding a dependency constraint on python itself and be able to install/upgrade/downgrade python like any other library. That's what makes conda special. Conda can install python packages, but it can also install non-python packages and even python interpreter itself. |
👍 Yeah, "solving" for Python would mean you end up solving for a specific Python version (as opposed to solving for a range, I guess). |
## Summary This PR modifies our `Requires-Python` handling to treat `Requires-Python` as a lower bound. There's extensive discussion around this in #4022 and the references linked therein. I think it's an experiment worth trying. Even in my own small projects, I'm running into issues whereby I'm being "forced" to add a `<4` upper bound to my `Requires-Python` due to these caps. Separately, we should explore adding a mechanism that's distinct from `Requires-Python` to enable users to declare a supported range for locking. Closes #4022.
See: #4021 (comment)
The text was updated successfully, but these errors were encountered: