Determine Python requirement boundary based on supported Requires-Python range

[charliermarsh]

Rather than writing the user's Requires-Python to the lockfile, we could instead write the minimum bounding Requires-Python of all the requirements. Or we could even omit it and use the Requires-Python of each package.

[charliermarsh]

I'm now somewhat uncertain that this makes sense, because we include the project itself in the lockfile, and so the range will be at most the range of the project itself.

[zanieb]

I think that's okay; the goal is to write the narrowest bound, right? That it's "at most the range of the project" makes a ton of sense i.e. you can constrain project's Python requirement more narrowly than its dependencies.

[charliermarsh]

I think it will always end up being exactly the bound of the project, right?

[charliermarsh]

Because we require that all dependencies have bounds that are >= the bound of the project. So the project is already the narrowest bound, by definition.

[zanieb]

Ah I was thinking we'd drop the requirement that the project's python bound is narrower than all its dependencies'.

It feels incorrect to require that a project's Python requirement is narrower than all of its dependencies unless they're all pinned (i.e. in a lockfile) because otherwise you're overly constraining against future versions of the dependencies.

Imagine if I'm developing a library and a new version of a dependency gets published with a relaxed Python requirement. If that new dependency version is within my published constraints, the supported Python range should expand but uv will have forced me to narrow my published supported Python range to the previous value.

[charliermarsh]

Sorry, can you give a concrete example?

[zanieb]

Henry's comment gives a concrete example, I'll modify it here.

Say your project has one dependency foo>=0.2.0 with one published version 0.2.0 which requires python>=3.9,<3.13. Poetry, and uv today, will force you to change your project's Python requirement to python>=3.9,<3.13 or narrower. However, if your project is being published as a Python library, that's not ideal. For example, if foo releases a new version 0.3.0 which requires python>=3.9 (drops the upper bound) or python>=3.9,<3.14 (adds explicit support for a new version) users of your library should be able to use foo==0.3.0 and python==3.13 but they cannot until you publish a new version of your project. In this scenario, a solver forced a constraint to be added to the published package's Python requirement to facilitate creation of a lockfile. Instead, the lockfile should contain the narrower constraint and the package metadata should be allowed to be broader. When generating a lockfile we have pinned versions for all dependencies and we know all of the Python requirements will be constant so the narrowest intersection is fixed. However, when publishing a project without pinned dependency versions, we don't know if the Python requirements will be relaxed in future releases, changing the narrowest intersection of the bounds.

[charliermarsh]

Ahh, I don't think that's solved by the thing in this issue, though. I thought that was solved by other things:

1. Avoiding requiring users to set an upper bound in the first place.
2. Separating the Requires-Python of the package from the supported Python range for installing it (from a lockfile). That is, the Python requirements for installing the project from the lockfile should not be coupled to the supported Python versions of the package itself.

When locking, we do need some user-defined range of "Python versions you want to lock for"; and then when solving, we do need to ensure that all the packages we include are supported on all of those versions. I think the critical thing, though, is that the "Python versions you want to lock for" should not be coupled to the Requires-Python of the project itself -- they should be different concepts, right? Like, the versions you locked against should not impact the versions that consumers of your package can use? But today, that is the case in Poetry.

[charliermarsh]

Like, maybe the code in the project itself only requires Python 3.7. Great. So you put Requires-Python = ">= 3.7" in your pyproject.toml.

But when locking, that might too permissive. You might want to lock against a narrower range, so that you're not required to use versions that support Python 3.7. This suggests that you want some other control that isn't Requires-Python.

[zanieb]

"Python versions you want to lock for" should not be coupled to the Requires-Python of the project itself -- they should be different concepts, right?

Yes this sounds good to me. I think we're on the same page. I don't think we should suggest constraining Requires-Python to match your dependencies. Is there a way we can start with Requires-Python then infer the narrower constraint required for resolution from the dependencies and write that to the lock file?

[charliermarsh]

Are you suggesting "enforce Requires-Python while locking (as we do now), but then write the less-strict constraint based on all your dependencies"? Or "ignore Requires-Python while locking, and then compute the constraint from the dependencies"?

[zanieb]

I'm thinking about that error case we raise saying to narrow your Requires-Python. Let's say we have some alternative field Lock-Python that has the same semantics but is null by default. Is there a way we can do the narrowing for the user (and write it to the lock file) instead of throwing an error saying you need to populate Lock-Python?

I think this is different than both of the options you presented. Though, I am confused by:

enforce Requires-Python while locking (as we do now), but then write the less-strict constraint based on all your dependencies

Isn't the failure case that the constraint of your dependencies is more strict than Requires-Python? I don't think I'm suggesting ignoring Requires-Python just allowing it to be "too broad". If Lock-Python is set, then we'd just make sure it's narrower than Requires-Python when locking.

[charliermarsh]

Isn't the failure case that the constraint of your dependencies is more strict than Requires-Python?

Yeah. In that case, I was suggesting that you have a successful resolve, and then we try to figure out how broad the Requires-Python of your dependencies could be. If we succeeded, then it must be at least as broad as your own Requires-Python, maybe broader. IDK what we would do with that though.

[charliermarsh]

What you've described above makes sense, though. If we fail, tell the user, "Hey, we could only create a universal environment for Python 3.10 and later, so set that as your lock constraint." And critically "your lock constraint" is not Requires-Python but something else.

[konstin]

But when locking, that might too permissive. You might want to lock against a narrower range, so that you're not required to use versions that support Python 3.7. This suggests that you want some other control that isn't Requires-Python.

If you'd lock for let's say >=3.8, it means ou can't run test for python 3.7 with the lockfile. The lockfile would lose a part of it's universally by breaking down for a section of possible environments.

If we want to achieve higher versions with more recent python versions, we'd need to apply fork-and-merge when encounter a requires-python bound splitting our supported range. When we lock for >=3.7 and encounter a python >=3.9 numpy, we resolve once for >=3.9 and once for <3.9,>=3.7, locking two different numpy versions depending on the python you install on. These would be two resolver modes: A python-universal mode that enforces >=3.7 support for all your dependencies, and a python-specific mode that gives you the highest version per python version.

While problems with using features from newer versions that will not be supported on older versions could be irritating when working on more complex projects, i believe the main problem is performance, we potentially get a lot of splits.