Each project should have autonomy over Python version restrictions

[aaronsteers]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the FAQ and general documentation and believe that my question is not already covered.

Background

Poetry has been mainstream for a while now, and the world is a better place for it... but what we're observing is that there are a world of really great and stable libraries out there which aren't actively maintained, and will always be slow to bump their max-python-version constraint, but are nevertheless working great on newer versions of Python - to the joy of developers. Yet, despite the fact that libraries just work with no changes, the project consuming these projects have zero resource to actually use them - unless we fork the project and bump the version constraints ourselves.

It is right and proper that every library builds in a python restriction of <[whatever-is-not-out-yet], but that best practice does not actually mean that the library won't work on the newer version. Months later, everyone is doing exactly the right thing: the python-core team are adding new features that are generally backwards-compatible, and the library was built using features that are all still supported in the latest version.

Feature Request

When all is working perfectly, can we allow consumers of perfectly-compatible dependencies to just 'tell' poetry that the library in question is indeed compatible?

The parent project is taking responsibility for end-to-end testing and presumably can also take responsibility for declaring that a sub-project is indeed compatible-enough with the newer version of Python in order to not force a hard failure during dependency resolution.

It is correct for each repo to declare which versions have and have not been tested - but it's also right that consumers should be able control their own declarations of python compatibility.

Proposal

Allow pyproject to override the max python version of a specific dependency, or (since our project already defines the desired python version constraint), simply specify ignore or inherit as the python constraint.

Sample case:

• Add support for python 3.11 MeltanoLabs/tap-github#162 (comment)

[neersighted]

Partially a duplicate of #697; especially as it makes more sense to "solve" dependency overrides generically if we do it at all. That being said, this is also more the wrong approach to the discussion at #6884 -- Poetry is well-behaved and works just like the rest of the ecosystem here, according to the semantics of Requires-Python.

Poetry however, has had a big part in encouraging upper bounds in Requires-Python, especially as we tie the range of acceptable Pythons for your project (your ask here) to your own Requires-Python, which results in the situation of the most restrictive set of Python versions being 'contagious' in the ecosystem.

Addressing this is fundamentally related (and incrementally solves your problem) without the inherit danger and support burden of #697, all while potentially pushing Python packaging in a "better" direction. As such, I'm going to close this as both duplicate/wontfix, but I would encourage you to put in an issue for the discussion that started in #6884 -- we could certainly continue that line of thinking and decide if that is a change we should consider/plan on (and alternatively, @henryiii, I know you feel strongly about this, though you're not familiar with Poetry internals. Would you be willing to open an issue with a general discussion of the problem, even if you can't formulate a specific proposal for our codebase?).

[aaronsteers]

@neersighted - The issues linked as related are all closed:

• Warn if newly introduced upper version bounds lead poetry to install older version of a package #6884
• Ability to override/ignore sub-dependencies #697

I read #697 before opening this and opened it specifically because that one had been closed.

I'm not sure of the path forward here - and it seems important to the future of Python and Poetry that we aren't dead-ending perfectly good libraries by default. Suggestions?

[neersighted]

#6884 is closed because it is not something likely to be accepted; however, the discussion there is about the real problems the user is trying to solve, not the proposed (mis)feature itself. Like I said, I would encourage you or anyone else interested to open an issue about Poetry's handling of Requires-Python and our impact on the ecosystem.

You real ask is that Poetry no longer "virally" links the Requires-Python of a dependency tree all the way up to the top -- if we make changes to how we generate package metadata (and disconnect it from the solver) as partially discussed in that issue (and partially discussed in Henry's blog posts), your ask will come true.

#697 is closed because the feature was rejected by the core team as not desirable/compatible with Poetry's design. Clearly there is a lot of interest in it, even if we are wary. As such, no one who works on Poetry regularly has taken any steps to implement such a feature, but we are willing to review a design and/or implementation if someone presents something that is well thought out, doesn't cause increased maintenance for the solver, and is of tested/mergable quality. So far, no one has seriously attempted to do so.

If anyone is interested in implementing something that "solves" #697, they are free to open an issue proposing a (comprehensive) design (and this is better than a PR as you may waste a lot of time writing code that's not usable if the design is considered bad). We have also suggested that those interested can get help on Discord to better understand Poetry/working on Poetry; no one has yet to take advantage of this either.

#697 will remain closed as a rejected feature and have duplicates assigned to it, but there has not been an outright embargo on the proposed feature if someone wants to attempt to design and realize it. The issue is closed as it is not on Poetry's roadmap nor the radar of anyone who regularly works on the project.

[aaronsteers]

@neersighted - Thanks very much for your thoughts and input here. It sounds like automatically defaulting to < constraints for any and all future unreleased Python constraints should become an anti-pattern in the Python community.

• Python version constraints: common wisdom vs pragmatic realities MeltanoLabs/Meta#24

To summarize the above, if there is no 'feature relief' from the Poetry build tools or other related packages in the build chain and/or dependency-resolution chain (for instance, allowing packages and applications to override stale runtime constraints of their dependencies), then we probably should suggest changes to accepted best practices: we should recomment Python version restrictions based only on future major versions and not for future minor unreleased minor versions.

Current best practices recommend constraining future Python versions to unreleased minor version, whereas most other dependency constraint best practices would recommend binding to major versions for dependencies.

Until and unless Poetry (and related build/install tools) are able to decouple the treatment of dependency version constraints from runtime version constraints, then the Python community probably needs to start decoupling the concept of 'tested Python runtimes' from 'constrained Python runtimes'.

While we can try to declare that libraries and applications are tested on the latest Python version, it should not follow that the library or application is restricted from be installed on future Python runtimes - at least not for minor Python version increments.

Wdyt?

[neersighted]

I don't think fixating on major vs minor Python versions is useful -- maybe @brettcannon or someone else from CPython wants to weigh in, but the general stance of the core contributors has been that the major version number is as meaningful as Linux's (that is to say, there will be no more transitions like 2 -> 3 and it's mostly meaningless).

It's best to think of Python versions as monotonically incrementing with moderate breakage over a multi-version timeline, with changes being primarily evolutionary.

That all goes to say, that best practices are likely this:

1. do not constrain Requires-Python based on a test matrix, but instead hard incompatibility only (e.g. Python extension ABI incompatibility)
2. Poetry must still solve on a limited range of Pythons, but those should not drive Requires-Python

I'd love to pick @henryiii and @dstufft's brains here (and they already weighed in a bit on Discord when you brought this topic up there) to see if they have any more thoughts.

[aaronsteers]

That all goes to say, that best practices are likely this:

1. do not constrain Requires-Python based on a test matrix, but instead hard incompatibility only (e.g. Python extension ABI incompatibility)
2. Poetry must still solve on a limited range of Pythons, but those should not drive Requires-Python

This sounds right to me. 👍

Admittedly, there's a lot of background I don't have on the internals, nuances, or side affects - but this aligns with how I'm coming around on this topic.

[brettcannon]

I don't think fixating on major vs minor Python versions is useful -- maybe @brettcannon or someone else from CPython wants to weigh in, but the general stance of the core contributors has been that the major version number is as meaningful as Linux's (that is to say, there will be no more transitions like 2 -> 3 and it's mostly meaningless).

We don't even know what a 3 -> 4 transition would look like as we haven't even agreed if we will ever even do one. 😁 All I can say is we will never do anything as drastic as 2 -> 3 and there will be a migration path for many years if we ever do a 3 -> 4 that has significant changes.

1. do not constrain Requires-Python based on a test matrix, but instead hard incompatibility only (e.g. Python extension ABI incompatibility)

This is definitely the advice I give anyone: just specify what you definitely won't work with and then leave it open-ended otherwise as you may work with other versions.

[aaronsteers]

Thanks, @brettcannon ! Super helpful. 🙏

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.