Document less conservative Python support policy

[astrojuanlu]

Description

Supporting old Python versions is important for corporate users that want newer features, but adds toil on the Kedro team. Maybe we could explore adopting a less conservative Python support policy.

Context

We are among the last ones to drop support for old Python versions, and this creates problems at the tail of the support window.

For example, pip-tools recently dropped support for Python 3.7, and it's affecting us:

#2813

Notice that, whatever we do, should be starting on 0.19, or on 0.19+.

Also important to bear in mind how this interacts with other issues:

• The fact that upgrading Kedro is hard Create migration content for users updating their Kedro version #1887
• The fact that we don't have a fixed or very predictable release cadence

Possible Implementation

https://numpy.org/neps/nep-0029-deprecation_policy.html

Possible Alternatives

NEP 29 might be a tad too conservative. Maybe something more specific, like

Once a Python version is EOL, we're removing it from our CI, and we do not guarantee support for it going forward.

Which is maybe the smallest advance that can be made from the current position, but would still go a long way.

[datajoely]

I think it's perfectly reasonable to drop support once Python version EOL is reached since all enterprise users will be barred from using it by then. Also if the PSF has dropped this, in my view this isn't a breaking change for Kedro.

I'd also look to build some sort of shelf life for major versions of Kedro - like do we have a policy of when we continue to support Kedro 0.16.x and 0.17.x projects especially with 0.19.x on the horizon

[astrojuanlu]

I'd also look to build some sort of shelf life for major versions of Kedro - like do we have a policy of when we continue to support Kedro 0.16.x and 0.17.x projects especially with 0.19.x on the horizon

That's an interesting side question I've been pondering recently too. On first read it might look like we'd be adding burden on the team, but the truth is that we're now supporting all Kedro versions - I don't remember telling anybody "sorry, your Kedro version is too old, please upgrade". So this would be more about making explicit what versions do we offer open-source support too (while keeping in mind that we'll probably keep supporting old versions for our internal clients for a much longer time).

Again, this interacts with how difficult or easy it is to upgrade #1887

[deepyaman]

NEP 29 might be a tad too conservative.

I was going to suggest NEP 29 (and think I brought it up in some call a couple weeks back), but saw you already have here. I'm personally in favor of adopting NEP 29.

While one could argue that we're not supporting as broad a range of Python versions as PSF, people realistically need to upgrade their Python version to use newer releases of most libraries they use in their pipelines--Matplotlib, NumPy, pandas, Ibis, Dask, xarray, and the list goes on. NEP 29, not PSF support, has become the ubiquitous support policy among Python data libraries.

The notable exception is perhaps PySpark, but how many teams only use Spark with Kedro AND would be unwilling to upgrade at least to a version of Python released within the past 3 years? Perhaps some enterprises, but if they're so slow to upgrade other stuff, why would they always need to be on latest Kedro?

[astrojuanlu]

PyTables just dropped support for Python 3.8 in version 3.9.0 PyTables/PyTables@ae1e60e

[astrojuanlu]

More PRs are surfacing dependencies that have dropped Python 3.8 kedro-org/kedro-plugins#355, kedro-org/kedro-plugins#165

I think maintaining 3.8 support in framework won't be hard, but doing so for datasets definitely will.

[merelcht]

How about we first adopt NEP 29 for kedro-datasets and for kedro treat removing python version support as non-breaking when it's no longer supported by PSF?

[astrojuanlu]

• Adopt NEP 29 for kedro-datasets 👍🏽
• Treat removing Python version support for kedro as non-breaking when it's no longer supported by PSF 👍🏽

If no one else objects, we have a plan 😄

[datajoely]

💪 fundamentally we shouldn't be encouraging anyone to use a version of python which isn't getting security updates

[astrojuanlu]

Opened kedro-org/kedro-plugins#404 for plugins.

For framework, for now we only need to document the policy. We will be supporting Python 3.8 until October 2024, as per https://peps.python.org/pep-0569/ (see also https://devguide.python.org/versions/)

[merelcht]

Completed in #3262