Skip to content

NEP 29: Python version strategy

Jump to bottom Edit New page
Matthias Köppe edited this page Jul 9, 2023 · 77 revisions

This page discusses the specifics of adopting NEP 29.

Undisputed facts

What NEP 29 is

NEP 29 defines a schedule of supporting versions of Python (and of NumPy/SciPy) that is faster than the end-of-life schedule of the Python project and is intended to be suitable as a standard for the scientific Python community. It is motivated by the fast pace of progress in scientific software development and the desire to be able to use new features of Python (and of NumPy/SciPy) in published projects.

NEP 29 includes a sample statement that it suggests projects to adopt in their development guidelines:

This project supports:

  • All minor versions of Python released 42 months prior to the project, and at minimum the two latest minor versions.
  • All minor versions of numpy released in the 24 months prior to the project, and at minimum the last three minor versions.

In setup.py, the python_requires variable should be set to the minimum supported version of Python. All supported minor versions of Python should be in the test matrix and have binary artifacts built for the release.

Minimum Python and NumPy version support should be adjusted upward on every major and minor release, but never on a patch release.

In this wording, "major", "minor", "patch" refer to the standard versioning scheme MAJOR.MINOR.PATCH such as SciPy 1.11.1. SageMath only uses MAJOR.MINOR (with rare exceptions, the most recent being 7.5.1 released in 2017).

There is also SPEC 0, which covers more packages and appears intended to supersede NEP 29. SPEC 0 uses the term "feature releases" instead of "minor releases" with the same intended meaning.

Which projects follow NEP 29

NEP 29 is followed by many scientific packages such as SciPy, Matplotlib, IPython, Jupyter, Pandas, scikit, astropy, cuda, cirq, jax, pytorch among others.

TBD: Expand with examples on what "following NEP 29" means for these projects.

  • SciPy: Minor releases every 6 months, followed by patch releases; policy: "For the general policy on dropping support [...], see NEP 29. The final decision on dropping support is always taken on the scipy-dev mailing list."

    • Last such scipy-dev posting seems to be: dropping Python 3.6, Aug 2020. Actual discussions happen on PRs that change setup.py python_minversion
    • scipy/scipy#13081 dropping Python 3.6 opened/merged Nov 2020 (NEP 29: Jun 2020) in main branch for anticipated Dec 2020 release of SciPy 1.6.
    • scipy/scipy#14655 dropping Python 3.7 opened/merged Aug 2021 (NEP 29: Dec 2021) in main branch for anticipated Feb 2022 release of SciPy 1.8. (Continued to make patch releases 1.7.2, 1.7.3 in Nov 2021 that kept Python 3.7 support.)
    • scipy/scipy#17959 dropping Python 3.8 opened/merged Feb 2023 (NEP 29: Apr 2023) in main branch for anticipated Jun 2023 release of SciPy 1.11.

  • Matplotlib: Minor releases 1-2 times per year, followed by patch releases. policy: "Matplotlib follows NEP 29]." Latest release 3.7.2 (Jul 2023) still suports Python 3.7. matplotlib/matplotlib#24919 dropping Python 3.8 opened/merged Jan 2023 (NEP 29: Apr 2023) in main branch leading to a future 3.8 release. (Continued to release 3.7 in Feb 2023, followed by patch releases.

There are not many other documented policies. As said above, most scientific python projects follow NEP 29. Most projects in web development (e.g flask) seem to drop a version once it reaches EOL. Machine learning projects follow a similar EOL policy (e.g. tensorflow) or roughly follow NEP 29 (scikit-learn). Some end-user applications have even stricter version constraints then NEP 29 (e.g. home-assistant only supports the two latest minor releases).

The conda-forge distribution dropped Python 3.7 in Oct 2022 (NEP 29: Dec 2021) and still supports Python 3.8 as of Jul 2023. (The conda software itself dropped Python 3.7 in May 2023.)

What it means for SageMath to "support" a version of Python

As "SageMath" can refer to both the distribution and the library, this may require clarification.

  • The Sage distribution supports a version of Python when its configure script accepts an existing installation of this version ("system Python") as the base for its virtual environment; currently it accepts 3.8.x–3.11.x. (We can ignore sage-bootstrap-python, which has a much wider version range.)
  • The Sage distribution ships a version of Python within this range and installs it when no suitable system Python is found, or on request.
  • The Sage library declares its support for a range of Python versions in setup.cfg, and all Sage library code works with these versions of Python.
  • As the main purpose of the Sage distribution is to provide a working Sage library, the Sage distribution's Python version range is always a subset of the Sage library's Python version range.
  • In all past practice, both version ranges have been the same.

What happens when we drop support for a version of Python

Dropping support means here that the relevant Github Actions workflows are removed, python_requires in setup.cfg is increased and similar changes. After this change, the older system Python will be rejected and sage-the-distribution will install a newer version (or the user can use other means to provide a supported system Python such as conda).

An example for a PR dropping Python 3.8 can be found at https://github.com/sagemath/sage/pull/35404.

Past and current practice of the SageMath project

Schedule (with comparison to NEP 29)

  • The capability of the Sage distribution to use a system- (or user-) provided Python interpreter is relatively new; it was released as part in Sage 9.1. Until then, the Sage library had supported exactly one Python minor version, namely the one that the Sage distribution shipped. In an upgrade ticket (such as #25680 to 3.7.x), the library would be ported to the new version, support for the old version abandoned, and the python3 package upgraded.

  • Sage 9.1 (May 2020) only supported using the same minor version, 3.7.x, of Python as was shipped in the Sage distribution. (It did not conform to NEP 29, which only dropped support for 3.6.x in Jun 2020.)

  • Sage 9.2 (Oct 2020) restored support for system Python 3.6.x and added support for Python 3.8.x and 3.9.x. (Thus it conformed to NEP 29 but additionally supported 3.6.)

  • Sage 9.4 (Aug 2021) dropped support for Python 3.6 again. At this time, the EOL of Python 3.6 and the NEP 29 drop of Python 3.7 were just a few months away (Dec 2021).

  • We dropped support for Python 3.7 a bit faster, in Sage 9.7 (Sep 2022); the EOL of Python 3.7 was Jun 2023. It would have been conforming to NEP 29 to already drop it in Sage 9.5 (Jan 2022).

    Comment by Tobias: This is a bit misleading. It was probably not clear at the beginning of the release cycle that 9.5 will be released in Jan 2022 and thus 3.7 would have most likely be dropped at the beginning of the release cycle of sage 9.6

    Comment by mkoeppe: I don't recall any expectation that 9.5 would be ready before the Dec 2021 NEP 29 drop date, but there is little value in debating this. But I think this illustrates that there is a need of further clarification of Proposal 1; and an elaboration of what it means for other projects to "follow NEP 29" (marked TBD above). NumPy/SciPy have about 6 months of development cycle for a new minor release; they drop support on their development branch before the NEP 29 drop date.

  • Python 3.8 was released not 1 year after Python 3.7, but rather 1 year and 4 months; likewise for the EOLs. The same drop speed (relative to EOL) that we used for Python 3.7 would mean to drop Python 3.8 in a Sage release with release date in December 2023 or January 2024. (NEP 29 dropped Python 3.8 in Apr 2023.)

Factors

The main driver for Python version support changes in SageMath has been the desire to upgrade key upstream packages in the SageMath distribution, either

  • for needed platform support (CPU architecture variants, OS versions, compilers),
  • for supporting a new Python version,
  • for interest in new features (e.g., the upgrade of SciPy, released in early 2021, in Sage 9.4, was motivated by a specific feature), or
  • for the benefit of downstream packagers of Sage,

or a combination of these. There is a very wide range of how complicated a package upgrade is. Because of the wide platform support that SageMath offers, we often run into portability problems that upstream is not aware of. As an example on the upper end, the upgrade ticket of SciPy to 1.9.x was opened in Jun 2022 (when the scipy rc was released) but it took until Nov 2022 for it to be ready for merging.

On the other hand, interest in new language/library features of new Python versions has been very limited.

The following factors have been considered:

  • What is provided as the default Python on various distributions.
  • What is available as an installable package on various distributions.

TBD expand, incorporate material from last section

Decision Making

List of PRs with changes to supported system version of python3 and other packages (git log --first-parent build/pkgs/*/{spkg-configure.m4,install-requires.txt}) since 2020:

  • #34857 Giac >= 1.9: opened Dec 2022, 3 participants; merged Jan 2023.
  • #34268 R >= 3.5: opened Aug 2022, 6 participants, elements of the discussion: the dropped support for old LTS platforms, whether to do >= 4.0, user base of SageMathCell; merged Oct 2022.
  • #34266 GCC >= 8: opened Aug 2022 (prereq of numpy upgrade), 2 participants, held to beginning of 9.8 cycle; merged Sep 2022.
  • #33316 GCC >= 6.3: opened Feb 2022 (as prereq of numpy/scipy/networkx, givaro/fflas-ffpack/linbox, normaliz/e-antic upgrades discussed since 2021), 4 participants, elements of the discussion: Debian 9 LTS supported to Jul 2022; merged Jun 2022 (early 9.7).
  • #32937 Python >= 3.8: opened Nov 2021, 3 participants, elements cited: Python 3.7 EOL, debian-buster Python versions and EOL, NEP 29, numpy/scipy/networkx upgrade; merged May 2022.
  • #33116 FreeType >= 2.8: opened Jan 2022, 2 participants; merged Mar 2022.
  • #32580 OpenSSL >= 1.1.1: opened Sep 2021, 3 participants; merged Oct 2021.
  • #31069#31525 FLINT >= 2.6: opened Dec 2020/Mar 2021, "too disruptive for Sage 9.3", marked "merge in 9.4 only"; merged Jul 2021.
  • #30551 Python >= 3.7: opened Sep 2020, 5 participants, elements of the discussion: NEP29 schedule, SciPy/NumPy/NetworkX drop schedule, Ubuntu 18.04's default Python, availability of newer Python in installable packages or pyenv; merged May 2021.
  • #29462 nauty >= 2.6: opened Apr 2020, 2 participants; merged Apr 2020.

List (but not quantification) of costs, risks, and benefits of version support policies

  • the cost for developers to stay informed about upstream projects, platforms to support, and their timelines
  • the cost for developers to participate in technical discussions
  • the cost for developers and release manager to monitor whether SageMath continues to work with the versions of upstream projects and platforms that are intended to be supported
  • the cost for developers / release managers to maintain more than 1 branch from which releases are made
  • the cost for developers to fix problems that newly arise with old supported versions of upstream projects / platforms
  • the cost for developers to prepare SageMath to work with new versions of upstream projects and platforms
  • the cost for developers to write the more complicated code that is necessary for the versions that are intended to be supported
  • the cost for developers and community members to give installation help or respond to other support questions
  • the cost for users to upgrade their OS distribution and to familiarize themselves with the upgraded distribution
  • the cost for users to build packages from source that could previously be taken from the system
  • the cost for users to port custom software to a newer Python version on a schedule

(more TBD)

Why we should set a policy regarding Python version support

  • To document the SageMath project's intentions to be a member of the scientific Python community.

  • To manage expectations regarding compatiblity; in particular to make clear that the project does not have the obligation to support all Python versions all the way to their EOL (which has never been the project's practice).

  • As the SageMath project has been opening itself to the wider Python community, setting clear project policies can reduce the risk that participants in a discussion engage in abusive conduct.

Proposals

There are two proposals of adding text to the developer documentation as a policy on which Python versions are supported.

Proposed formulation 1 (prescribes the faster NEP 29 drop schedule for the Sage project)

We follow the guidelines outlined in NEP 29 for dropping support for older versions of Python. Specifically, we support all minor versions of Python released 42 months prior to the project, as well as the two latest minor versions. If at the anticipated release date of the next Sage version, a Python version is no longer included in this set, we usually drop support for it early in the release cycle.

The drop schedule for Python versions is as follows:

Support for Python 3.9 (initially released in October 2020) is dropped in April 2024.

Support for Python 3.10 (initially released in October 2021) is dropped in April 2025.

While we make our best effort to support the latest version of Python, we cannot guarantee that our software works with the latest version at all times. This is due to the fact that new versions of Python may introduce breaking changes or other issues that we cannot anticipate. By following this policy, we aim to provide our users with a stable and reliable software experience, while also keeping up with the latest developments in the Python ecosystem.

What this means in practice

As with every policy, the drop schedule proposed above is not absolutely binding. In particular, someone needs to find the time and motivation to actually create a PR that drops support and this PR needs to go through the usual review process. In exceptional cases, we can deviate from the drop schedule after a discussion on the mailing list.

The NEP 29 schedule is about one release cycle faster than the previous drops (e.g. Python 3.7 support was dropped in Sage 9.7 while according to NEP 29 it would have been Sage 9.6). (TBD: I don't think estimate is correct; see timeline above. –mkoeppe)

Proposed formulation 2 (describes the NEP 29–compliant current practice of the Sage project)

The SageMath project intends to support at least the following versions of Python: all minor versions of Python released 42 months prior to the project's release, as well as the two latest minor versions. Hence SageMath intends to conform to NEP 29.

While we make our best effort to support the latest version of Python, we cannot guarantee that SageMath works with the latest version of Python at the time of the Python release. This is due to the fact that SageMath is downstream of an unusually large number of packages, and thus our porting efforts are sometimes blocked until an upstream project has completed its porting to the newest Python release.

We emphasize that the SageMath project uses the interpretation that NEP 29 only defines the minimum supported Python version. As the NEP 29 document contains example language such as "Support for Python 3.9 (initially released in October 2020) is dropped in April 2024" that could be read as prescriptive, the SageMath project clarifies that it is invalid for a developer to demand that we drop support immediately at the stated time.

The SageMath project is fortunate to have dedicated contributors with the expertise and professionalism to discuss version support questions and reach well balanced decisions.

What this means in practice

The SageMath developers who engage in the maintenance of the Sage distribution and in downstream packaging of Sage in OS distributions are aware of the release schedules and version support policies of the upstream projects. Whenever a new version of a relevant package is on the horizon:

  • developers open an issue / PR for the upgrade of the package in the Sage distribution;
  • if the Sage library needs changes that add support for the new version, often these changes are put on a separate PR because it will be suitable for immediate merging; this is a dependency of the upgrade PR
  • if the package drops support for a Python version 3.x currently supported by SageMath, open an issue / PR titled "Drop support for Python 3.x" with no milestone set; this is a dependency of the upgrade PR
  • organize the benefits vs. costs of dropping the support on the "Drop support" issue / PR;
  • engage in a discussion that weighs the costs vs. benefits, specific to a given point in time / a planned SageMath release 1s.t;
  • when the development cycle leading to the release 1s.t has started, label the PR "positive review"; then the Release Manager's scripts will merge the PR in the next batch.

As with all PRs in the SageMath project, the burden of building a case that the PR is an improvement for SageMath is on the PR author. (TBD: expand and refer back to current practice) In the case of PRs that drop support for a Python version or system configuration, usually all participants in a discussion are actually in favor of dropping support at some point the future. The review discussion is regarding the specifics of the timing of it.

It is invalid to suggest that the burden is reversed, for example by demanding that the PR has to be approved and merged immediately because the work has been done already; or that the reviewer first has to build a detailed case against it; or to demand that the PR has to be approved because the discussion of it wastes developer time.

Discussion

(TBD: expand, cover prop 2. –mkoeppe)

Advantages of proposal 1

  • The current practice in Sage is to evaluate whether to increase the minimum version of Python supported at the beginning of each release cycle. With regard to such a practice, the NEP 29 policy remarks "As there is no objective threshold to when the minimum version should be dropped, it is easy for these version support discussions to devolve into bike shedding and acrimony." Sadly, an example of this can be found in the current discussion of dropping Python 3.8 support in https://github.com/sagemath/sage/pull/35404 with emotions running so high that sage-abuse had to step in. Adopting a version policy would prevent such discussions.

  • The main idea of NEP 29 is to have a community-wide standard. It is followed by many scientific packages (examples listed above). The adoption of NEP 29 will harmonize Sage's deprecation policy with these other major libraries.

  • The faster drop schedule will free developer resources (less systems to test) and potentially increase developer productivity as it allows us to use newer language features.

  • The faster drop schedule would force users to upgrade to newer Python versions and thereby profit from fewer bugs and security issues. It is however questionable if Sage should play this educator role.

  • One of the main goals of NEP 29 is to improve downstream project planning by having a community-wide standard. This is currently not very relevant for us as Sage is currently upstream of nothing except for some user packages. With the modularization effort, this may change in the future.

Disadvantages

  • The faster drop schedule might be inconvenient for users who rely on older Python versions. To some extend this is remedied by our python install package, and relatively straightforward upgrade paths on most system. One should also note that users relying on other scientific python packages are likely forced to upgrade anyway, since these other packages likely follow NEP 29.

  • By following a given policy, we would lose some flexibility.

  • The NEP 29 drop schedule is much faster than the EOL schedule of Python itself or what common Linux distributions provide by default. For example, Python 3.8 is supported until 2024-10, but NEP 29 already drops it 2023-04. On the other hand, adhering to the slow EOL schedule would prevent us to updating dependencies that follow NEP 29 or other faster drop schedules.

  • In Sage, we only have one line of releases. Hence users who want any bug fixes need to use our latest version. In contrast, just like Python itself, many other projects have at least two separate branches: A branch on which the cutting edge development takes place (new features etc.), and a branch from which maintenance updates are made. In principle, we could also use our master branch as a basis for similar hotfix releases.

  • Sage is downstream of lots of Python packages; before we can offer support for a new version of Python, we often have to wait until all or most of our dependencies provide support for that new version.

Alternatives

(TBD: It's not neutral to call it "Ad-Hoc"; "there are no objective evaluations" attacks an exaggerated standard; and nobody has proposed the two other policies other than as factors in decision making. Fold it into the description of current practices above. –mkoeppe)

Ad-Hoc version support

Evaluate whether to increase the minimum version of Python supported at the beginning of each release cycle (this is the roughly the current practice). That this is not working very well can be seen in the discussion of https://github.com/sagemath/sage/pull/35404. One of the main issues is that there are no objective evaluations for the negative impact that users will experience by the version support drop versus the positive gains we developer gain for easier maintainability, reduced overhead etc.

Following the version policy of other dependencies

Drop support of a Python version whenever one of our dependencies does so. Since many of the essential dependencies of sage are following NEP 29 already, this would effectively amount to a NEP 29 drop schedule shifted by a few months (the time it takes for the other dependencies to release a new version). Moreover, we would give control over the version policy to the version policy of dependencies. For example, if a dependency follows a very narrow Python version policy it might be preferable to still use older versions of that dependency rather than following a similar narrow Python version policy.

Following the default distribution of major Linux distributions

The policy could be to support the version of Python that ships by default in the latest Ubuntu LTS or CentOS/RHEL release. However, we would still have to standardize across the community which distribution to follow. By following the versions supported by major Linux distributions, we are giving up technical control of our projects to external organizations that may have different motivations and concerns than we do.

References

Links to the previous discussions regarding NEP 29 in the SageMath community:

Add a custom footer
Clone this wiki locally