Support for Python 3.13 #319

zanieb · 2024-09-06T13:48:42Z

A continuation of the wonderful work by @kpfleming in #264

Adds initial support for Python 3.13 using CPython 3.13.0rc1.

There are a few caveats and interesting details:

BOLT is disabled. There's a segmentation fault in the tests with the BOLT instrumented binaries. The BOLT optimizations are not critical, so we'll follow up on this separately. See more context.
mpdecimal is now built from source on Windows. We already did this in Unix builds, but in Windows we were still using the bundled library. The bundled library is no longer used upstream and it seemed prudent to switch though it will remain available until 3.15. See more context.
Apple cross-compilation is not available. I have a patch that adds support, but need to test it and it's not needed for these builds. See more context.
run_tests.py was removed upstream. We provide a compatibility script that calls the appropriate command still so that our distributions are stable. We may want to change how run_tests.py is declared in the distribution metadata though. See more context .

zanieb · 2024-09-06T13:55:46Z

cpython-unix/build-cpython.sh

+# testembed links against Tcl/Tk and libpython which already includes Tcl/Tk leading duplicate
+# symbols and warnings from objc (which then causes failures in `test_embed` during PGO).
+if [ -n "${PYTHON_MEETS_MAXIMUM_VERSION_3_13}" ]; then
+  patch -p1 -i ${ROOT}/patch-make-testembed-nolink-tcltk.patch
+fi


See zanieb#6 (comment) for more context

zanieb · 2024-09-06T13:59:02Z

cpython-unix/build-cpython.sh

+# ptsrname_r is only available in SDK 13.4+, but we target a lower version for compatibility.
+if [ "${PYBUILD_PLATFORM}" = "macos" ]; then
+    CONFIGURE_FLAGS="${CONFIGURE_FLAGS} ac_cv_func_ptsname_r=no"
+fi


See zanieb#6 (comment) for more context.

Python 3.13 only supports macOS 10.13+. The official build are built for 10.13+. So I'd think the macOS target version should be Python version dependent?

I think we're intentionally setting a lower target deployment for compatibility with older operating systems. We do this in various other places already and CPython 3.13 has code to behave correctly if ptsname_r is not available.

We could consider bumping the minimum target version, but I think that should be separate from initial suport.

https://pythoninsider.blogspot.com/2024/06/python-3130-beta-2-released.html?m=1

The minimum supported macOS version was changed from 10.9 to 10.13 (High Sierra). Older macOS versions will not be supported going forward.

Well, that's good to know. Regardless, I'm not sure that means that we need to drop support yet. I actually tried to bump the minimum version first and it didn't resolve the commented issue because ptsname_r is only available in 10.13.4 not 10.13.0. Changing to a minimum supported patch version is a larger change and one I'm hesitant about it. Separately, there's not support for conditional macOS target versions yet and I'd need to implement that. I'd rather get this out the door first.

cc @hugovk (if you're interested)

The comment about changing the minimum support macOS version from 10.9 to 10.13 does refer to the python.org macOS installer and not necessarily to CPython itself. (At the moment, we don't have an officially published support policy for which macOS releases are supported by CPython although we plan to make one.) But there is no intent to deliberately break support on older releases. The above patch, though, does point out a deficiency in CPython today as @zanieb notes, as there currently is no run-time test for the presence of ptsname_r, whose use was introduced fairly recently, which would be needed to maintain full weak-linking support on older systems. It would be good to open a cpython issue about that.

Thanks for weighing in @ned-deily!

Here's an upstream issue python/cpython#123797

I would be fine dropping 10.9 support on 3.13. I think targeting what the official installers support is good enough.

If some random user on 10.9-10.2 complains, they can always build things from source. Presumably they've been doing that for years because the ~entire world has dropped support for their OS version years ago.

zanieb · 2024-09-06T13:59:35Z

cpython-unix/build-cpython.sh

+    # In Python 3.13+, the musl target is identified in cross compiles and the output directory
+    # is named accordingly.
+    if [ "${CC}" = "musl-clang" ] && [ "${PYTHON_MAJMIN_VERSION}" = "3.13" ]; then
+        PYTHON_ARCH="x86_64-linux-musl"
+    else
+        PYTHON_ARCH="x86_64-linux-gnu"
+    fi


See zanieb#6 (comment) for context.

src/validation.rs

charliermarsh · 2024-09-06T16:04:32Z

cpython-windows/build.py

-            out_dir / "python" / "build" / "run_tests.py",
-        )
+        # CPython 3.13 removed `run_tests.py`, we provide a compatibility script
+        # for now.


How is this used? Is it just something we invoke later on?

We don't invoke it. It's placed in the distribution archive for consumers and there's a run_tests field in the PYTHON.json metadata which points to this script. This is briefly mentioned in the pull request summary, but I think we'll probably open an issue to just remove this in the future, allow it to be null in the metadata, or provide a command in the metadata instead of a path? All of these alternatives are technically breaking changes to the distributions though.

For tracking, here was the first commit I did to get past this change 5afde6d — here we resolve the TODO there and provide the compatibility script.

I tested the script on Linux and macOS, but not Windows.

cpython-unix/extension-modules.yml

zanieb · 2024-09-06T16:36:45Z

I have a fix for the Windows Python 3.8 failures and have addressed the review comments but will wait to push until I see if anything else fails.

Co-authored-by: Zanie Blue <contact@zanie.dev>

indygreg

I skimmed this and it looks great!

I didn't thoroughly review the new variant CPython patches. Those can be tricky as the CPython build system is a moving target and we're doing some questionable things with some of these patches. But the Rust distribution validation has gotten pretty robust over the years and if things pass, I feel pretty confident about correctness.

FeodorFitsner · 2024-09-07T16:17:45Z

Python 3.13.0rc2 is out.

Is 3.13 pre-release binaries going to be published to releases or should be downloaded from build artifacts?

zanieb · 2024-09-07T16:45:37Z

@FeodorFitsner we'll probably release them soon.

I'll work on rc2 support next week.

indygreg · 2024-09-07T16:48:37Z

There was a joint release of all supported CPython versions a few minutes ago. Includes a ton of security fixes. We should try to get a release out soon.

https://www.python.org/downloads/release/python-3126/

zanieb · 2024-09-07T16:52:09Z

👍 I'll work on all those next week then haha

zanieb · 2024-09-07T16:57:47Z

Also of note

Also, as mentioned in the previous release of 3.12, this release drops support for macOS versions 10.9 through 10.12. Versions of macOS older than 10.13 haven't been supported by Apple since 2019, and maintaining support for them has become too difficult. (All versions of Python 3.13 have already dropped support for them.)

indygreg · 2024-09-07T17:02:16Z

Yes this project has had no problem building them. Actually testing on 10.9 and verifying builds work is another story, of course.

zanieb · 2024-09-07T17:06:39Z

As a heads up to avoid duplicate work, I started bumping the patch versions #322

ned-deily · 2024-09-07T18:41:50Z

Yes this project has had no problem building them. Actually testing on 10.9 and verifying builds work is another story, of course.

FTR, the release notice was a bit terse about that. We currently don't have any problem building Python for 10.9 and manually testing it. In fact, it's because of that we discovered at the last minute during the previous, 3.12.5, release, that the latest release of pip 24.2 does not work on older macOS versions and that was a release blocker. Since it was at the last minute, we added a partial workaround to try to pin pip to its previous release on older systems but there have been and will be more issues like this so we decided to bite the bullet and bump to 10.13 for this current 3.12 release.

zanieb commented Sep 6, 2024

View reviewed changes

zanieb force-pushed the zb/3.13-rc1 branch from d6a55fa to a9d1fb5 Compare September 6, 2024 14:08

This comment was marked as outdated.

Sign in to view

zanieb force-pushed the zb/3.13-rc1 branch 2 times, most recently from 25a22c7 to d3da7b3 Compare September 6, 2024 14:48

kpfleming mentioned this pull request Sep 6, 2024

Initial attempt at support for Python 3.13. #264

Closed

charliermarsh reviewed Sep 6, 2024

View reviewed changes

src/validation.rs Show resolved Hide resolved

charliermarsh reviewed Sep 6, 2024

View reviewed changes

cpython-unix/extension-modules.yml Show resolved Hide resolved

charliermarsh approved these changes Sep 6, 2024

View reviewed changes

bschoenmaeckers mentioned this pull request Sep 6, 2024

Add python 3.13 free-threaded build #320

Closed

Support for Python 3.13.

0b230ff

Co-authored-by: Zanie Blue <contact@zanie.dev>

zanieb force-pushed the zb/3.13-rc1 branch from d3da7b3 to 0b230ff Compare September 6, 2024 17:18

zanieb mentioned this pull request Sep 6, 2024

Availability of ptsname_r is not checked at runtime on macOS python/cpython#123797

Closed

charliermarsh merged commit 483e6a4 into astral-sh:main Sep 6, 2024
291 checks passed

indygreg reviewed Sep 7, 2024

View reviewed changes

zanieb mentioned this pull request Sep 7, 2024

Add patch for Apple cross compilation on 3.13+ #323

Merged

KotlinIsland mentioned this pull request Sep 8, 2024

Support for Python beta releases (3.13)? #263

Closed

zanieb mentioned this pull request Sep 9, 2024

Enable BOLT optimizations for Python 3.13 #325

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Python 3.13 #319

Support for Python 3.13 #319

zanieb commented Sep 6, 2024 •

edited

Loading

zanieb Sep 6, 2024

zanieb Sep 6, 2024

henryiii Sep 6, 2024 •

edited

Loading

zanieb Sep 6, 2024

henryiii Sep 6, 2024

zanieb Sep 6, 2024

zanieb Sep 6, 2024

ned-deily Sep 6, 2024

zanieb Sep 6, 2024

zanieb Sep 6, 2024

indygreg Sep 7, 2024

zanieb Sep 6, 2024

This comment was marked as outdated.

charliermarsh Sep 6, 2024

zanieb Sep 6, 2024

zanieb Sep 6, 2024 •

edited

Loading

zanieb Sep 6, 2024 •

edited

Loading

zanieb commented Sep 6, 2024 •

edited

Loading

indygreg left a comment

FeodorFitsner commented Sep 7, 2024

zanieb commented Sep 7, 2024

indygreg commented Sep 7, 2024

zanieb commented Sep 7, 2024

zanieb commented Sep 7, 2024

indygreg commented Sep 7, 2024

zanieb commented Sep 7, 2024

ned-deily commented Sep 7, 2024

Support for Python 3.13 #319

Support for Python 3.13 #319

Conversation

zanieb commented Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryiii Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

zanieb Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

zanieb commented Sep 6, 2024 • edited Loading

indygreg left a comment

Choose a reason for hiding this comment

FeodorFitsner commented Sep 7, 2024

zanieb commented Sep 7, 2024

indygreg commented Sep 7, 2024

zanieb commented Sep 7, 2024

zanieb commented Sep 7, 2024

indygreg commented Sep 7, 2024

zanieb commented Sep 7, 2024

ned-deily commented Sep 7, 2024

zanieb commented Sep 6, 2024 •

edited

Loading

henryiii Sep 6, 2024 •

edited

Loading

zanieb Sep 6, 2024 •

edited

Loading

zanieb Sep 6, 2024 •

edited

Loading

zanieb commented Sep 6, 2024 •

edited

Loading