[Python] Support for the free-threaded build of CPython 3.13 #43536

lysnikolaou · 2024-08-02T09:51:54Z

This is done by passing an extra flag when building the Cython extension modules. It is needed so that the GIL is not dynamically reenabled when importing `pyarrow.lib`.

jorisvandenbossche · 2024-08-13T11:00:53Z

@lysnikolaou for adding a free-threaded test build, I suppose we will want to add a wheel build as well (so after #43539 is finally ready), to ensure we have nightly wheels people can test with.
But regardless of wheels, it might be good to have a plain test build as well, which could mimic the cpython-debug build that Antoine added? (#43565) That is something that doesn't have to wait on the wheels / general python 3.13 support in CI, I think

lysnikolaou · 2024-08-13T11:02:07Z

But regardless of wheels, it might be good to have a plain test build as well, which could mimic the cpython-debug build that Antoine added? (#43565) That is something that doesn't have to wait on the wheels / general python 3.13 support in CI, I think

On it.

…d build

### Rationale for this change This is done by passing an extra flag when building the Cython extension modules. It is needed so that the GIL is not dynamically reenabled when importing `pyarrow.lib`. ### What changes are included in this PR? Changes to CMake so that the extra flag is passed when building Cython extension modules. * GitHub Issue: #43536 Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

kou · 2024-08-13T23:46:11Z

Issue resolved by pull request 43606
#43606

kou · 2024-08-13T23:52:01Z

@lysnikolaou Could you create a sub issue per sub item? Our release management scripts don't support multiple PRs per issue for now.

jorisvandenbossche · 2024-08-14T06:43:14Z

This is fine to keep as the general tracking issue (the release script will just only list this issue in the changelog, which IMO is perfectly fine). It is easier project management to keep track of this in a single issue.

lysnikolaou · 2024-08-15T08:26:00Z

I agree that this issue should be kept open, since it provides a nice overview of what level of support can be expected.

@kou Sure! Should I open subissues for the already open/merged PRs as well or is it okay to just do it in the future?

kou · 2024-08-15T08:38:47Z

If @jorisvandenbossche doesn't like create subissues, I don't ask you to create subissues strongly. (As a committer, I don't want to reopen this issue manually after I merge a PR by our merge script.)
@raulcd What do you think about this as a release manager? We can't assign multiple milestones to one issue. If we slip some subtasks to 19.0.0, it may cause a problem.

FYI: We used an umbrella issue and subissues when we added support for Azure filesystem: #18014

raulcd · 2024-08-15T09:22:53Z

At the moment all our merge (closes issues automatically) and release scripts (generation of release notes, cherry picking) expect one issue per PR.

This was a requirement for JIRA but now that we are tracking issues on GH I think we can revisit that pattern, we could even avoid some issues and just use PRs for some things but this would require some time to adapt our current scripts.

As per this particular issue, the biggest potential problems for the release:

As you have already seen the issue will keep getting closed every time we merge a subtask.
PRs merged pointing to this PR will probably not appear on the release notes (as far as I know release notes are generated from issues tagged with the correct milestone not PRs)
If we merge something after the feature freeze it won't get cherry-picked by our scripts

To be honest not really breaking issues for the release but if this is a workflow we want to use (which I think we should) we should adapt our scripts. I am currently on vacation so might take some time responding.

raulcd · 2024-08-15T09:25:18Z

@kou Sure! Should I open subissues for the already open/merged PRs as well or is it okay to just do it in the future?

Already merged PRs point to the "parent" issue on the commit message, this is what is used to link the commits to issues on the release scripts, unless we want to change the commit history (which we are not going to do) it doesn't matter if we open issues for the ones already merged :)

BTW, thanks for the great work you are doing @lysnikolaou

lysnikolaou · 2024-08-15T10:08:47Z

Ok then, I'll be creating subissues for PRs I open from now on. Thanks everyone!

### Rationale for this change For better reference safety under Python free-threaded builds (i.e. with the GIL removed), we should be using `Py(List|Dict)_GetItemRef` that return strong references and are implemented in a thread-safe manner. ### What changes are included in this PR? - Vendor a copy of https://github.com/python/pythoncapi-compat - Port to strong reference APIs for lists and dicts ### Are these changes tested? I ran the tests with the free-threaded build before and after, and there's the same expected failures. * GitHub Issue: #43536 Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

pitrou · 2024-08-15T12:24:41Z

Issue resolved by pull request 43540
#43540

lysnikolaou · 2024-08-15T13:59:02Z

Can we please reopen this? These are PRs that I had created before we discussed this. Sorry!

pitrou · 2024-08-15T14:01:09Z

Woops, sorry, I've been a bit too optimistic here :)

…d build

#43671) ### Rationale for this change Testing with the free-threaded build is required for adding support for it. (see #43536) ### What changes are included in this PR? - Add a Docker build with the CPython free-threaded build from deadsnakes. - Add a Crossbow job to run said Docker build with Python 3.13t ### Are there any user-facing changes? No. * GitHub Issue: #43536 Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

…d build (apache#43671) ### Rationale for this change Testing with the free-threaded build is required for adding support for it. (see apache#43536) ### What changes are included in this PR? - Add a Docker build with the CPython free-threaded build from deadsnakes. - Add a Crossbow job to run said Docker build with Python 3.13t ### Are there any user-facing changes? No. * GitHub Issue: apache#43536 Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

#43965) ### Rationale for this change Building free-threaded wheels is necessary to support the free-threaded build. We probably want to upload these wheels as nightlies somewhere as well, so that downstream users can test the free-threading-related changes. ### What changes are included in this PR? - Add necessary configuration to build 3.13 free-threading wheels on `manylinux`. - Do necessary changes to build free-threaded wheels on macOS as well. ### Are these changes tested? I tested the `manylinux` wheel builds. macOS is still untested, since it's not dockerized. ### Are there any user-facing changes? No. Related to #43536. * GitHub Issue: #43964 Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

pitrou · 2024-10-08T08:01:34Z

Thanks to astral-sh/python-build-standalone#326 , there should be very soon free-threaded builds available on https://github.com/indygreg/python-build-standalone/ . That might be easier than using a PPA and being constrained to Ubuntu.

raulcd · 2024-10-09T13:21:08Z

@jorisvandenbossche @pitrou is there anything to be done here for 18.0.0?

pitrou · 2024-10-09T13:41:49Z

@raulcd I don't think so. The wheels are produced fine AFAICT (hopefully they will correctly be uploaded on PyPI as well? :-)).

jorisvandenbossche · 2024-10-09T17:57:59Z

Indeed, I think the basics are ready (I also had the question whether the wheels would automatically be uploaded, but @raulcd thought that would be the case).

At the moment the linux wheels builds for free-threaded are failing though, but created a separate issue for that: #44355

I will open a follow-up issue for the remaining tasks, so we can close this was as done for 18.0

raulcd · 2024-10-13T10:40:14Z

I am closing this as done for 18.0.0. The only sub-task missing is around testing Cython. Please feel free to open a new issue for that if necessary.

lysnikolaou added the Type: enhancement label Aug 2, 2024

github-actions bot added the Component: Python label Aug 2, 2024

github-actions bot mentioned this issue Aug 2, 2024

GH-43536: [Python] Do not use borrowed references APIs #43540

Merged

github-actions bot assigned lysnikolaou Aug 2, 2024

github-actions bot mentioned this issue Aug 7, 2024

GH-43536: [Python] Declare support for free-threading in Cython #43606

Merged

lysnikolaou mentioned this issue Aug 7, 2024

Status of PyArrow compatibility Quansight-Labs/free-threaded-compatibility#57

Closed

lysnikolaou added a commit to lysnikolaou/arrow that referenced this issue Aug 13, 2024

apacheGH-43536: [Python][CI] Add a Crossbow job with the free-threade…

1cb315b

…d build

github-actions bot mentioned this issue Aug 13, 2024

GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build #43671

Merged

kou added this to the 18.0.0 milestone Aug 13, 2024

kou closed this as completed Aug 13, 2024

lidavidm mentioned this issue Aug 13, 2024

python: add CI build using nogil python apache/arrow-adbc#2076

Open

jorisvandenbossche reopened this Aug 14, 2024

pitrou closed this as completed Aug 15, 2024

pitrou reopened this Aug 15, 2024

jmao-denver mentioned this issue Aug 28, 2024

ENH: Support free-threaded python build jpy-consortium/jpy#163

Closed

This was referenced Sep 5, 2024

[Python] Build wheels for the 3.13 free-threaded build #43964

Closed

GH-43964: [Python] Build macOS and manylinux wheels for free-threading #43965

Merged

pitrou pushed a commit to lysnikolaou/arrow that referenced this issue Sep 9, 2024

apacheGH-43536: [Python][CI] Add a Crossbow job with the free-threade…

af85b16

…d build

lysnikolaou mentioned this issue Sep 10, 2024

[Python] Fix threading bugs under the free-threaded version related to borrowed references and Pandas initialization #44046

Closed

raulcd closed this as completed Oct 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Python] Support for the free-threaded build of CPython 3.13 #43536

[Python] Support for the free-threaded build of CPython 3.13 #43536

lysnikolaou commented Aug 2, 2024 •

edited by AlenkaF

Loading

jorisvandenbossche commented Aug 13, 2024 •

edited

Loading

lysnikolaou commented Aug 13, 2024

kou commented Aug 13, 2024

kou commented Aug 13, 2024

jorisvandenbossche commented Aug 14, 2024 •

edited

Loading

lysnikolaou commented Aug 15, 2024

kou commented Aug 15, 2024

raulcd commented Aug 15, 2024

raulcd commented Aug 15, 2024

lysnikolaou commented Aug 15, 2024

pitrou commented Aug 15, 2024

lysnikolaou commented Aug 15, 2024

pitrou commented Aug 15, 2024

pitrou commented Oct 8, 2024

raulcd commented Oct 9, 2024

pitrou commented Oct 9, 2024

jorisvandenbossche commented Oct 9, 2024

raulcd commented Oct 13, 2024

[Python] Support for the free-threaded build of CPython 3.13 #43536

[Python] Support for the free-threaded build of CPython 3.13 #43536

Comments

lysnikolaou commented Aug 2, 2024 • edited by AlenkaF Loading

Describe the enhancement requested

Component(s)

jorisvandenbossche commented Aug 13, 2024 • edited Loading

lysnikolaou commented Aug 13, 2024

kou commented Aug 13, 2024

kou commented Aug 13, 2024

jorisvandenbossche commented Aug 14, 2024 • edited Loading

lysnikolaou commented Aug 15, 2024

kou commented Aug 15, 2024

raulcd commented Aug 15, 2024

raulcd commented Aug 15, 2024

lysnikolaou commented Aug 15, 2024

pitrou commented Aug 15, 2024

lysnikolaou commented Aug 15, 2024

pitrou commented Aug 15, 2024

pitrou commented Oct 8, 2024

raulcd commented Oct 9, 2024

pitrou commented Oct 9, 2024

jorisvandenbossche commented Oct 9, 2024

raulcd commented Oct 13, 2024

lysnikolaou commented Aug 2, 2024 •

edited by AlenkaF

Loading

jorisvandenbossche commented Aug 13, 2024 •

edited

Loading

jorisvandenbossche commented Aug 14, 2024 •

edited

Loading