Python 3.13 support

[burgholzer]

What should we add?

Python 3.13 has reached release candidate status, which is deemed ABI-stable.
Hence, projects should start building and distributing wheels for it.
Cibuildwheel 2.20 just released with support for the 3.13 release candidate on by default.
That release also contains opt-in support for the new free-threaded variant of Python 3.13, which requires extra wheels to be built (see https://cibuildwheel.pypa.io/en/stable/options/#free-threaded-support)
The earlier the whole ecosystem tries to support 3.13 and its free-threaded variant, the broader and quicker its adoption will be.
Consider this issue a gentle nudge to start exploring Python 3.13 support within the project.

[jakelishman]

Thanks. Our wheels are built to the stable ABI (abi3) - we've technically had 3.13-compatible wheels released for a year.

We're aware of the release candidate, but at present, there aren't even wheels on PyPI for Numpy for the rc1 yet, let alone some of our other less core dependencies. I haven't tested to see if there are code changes that will break us yet (seems unlikely) because our dependencies aren't 3.13-compatible yet.

We will not be enabling free-threaded support any time soon in Qiskit; we're not set up for it, and we supply a lot of extension-module code that handles its own threading, including a reliance on the GIL to mediate access to internally held Python objects. The stable C API does not support free threading at all with Python 3.13.

We are interested in it, and will look into it in the future, but it will be a lot of work for us, and will likely be a follow-on project that takes places after other things, like our own stable C API. If there ends up being any requirement on us to maintain separate code paths to support free-threading (even if only for performance in the non-free-threaded builds), it might be that we wait until the version of Python that stabilises free-threaded builds is our minimum required Python, which could be in a number of years - it depends on the maintenance trade-off of providing multiple code paths, since we are already doing our own multithreading so there is less likely to be as large an advantage to Python threading on top than for other libraries.

[jakelishman]

Fwiw, there are Numpy and Scipy wheels in the nightly channel on conda, but we'd still have problems with other core dependencies, like symengine, and until we're closer to that goal, it's a bunch of work to set up CI to pull particular dependencies from a non-PyPI source for a test run that won't pass.

[burgholzer]

Thanks for the quick response and great to hear that you are already thinking about these things.
I already forgot that Qiskit switched to building abi3 wheels, which is great. I guess for the most part it's about all the external dependencies then to support 3.13.
Might be worth giving some of the smaller ones a little nudge to also support it rather sooner than later.
Bigger projects like numpy and scipy are already working on producing 3.13 wheels (and as you point out, already ship them in nightly builds).

As for the free-threaded variant: totally relatable. This is definitely a bigger project.
Good to know that projects depending on Qiskit can consider the question of "will free-threading support be coming to Qiskit" answered (at least for the foreseeable future).
Just for anyone interested: Both numpy and scipy are actively exploring free-threading support at the moment: numpy/numpy#26157 and scipy/scipy#20669

[jakelishman]

Numpy at least has free-threaded wheels in its nightlies on conda, though in practice I don't know how much additional work in the APIs is going to be needed here to make things that work sensibly, since much of what we use Numpy (and Scipy) for is their ability to dynamically call BLAS/LAPACK routines that are themselves threaded. If the Python code is threading as well, there's likely going to be thread contention in unexpected places. The same thing will happen on Qiskit's side - we have our own threaded parallelism in various places in the compiler, which was written with the GIL in mind.

Just to add to our support of free-threading: we are interested in it, but it's not going to be a high priority in the near term because of all the other performance work that's on-going. We can already use a full set of threads for some of the largest compiler passes, and are moving into a position where some other major transpiler passes can too. We don't need Python free-threading to make that possible, and we don't want to base performance goals for the library around an experimental implementation in only a special build of the latest version of Python (there's explicitly no guarantee that PEP 703 won't be revoked, if it ends up having knock-on effects) - it's a bit of a different story if you're writing a binary, though.

From a technical perspective, beyond the social one above, our hands are tied primarily by us using the stable API, and by PyO3 not yet having support for free-threaded Python (they're also interested: see PyO3/pyo3#4265). PyO3 is the only 100% hard blocker on us, since we rely on them, but there's still various bits of Qiskit that would need updating at a minimum - we hide various static initialisations in GIL-protected cells (since that's zero-cost for things that implied the GIL anyway), and such like. Since we'd have to be making a second build for the free-threaded version anyway, chances are we could probably just drop the stable-API build for that, so that's not a super hard blocker, it's just busy work.

---

For our upstream libraries: I generally trust that other maintainers of core Python libraries are also aware of the state of their language and new features coming for it - it happens every year. If they haven't got round to making a new release yet within the first four days of 3.13rc1, I'm quite happy giving them a bit more time to fit it into their own priorities.

[1ucian0]

For context, the Python release schedule:

• 3.13.0 candidate 1: Thursday, 2024-08-01
• 3.13.0 candidate 2: Tuesday, 2024-09-03
• 3.13.0 final: Tuesday, 2024-10-01

[ngoldbaum]

Hi! I'm working on ecosystem support for free-threading and happned to stumble on this.

our hands are tied primarily by us using the stable API

Python 3.14 is the first version planned to support free-threading and the limited API simultaneously. It'll be a fair amount of work to get there though, including defining abi4, so it's not yet clear if it will make it into 3.14.