Add new states ATTACHED, DETACHED, GC to PyThreadState to support PEP 703

[colesbury]

Feature or enhancement

To support pausing threads for garbage collection, PEP 703 requires identifying which threads may be accessing Python object internals (like reference counts). Currently, this is effectively controlled by acquiring the GIL. The PEP proposes three thread-states (https://peps.python.org/pep-0703/#thread-states):

• ATTACHED
• DETACHED
• GC

I propose adding these states and a new field to PyThreadState. Like all the other fields in PyThreadState (other than interp), the field is intended to be private.

#define _Py_THREAD_DETACHED 0
#define _Py_THREAD_ATTACHED 1
#define _Py_THREAD_GC 2

struct _ts {
   ...
   int state;
}

Unfortunately, we can't add this to the _status bitfield because operations need to be atomic, and atomic operations don't work well with C bitfields.

Default build (with GIL)

The attached and detached states will correspond precisely with acquiring and releasing the GIL. A thread will set it's state to _Py_THREAD_ATTACHED immediately after acquiring the GIL lock and set it to _Py_THREAD_DETACHED immediately before releasing the GIL lock.

The _Py_THREAD_GC state will not be used in the default build.

--disable-gil build

From https://peps.python.org/pep-0703/#thread-states:

When compiling without the GIL, functions that previously acquired the GIL instead transition the thread state to ATTACHED, and functions that previously released the GIL transition the thread state to DETACHED. Just as threads previously needed to acquire the GIL before accessing or modifying Python objects, they now must be in the ATTACHED state before accessing or modifying Python objects. Since the same public C-API functions “attach” the thread as previously acquired the GIL (e.g., PyEval_RestoreThread), the requirements for thread initialization in extensions remain the same. The substantial difference is that multiple threads can be in the attached state simultaneously, while previously only one thread could acquire the GIL at a time.

cc @ericsnowcurrently

Linked PRs

• gh-109549: Add new states to PyThreadState to support PEP 703 #109915

[ericsnowcurrently]

Unfortunately, we can't add this to the _status bitfield because operations need to be atomic, and atomic operations don't work well with C bitfields.

Yeah, having a separate field is fine. Ideally this would be part of _status but the constraint with atomics makes sense.

Presumably "with GIL" builds won't need the field to be atomic. Is that right?

[colesbury]

Presumably "with GIL" builds won't need the field to be atomic. Is that right?

That's right