An interpreter's initial thread can be accessed while finalizing

[ZeroIntensity]

Crash report

What happened?

While working on gh-126644, I came across an elusive bug that seems to occur when a lot of threads are trying to create a thread state for the same interpreter. Initially, I thought it was an issue with free-threading, but it occurs on the default build as well.

Here's a small reproducer:

from threading import Thread
import _interpreters

interp = _interpreters.create()

def run():
    this_interp = _interpreters.create()
    _interpreters.run_string(this_interp, f"import _interpreters; _interpreters.run_string({interp}, '1')")

threads = [Thread(target=run) for _ in range(1000)]
for thread in threads:
    thread.start()

for thread in threads:
    thread.join()

As it turns out, this is because of this check: https://github.com/python/cpython/blob/main/Python/pystate.c#L1531

Inside tstate_delete_common, the runtime lock is released after the threads.head has already been set to NULL, so other threads think erroneously think it's OK to try and use _initial_thread while it's still getting finalized. Possibly related, free_threadstate tries to reset _initial_thread back to the default settings without the runtime lock, which is possibly racy.

There's a few ways to fix this, but I think the easiest (and has the least amount of risk for backporting) is to just hold the runtime lock for all of thread state deletion. This will introduce a very slight slowdown due to lock contention, but that should get better once #114940 lands. I have a PR ready.

CPython versions tested on:

3.12, 3.13, 3.14, CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

• gh-126914: Use an atomic field for determining if the initial thread can be used #126915
• gh-126914: Store the Preallocated Thread State's Pointer in a PyInterpreterState Field #126989
• [3.13] gh-126914: Store the Preallocated Thread State's Pointer in a PyInterpreterState Field #127114

[ZeroIntensity]

I think the easiest (and has the least amount of risk for backporting) is to just hold the runtime lock for all of thread state deletion.

Apparently, this is problematic for the free-threaded build, because QSBR and thread-local refcounting initialization cause a lock-ordering deadlock in some cases. So, instead, my solution is to just add a new field to the interpreter state that indicates when it's safe to access _initial_thread, and that field is only used atomically (unlike something like tstate->_status). I'm hoping that should fix it without issues.