threading Thread.join should call the OS join API

[gpshead]

Feature or enhancement

Proposal:

threading.Thread.join() only waits for the CPython internals to wash its hands of the underlying thread. It doesn't actually wait for the OS thread itself to exit, which in theory happens rapidly as its final internal code completes quickly - but we have no good wait to determine.

Why finally do this now? Now that we're encouraging people to notice and avoid threading existing when os.fork is called in 3.12, a use case has come up for deterministically knowing when the thread is done at the OS level so that the code can proceed with os.fork.

#110510 could use this in an atfork before fork handler for example.

POSIX has pthread_join, we should be able to expose and use via _thread. Windows presumably has an equivalent concept API.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• GH-110829: Ensure Thread.join() joins the OS thread #110848

[pitrou]

On Windows, we start the thread with _beginthreadex, so we should be able to call WaitForSingleObject to ensure that the thread has properly terminated. See second example here: https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/beginthread-beginthreadex?view=msvc-170#examples

But, of course, on Windows the entire fork() problem doesn't exist, so it's less of a problem if we don't provide lesser guarantees.

Note that right now we detach the thread as soon as it is started, so we may want to expose a different internal API that doesn't detach the thread.

[ericsnowcurrently]

FWIW, the gap between Python-thread-ends and OS-thread-exits bit me after I added the per-interpreter GIL (PEP 684). We ended up with crash due to that interval (a race in the GIL's drop_gil(), which I fixed in gh-105109. The linked PRs on gh-104341 include a couple other approaches I tried out to solve the race (with extra complexity), and IIRC at least one of them would have (partially?) dealt with closing the above gap.

[pitrou]

The API added here is only for the threading module, though, because it's quite inflexible (see comments in the .h file).

[hugovk]

Closing as the PR has been merged.

[pitrou]

Oops, sorry for forgetting to close this issue!