TSan: data race with _PyThreadState_Attach and _PyThreadState_Detach

[pitrou]

Bug report

Bug description:

Running test_concurrent_futures.test_process_pool with TSan enabled and the GIL enabled yields the two following race conditions:

1. with _PyThreadState_Attach:

test_ressources_gced_in_workers (test.test_concurrent_futures.test_process_pool.ProcessPoolSpawnProcessPoolExecutorTest.test_ressources_gced_in_workers) ... 
==================
WARNING: ThreadSanitizer: data race (pid=86884)
  Write of size 8 at 0x7b4800010b20 by main thread:
    #0 memset <null> (python+0xd3f5d) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #1 fill_mem_debug /home/antoine/cpython/default/Objects/obmalloc.c:2624:5 (python+0x30182a) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #2 _PyMem_DebugRawFree /home/antoine/cpython/default/Objects/obmalloc.c:2753:5 (python+0x30182a)
    #3 PyMem_RawFree /home/antoine/cpython/default/Objects/obmalloc.c:963:5 (python+0x2ff108) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #4 free_threadstate /home/antoine/cpython/default/Python/pystate.c:1287:9 (python+0x525457) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #5 _PyThreadState_DeleteExcept /home/antoine/cpython/default/Python/pystate.c:1721:9 (python+0x525457)
    #6 Py_FinalizeEx /home/antoine/cpython/default/Python/pylifecycle.c:1931:5 (python+0x4fbc98) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #7 Py_Exit /home/antoine/cpython/default/Python/pylifecycle.c:3138:9 (python+0x4fe745) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #8 handle_system_exit /home/antoine/cpython/default/Python/pythonrun.c:606:9 (python+0x52a20e) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #9 _PyErr_PrintEx /home/antoine/cpython/default/Python/pythonrun.c:615:5 (python+0x52a20e)
[...]

  Previous read of size 8 at 0x7b4800010b20 by thread T10:
    #0 _PyThreadState_MustExit /home/antoine/cpython/default/Python/pystate.c:2788:64 (python+0x527e81) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #1 take_gil /home/antoine/cpython/default/Python/ceval_gil.c:289:9 (python+0x4b7a90) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #2 _PyEval_AcquireLock /home/antoine/cpython/default/Python/ceval_gil.c:555:5 (python+0x4b81be) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #3 _PyThreadState_Attach /home/antoine/cpython/default/Python/pystate.c:1878:5 (python+0x525943) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #4 _Py_HandlePending /home/antoine/cpython/default/Python/ceval_gil.c:1080:9 (python+0x4b8fe8) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
[...]

2. with _PyThreadState_Detach:

  Write of size 8 at 0x7b48000007a8 by main thread:
    #0 memset <null> (python+0xd3f5d) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #1 fill_mem_debug /home/antoine/cpython/default/Objects/obmalloc.c:2624:5 (python+0x30182a) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #2 _PyMem_DebugRawFree /home/antoine/cpython/default/Objects/obmalloc.c:2753:5 (python+0x30182a)
    #3 PyMem_RawFree /home/antoine/cpython/default/Objects/obmalloc.c:963:5 (python+0x2ff108) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #4 free_threadstate /home/antoine/cpython/default/Python/pystate.c:1287:9 (python+0x525457) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #5 _PyThreadState_DeleteExcept /home/antoine/cpython/default/Python/pystate.c:1721:9 (python+0x525457)
    #6 Py_FinalizeEx /home/antoine/cpython/default/Python/pylifecycle.c:1931:5 (python+0x4fbc98) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #7 Py_Exit /home/antoine/cpython/default/Python/pylifecycle.c:3138:9 (python+0x4fe745) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #8 handle_system_exit /home/antoine/cpython/default/Python/pythonrun.c:606:9 (python+0x52a20e) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #9 _PyErr_PrintEx /home/antoine/cpython/default/Python/pythonrun.c:615:5 (python+0x52a20e)
[...]

  Previous atomic read of size 8 at 0x7b48000007a8 by thread T1:
    #0 _Py_atomic_load_uintptr_relaxed /home/antoine/cpython/default/./Include/cpython/pyatomic_gcc.h:347:10 (python+0x4b805f) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #1 _Py_eval_breaker_bit_is_set /home/antoine/cpython/default/./Include/internal/pycore_ceval.h:239:19 (python+0x4b805f)
    #2 drop_gil /home/antoine/cpython/default/Python/ceval_gil.c:255:9 (python+0x4b805f)
    #3 _PyEval_ReleaseLock /home/antoine/cpython/default/Python/ceval_gil.c:564:5 (python+0x4b8214) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #4 detach_thread /home/antoine/cpython/default/Python/pystate.c:1918:5 (python+0x525c39) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #5 _PyThreadState_Detach /home/antoine/cpython/default/Python/pystate.c:1924:5 (python+0x522a9b) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #6 PyEval_SaveThread /home/antoine/cpython/default/Python/ceval_gil.c:611:5 (python+0x4b8402) (BuildId: 80a378e5bf8e9a3b503d1ddb455283576a308db6)
    #7 sock_call_ex /home/antoine/cpython/default/./Modules/socketmodule.c:967:13 (_socket.cpython-313d-x86_64-linux-gnu.so+0x11344) (BuildId: 0722d7ca606ecbe349d256ecc85d7833588ba216)
[...]

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

[pitrou]

@colesbury @mpage

[colesbury]

This is related to #110052 (comment)