Race condition in subinterpreters during subinterpreter creation on Windows debug build

[Fidget-Spinner]

Windows reproducer on main branch: run .\PCbuild\amd64\python_d.exe -m test test__xxsubinterpreters -F and leave it running. If by run 100 nothing crashes, cancel and re-run again until you get a segfault.

Traceback:

Thread 0x000022ec (most recent call first):
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\test__xxsubinterpreters.py", line 788 in setUp
  File "D:\Ken\Documents\GitHub\cpython\Lib\unittest\case.py", line 576 in _callSetUp
  File "D:\Ken\Documents\GitHub\cpython\Lib\unittest\case.py", line 619 in run
  File "D:\Ken\Documents\GitHub\cpython\Lib\unittest\case.py", line 678 in __call__
  File "D:\Ken\Documents\GitHub\cpython\Lib\unittest\suite.py", line 122 in run
  File "D:\Ken\Documents\GitHub\cpython\Lib\unittest\suite.py", line 84 in __call__
  File "D:\Ken\Documents\GitHub\cpython\Lib\unittest\suite.py", line 122 in run
  File "D:\Ken\Documents\GitHub\cpython\Lib\unittest\suite.py", line 84 in __call__
  File "D:\Ken\Documents\GitHub\cpython\Lib\unittest\suite.py", line 122 in run
  File "D:\Ken\Documents\GitHub\cpython\Lib\unittest\suite.py", line 84 in __call__
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\support\testresult.py", line 140 in run
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\support\__init__.py", line 1099 in _run_suite
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\support\__init__.py", line 1225 in run_unittest
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\runtest.py", line 281 in _test_module
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\runtest.py", line 317 in _runtest_inner2
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\runtest.py", line 360 in _runtest_inner
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\runtest.py", line 235 in _runtest
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\runtest.py", line 265 in runtest
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\main.py", line 456 in run_tests_sequential
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\main.py", line 574 in run_tests
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\main.py", line 752 in _main
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\main.py", line 711 in main
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\libregrtest\main.py", line 775 in main
  File "D:\Ken\Documents\GitHub\cpython\Lib\test\__main__.py", line 2 in <module>
  File "D:\Ken\Documents\GitHub\cpython\Lib\runpy.py", line 88 in _run_code
  File "D:\Ken\Documents\GitHub\cpython\Lib\runpy.py", line 198 in _run_module_as_main

Somehow after calling __xxsubinterpreters.create: in drop_gil the &ceval2->gil_drop_request is non-NULL but invalid, so when we try to _Py_atomic_load load from that address it segfaults.

I have my suspicions it's due to swapping of GIL states but this is incredibly hard to debug because the race condition sometimes triggers when I run the test 4 times in a row, and sometimes doesn't trigger after 2 hours of continuous running.

[Fidget-Spinner]

cc @ericsnowcurrently

[ericsnowcurrently]

I'll take a look tomorrow. Thanks for finding this!

[ericsnowcurrently]

What's the best way to reproduce the problem?

[Fidget-Spinner]

Sorry, I have not found a reliable/minimal reproducer. The only thing I found was that running
.\PCbuild\amd64\python_d.exe -m test test__xxsubinterpreters -F on Windows
eventually triggers it (sometimes it takes 100 test iterations). If you don't get a crash by then, try rerunning it.

[ericsnowcurrently]

Is the race in the main branch? Any thoughts on why you noticed it while working on gh-100492?

[Fidget-Spinner]

Yep it's on main. I'm not sure why but it requires less iterations to trigger on that PR. It requires many more iterations on main.

[ericsnowcurrently]

I took a quick look at ceval_gil.c and may have a sense of what's going wrong. FWIW, I expect that the race has been around at least since the "new GIL" was added in 2009, if not longer, and that _xxsubinterpreters has only brought it to our attention (rather than causing the problem).

The main thing I noticed (in ceval_gil.c) is that both drop_gil() and take_gil() take a PyThreadState argument for the thread that is taking/releasing the GIL, AKA "thread A". It is possible for that thread state to be destroyed (directly, or indirectly when its interpreter is destroyed) in another thread, AKA "thread B", at any point the GIL is no longer held by thread A.

If the given thread state (thread A) were used at any such point within drop_gil() or take_gil() then I'd expect a segfault. Here are places where that is possible:

• drop_gil()
  • before GIL-protecting mutex aquired
    • set _PyRuntimeState.ceval.gil.last_holder to thread A
  • after GIL-protecting mutex released
    • read tstate->interp->ceval.gil_drop_request
    • compared tstate to _PyRuntimeState.ceval.gil.last_holder
    • RESET_GIL_DROP_REQUEST() called with tstate->interp
• take_gil()
  • before GIL-protecting mutex acquired
    • read tstate->interp (used after GIL mutex acquired)
    • read tstate->interp->ceval (used after GIL mutex acquired)
  • while waiting for GIL
    • (during interpreter/runtime fini) RESET_GIL_DROP_REQUEST() called with tstate->interp
    • SET_GIL_DROP_REQUEST() called with tstate->interp
  • after GIL acquired (but thread A might have been destroyed while waiting)
    • set _PyRuntimeState.ceval.gil.last_holder to thread A
    • (during interpreter/runtime fini) drop_gil() called with tstate
    • read tstate->interp->ceval.gil_drop_request
    • RESET_GIL_DROP_REQUEST() (or COMPUTE_EVAL_BREAKER() ) called with tstate->interp
    • read tstate->async_exc
    • _PyEval_SignalAsyncExc() called with tstate->interp

Some of these might be safe due to the logic but we need to verify that and fix the race on the rest. A race on any of these is unlikely, but possible if one thread is cleaning up interpreters/threads while another is creating them (as happens in test__xxsubinterpreters.

As to the relationship with Windows, it looks like the sleep granularity (while waiting for a lock) there is much more coarse than it is with pthreads (e.g. on linux).

Anyone is welcome to investigate further or propose solutions. I'll be looking at this more later on today.

Some observations (from ceval_gil.c):

• PyInterpreterState.ceval.gil_drop_request
  • a runtime-global boolean value (stored in an atomic int)
  • set by SET_GIL_DROP_REQUEST() and RESET_GIL_DROP_REQUEST()
    • RESET_GIL_DROP_REQUEST() (i.e. "clear") is used by drop_gil() and take_gil()
    • SET_GIL_DROP_REQUEST() is used only by take_gil()
    • both also touch the global eval_breaker
  • read by:
    • COMPUTE_EVAL_BREAKER() (if set then sets eval breaker)
    • drop_gil() (if set and calling tstate is still "holder" then call RESET_GIL_DROP_REQUEST() and wait)
    • take_gil() (if set then call RESET_GIL_DROP_REQUEST())
    • _Py_HandlePending() (if set then release and reacquire the GIL)
• _PyRuntimeState.ceval.gil.last_holder

[Fidget-Spinner]

Thanks for your insightful comment! My own suspicion was that the GILState struct is a stack-allocated variable (it's a variable in a function rather than malloc-ed). So it's possible for one subinterpreter to outlive the GILState struct and segfault.

[ericsnowcurrently]

Can you verify if this is fixed now? This may have been a case of gh-104341.

[picnixz]

friendly reminder @Fidget-Spinner