[BUG] Fast thread state access is broken by Py3.12a6

[tacaswell]

Describe the bug

Cython fails to build itself on cpython main:

  gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/tcaswell/.virtualenvs/bleeding/include -I/home/tcaswell/.pybuild/bleeding/include/python3.12 -c /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c -o build/temp.linux-x86_64-cpython-312/home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.o
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c: In function ‘__pyx_f_6Cython_8Compiler_7Parsing_p_dict_or_set_maker’:
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:31385:9: warning: ‘Py_OptimizeFlag’ is deprecated [-Wdeprecated-declarations]
  31385 |         if (unlikely(!Py_OptimizeFlag)) {
        |         ^~
  In file included from /home/tcaswell/.pybuild/bleeding/include/python3.12/Python.h:48,
                   from /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:31:
  /home/tcaswell/.pybuild/bleeding/include/python3.12/cpython/pydebug.h:13:37: note: declared here
     13 | Py_DEPRECATED(3.12) PyAPI_DATA(int) Py_OptimizeFlag;
        |                                     ^~~~~~~~~~~~~~~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c: In function ‘__Pyx_PyErr_ExceptionMatchesInState’:
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84939:32: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  84939 |     PyObject *exc_type = tstate->curexc_type;
        |                                ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c: In function ‘__Pyx_ErrRestoreInState’:
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84952:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  84952 |     tmp_type = tstate->curexc_type;
        |                      ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84953:23: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_value’
  84953 |     tmp_value = tstate->curexc_value;
        |                       ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84954:20: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  84954 |     tmp_tb = tstate->curexc_traceback;
        |                    ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84955:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  84955 |     tstate->curexc_type = type;
        |           ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84956:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_value’
  84956 |     tstate->curexc_value = value;
        |           ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84957:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  84957 |     tstate->curexc_traceback = tb;
        |           ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c: In function ‘__Pyx_ErrFetchInState’:
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84963:19: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  84963 |     *type = tstate->curexc_type;
        |                   ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84964:20: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_value’
  84964 |     *value = tstate->curexc_value;
        |                    ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84965:17: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  84965 |     *tb = tstate->curexc_traceback;
        |                 ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84966:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  84966 |     tstate->curexc_type = 0;
        |           ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84967:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_value’
  84967 |     tstate->curexc_value = 0;
        |           ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:84968:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  84968 |     tstate->curexc_traceback = 0;
        |           ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c: In function ‘__Pyx_IterFinish’:
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:85724:32: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  85724 |     PyObject* exc_type = tstate->curexc_type;
        |                                ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:85728:31: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_value’
  85728 |             exc_value = tstate->curexc_value;
        |                               ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:85729:28: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  85729 |             exc_tb = tstate->curexc_traceback;
        |                            ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:85730:19: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  85730 |             tstate->curexc_type = 0;
        |                   ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:85731:19: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_value’
  85731 |             tstate->curexc_value = 0;
        |                   ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:85732:19: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  85732 |             tstate->curexc_traceback = 0;
        |                   ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c: In function ‘__Pyx__GetException’:
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:86515:24: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  86515 |     local_type = tstate->curexc_type;
        |                        ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:86516:25: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_value’
  86516 |     local_value = tstate->curexc_value;
        |                         ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:86517:22: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  86517 |     local_tb = tstate->curexc_traceback;
        |                      ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:86518:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  86518 |     tstate->curexc_type = 0;
        |           ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:86519:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_value’
  86519 |     tstate->curexc_value = 0;
        |           ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:86520:11: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  86520 |     tstate->curexc_traceback = 0;
        |           ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:86526:24: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_type’
  86526 |     if (unlikely(tstate->curexc_type))
        |                        ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:1279:43: note: in definition of macro ‘unlikely’
   1279 |   #define unlikely(x) __builtin_expect(!!(x), 0)
        |                                           ^
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c: In function ‘__Pyx_Raise’:
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:86861:34: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  86861 |         PyObject* tmp_tb = tstate->curexc_traceback;
        |                                  ^~
  /home/tcaswell/source/p/cython/cython/Cython/Compiler/Parsing.c:86864:19: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
  86864 |             tstate->curexc_traceback = tb;
        |                   ^~
  error: command '/lib/ccache/bin/gcc' failed with exit code 1

I believe that this is caused by feec49c40736fc05626a183a8d14c4ebbea5ae28 via python/cpython#101607 working on python/cpython#101578 . Rolling back to 027adf42cd85db41fee05b0a40d89ef822876c97 (the previous commit) does fix cython building itself.

Code to reproduce the behaviour:

pip inst -v --no-build-isolation .

Expected behaviour

No response

Environment

OS: [e.g. Linux, Windows, macOS] linux
Python version [e.g. 3.10.2] main (2f62a5da949cd368a9498e6a03e700f4629fa97f)
Cython version [e.g. 3.0.0a11] master (94bea66)

Additional context

No response

[hrnciar]

This is now blocking building Cython with Python 3.12.0a6.

[hroncok]

(This also affects building other packages, not just Cython itself.)

[tacaswell]

The footprint of this in cython seems to be:

$ ack '.curexc'
Cython/Compiler/Nodes.py
10230:# type/value/tb in PyThreadState->curexc_*.  When being caught by an
10231:# 'except' statement, curexc_* is moved over to exc_* by
10248:    # the exception from tstate->curexc_* to tstate->exc_*, which prevents

Cython/Debugger/libpython.py
2507:                inf_type = tstate['curexc_type']
2508:                inf_value = tstate['curexc_value']

Cython/Utility/Exceptions.c
4:// type/value/tb in PyThreadState->curexc_*.  When being caught by an
5:// 'except' statement, curexc_* is moved over to exc_* by
32:#define __Pyx_PyErr_Occurred()  $local_tstate_cname->curexc_type
69:    PyObject *exc_type = tstate->curexc_type;
113:    tmp_type = tstate->curexc_type;
114:    tmp_value = tstate->curexc_value;
115:    tmp_tb = tstate->curexc_traceback;
116:    tstate->curexc_type = type;
117:    tstate->curexc_value = value;
118:    tstate->curexc_traceback = tb;
125:    *type = tstate->curexc_type;
126:    *value = tstate->curexc_value;
127:    *tb = tstate->curexc_traceback;
128:    tstate->curexc_type = 0;
129:    tstate->curexc_value = 0;
130:    tstate->curexc_traceback = 0;
306:        PyObject* tmp_tb = tstate->curexc_traceback;
309:            tstate->curexc_traceback = tb;
374:    local_type = tstate->curexc_type;
375:    local_value = tstate->curexc_value;
376:    local_tb = tstate->curexc_traceback;
377:    tstate->curexc_type = 0;
378:    tstate->curexc_value = 0;
379:    tstate->curexc_traceback = 0;
385:    if (unlikely(tstate->curexc_type))

Cython/Utility/ObjectHandling.c
242:    PyObject* exc_type = tstate->curexc_type;
246:            exc_value = tstate->curexc_value;
247:            exc_tb = tstate->curexc_traceback;
248:            tstate->curexc_type = 0;
249:            tstate->curexc_value = 0;
250:            tstate->curexc_traceback = 0;

and it is all behind a CYTHON_FAST_THREAD_STATE guard.

I started to look at how to fix this, but did not make it very far.