Simplify the interpreter's (type, val, tb) exception representation #89874
Comments
|
Exceptions are represented in the interpreter as (type, val, tb) triplets which most of the time contain redundant information (the type is the type of val and the tb is also on the exception). This complicates the code and is inefficient as opcodes that manage exceptions push and pop 3 items for each exception. We will change the internal representation to be (1) just the exception value if it is normalised and (2) a tuple of the 3 values for the uncommon case where they are all needed. See also faster-cpython/ideas#106. |
iritkatriel
added
3.11
|
Would there be any change at the Python level? |
|
Initially not, neither in python nor in the c api. It would be nice to replace PyErr_Fetch/Restore by a version that takes just an exception but that’s a long deprecation. |
|
Following the analysis/discussion on faster-cpython/ideas#106:
|
|
FYI, we track the Cython side of this in |
|
This is done, there will be more cleanups that this enables but they can have their own issues. |
|
Would it be possible to revert the change until Cython and numpy are ready for it?
This change broke numpy which uses C code generated by Cython for coroutines in numpy/random/_mt19937.c (file generated from numpy/random/_mt19937.pyx). Example of Cython code which no fails with a compiler error since exc_info->exc_type has been removed: /* GetTopmostException */
#if CYTHON_USE_EXC_INFO_STACK
static _PyErr_StackItem *
__Pyx_PyErr_GetTopmostException(PyThreadState *tstate)
{
_PyErr_StackItem *exc_info = tstate->exc_info;
while ((exc_info->exc_type == NULL || exc_info->exc_type == Py_None) &&
exc_info->previous_item != NULL)
{
exc_info = exc_info->previous_item;
}
return exc_info;
}
#endifI propose this plan:
It should increase the number of packages when Python 3.11 will be released. In Cython, the issue is tracked as: cython/cython#4500 |
|
I can probably just put back the two fields in _PyErr_StackItem and make sure they get updated when exc_value is set. Reverting the whole thing would include big changes in compile.c and ceval.c, so I'd rather avoid that if we can. |
|
Does unsetting CYTHON_USE_EXC_INFO_STACK still work? #if CYTHON_USE_EXC_INFO_STACK
// See https://bugs.python.org/issue25612
#define __Pyx_ExcInfoStruct _PyErr_StackItem
#else
// Minimal replacement struct for Py<3.7, without the Py3.7 exception state stack.
typedef struct {
PyObject *exc_type;
PyObject *exc_value;
PyObject *exc_traceback;
} __Pyx_ExcInfoStruct;
#endif |
I don't understand the Cython status. The code was updated in the master branch: cython/cython#4500 was closed even if the change was not backported to 0.29.x. I understand that there is no released Cython 0.29.x version compatible with this change. |
|
I commented the Cython issue: I also opened a discussion on python-dev: "Please update Cython before introcuding C API incompatible changes in Python". |
|
Irit Katriel:
It was a good idea to split changes of this issue into multiple commits :-) I'm only proposing to revert the one which introduces the C API incompatible changes: the commit 396b583 "bpo-45711: Remove type and traceback from exc_info (GH-30122)". Maybe Cython can be quickly updated, but the situation seems to be complicated :-( cython/cython#4500 |
|
That commit has significant changes in ceval.c and compile.c. They don't need to be reverted to unbreak cython. I'm working on a PR for a simpler change. Have you tried with CYTHON_USE_EXC_INFO_STACK undefined? |
|
If this is still the position of cython maintainers: then I will need to revert the change until 3.12. |
|
Time to insist on directly communicating with the Cython team (esp. @scoder) and broker some kind of compromise. |
Python provides a *private* _PyErr_GetTopmostException(tstate) function, but Cython reimplements its own function. I'm not sure why. #74716 proposes adding PyErr_GetActiveException() function which has no parameter, but Cython __Pyx_PyErr_GetTopmostException() has a tstate parameter. Simplified example numpy/random/_mt19937.c code: static CYTHON_INLINE void
__Pyx__ExceptionSave(PyThreadState *tstate, PyObject **type,
PyObject **value, PyObject **tb)
{
_PyErr_StackItem *exc_info = __Pyx_PyErr_GetTopmostException(tstate);
*type = exc_info->exc_type;
*value = exc_info->exc_value;
*tb = exc_info->exc_traceback;
Py_XINCREF(*type);
Py_XINCREF(*value);
Py_XINCREF(*tb);
}
static CYTHON_INLINE void
__Pyx__ExceptionReset(PyThreadState *tstate, PyObject *type,
PyObject *value, PyObject *tb)
{
PyObject *tmp_type, *tmp_value, *tmp_tb;
_PyErr_StackItem *exc_info = tstate->exc_info;
tmp_type = exc_info->exc_type;
tmp_value = exc_info->exc_value;
tmp_tb = exc_info->exc_traceback;
exc_info->exc_type = type;
exc_info->exc_value = value;
exc_info->exc_traceback = tb;
Py_XDECREF(tmp_type);
Py_XDECREF(tmp_value);
Py_XDECREF(tmp_tb);
}Cython saves/restores the current exception of tstate. Maybe we need to provide a high-level API for that as well? |
|
This is a backport of @scoder's patch to 0.29.x. (I don't know if this is helpful). https://github.com/cython/cython/compare/master...iritkatriel:exc_info?expand=1 |
I've now updated it to follow the pattern of other functions, where the is a private function that takes tstate and the public function calls it. So it adds in Include/pyerrors.h PyAPI_FUNC(PyObject*) PyErr_GetActiveException(void);
PyAPI_FUNC(void) PyErr_SetActiveException(PyObject *);and in Include/cpython/pyerrors.h PyAPI_FUNC(PyObject*) _PyErr_GetActiveException(PyThreadState *);
PyAPI_FUNC(void) _PyErr_SetActiveException(PyThreadState *, PyObject *); |
|
Just a quick comment on Cython and these changes: Cython 0.29 can build itself on Python 3.11a4 with I think there's some coroutine code left that isn't fixed by those flags, but a large chunk of Cython libraries won't use coroutines and so will work fine with these flags. Hopefully that unblocks some stuff for you. |
|
lxml does crash on the current Cython 0.29.x development branch. |
|
We have agreed on python-dev [1] that the cpython changes will not be reverted, and the issue will be fixed in cython. So I am closing this again. |
Closes mypyc/mypyc#931 Closes mypyc/mypyc#923 Related CPython modification: python/cpython#89874 python/cpython#31530
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: