Skip to content

gh-102304: Fix Py_INCREF() stable ABI in debug mode #104763

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 6, 2023
Merged

gh-102304: Fix Py_INCREF() stable ABI in debug mode #104763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions Doc/whatsnew/3.13.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,15 @@ Build Changes
:file:`!configure`.
(Contributed by Christian Heimes in :gh:`89886`.)

* C extensions built with the :ref:`limited C API <limited-c-api>`
on :ref:`Python build in debug mode <debug-build>` no longer support Python
3.9 and older. In this configuration, :c:func:`Py_INCREF` and
:c:func:`Py_DECREF` are now always implemented as opaque function calls,
but the called functions were added to Python 3.10. Build C extensions
with a release build of Python or with Python 3.12 and older, to keep support
for Python 3.9 and older.
(Contributed by Victor Stinner in :gh:`102304`.)


C API Changes
=============
Expand Down
22 changes: 8 additions & 14 deletions Include/object.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -585,20 +585,14 @@ decision that's up to the implementer of each new type so if you want,
you can count such references to the type object.)
*/

#ifdef Py_REF_DEBUG
# if defined(Py_LIMITED_API) && Py_LIMITED_API+0 < 0x030A0000
extern Py_ssize_t _Py_RefTotal;
# define _Py_INC_REFTOTAL() _Py_RefTotal++
# define _Py_DEC_REFTOTAL() _Py_RefTotal--
# elif !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030C0000
#if defined(Py_REF_DEBUG) && !defined(Py_LIMITED_API)
PyAPI_FUNC(void) _Py_NegativeRefcount(const char *filename, int lineno,
PyObject *op);
PyAPI_FUNC(void) _Py_IncRefTotal_DO_NOT_USE_THIS(void);
PyAPI_FUNC(void) _Py_DecRefTotal_DO_NOT_USE_THIS(void);
# define _Py_INC_REFTOTAL() _Py_IncRefTotal_DO_NOT_USE_THIS()
# define _Py_DEC_REFTOTAL() _Py_DecRefTotal_DO_NOT_USE_THIS()
# endif
PyAPI_FUNC(void) _Py_NegativeRefcount(const char *filename, int lineno,
PyObject *op);
#endif /* Py_REF_DEBUG */
#endif // Py_REF_DEBUG && !Py_LIMITED_API

PyAPI_FUNC(void) _Py_Dealloc(PyObject *);

Expand All @@ -616,8 +610,8 @@ PyAPI_FUNC(void) _Py_DecRef(PyObject *);

static inline Py_ALWAYS_INLINE void Py_INCREF(PyObject *op)
{
#if defined(Py_REF_DEBUG) && defined(Py_LIMITED_API) && Py_LIMITED_API+0 >= 0x030A0000
// Stable ABI for Python 3.10 built in debug mode.
#if defined(Py_REF_DEBUG) && defined(Py_LIMITED_API)
// Stable ABI for Python built in debug mode
_Py_IncRef(op);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_Py_IncRef was only added in 3.10, so I don't think the Py_LIMITED_API+0 >= 0x030A0000 check can go away.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The #else code path uses op->ob_refcnt_split and _Py_IsImmortal() which were added to Python 3.12. Choose your poison: no code path work on Python 3.9 and older.

I'm convinced that building a C extension targeting an old version of the stable ABI with a new Python never worked because of these subtle implementation details (especially around Py_INCREF/Py_DECREF). I would prefer to always convert Py_INCREF/Py_DECREF to opaque function calls in the stable ABI: the current implementation is badly broken because of that. Honestly, I don't think that relying on union, endianness, accessing direclty PyObject members, is a future proof. Extract of Py_INCREF implementation in Python 3.13:

    // Portable saturated add, branching on the carry flag and set low bits
    PY_UINT32_T cur_refcnt = op->ob_refcnt_split[PY_BIG_ENDIAN];
    PY_UINT32_T new_refcnt = cur_refcnt + 1;
    if (new_refcnt == 0) {
        return;
    }
    op->ob_refcnt_split[PY_BIG_ENDIAN] = new_refcnt;

It's not backward compatible at least.

Moreover, this change only impacts Python built in debug mode. That's really a corner case: nobody should distribute a wheel package with Python built in debug mode, especially when the stable ABI is targeted.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That refcnt_split is going to change again anyway - @eduardo-elizondo FYI.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That refcnt_split is going to change again anyway - @eduardo-elizondo FYI.

which makes my point even more relevant :-)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C extensions built for the stable ABI with Python 3.10 or older don't know about immortal objects :-(

Copy link
Contributor

@eduardo-elizondo eduardo-elizondo Jun 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not backward compatible at least.

@vstinner let's dive a bit more into this! I don't think that this is actually an issue but I want to see if I'm missing something here.

To be more specific do you have an example of what exactly will not be backwards compatible? The semantics did change, but that does not mean that it will be a backwards incompatible change. The updates to the headers is done in a way that it doesn't matter if previous versions don't know about immortal headers, it will still work as intended.

^ Note that this comment is not necessarily about this PR as it totally makes sense to use _Py_IncRef directly in debug mode.

#else
// Non-limited C API and limited C API for Python 3.9 and older access
Expand Down Expand Up @@ -647,8 +641,8 @@ static inline Py_ALWAYS_INLINE void Py_INCREF(PyObject *op)
# define Py_INCREF(op) Py_INCREF(_PyObject_CAST(op))
#endif

#if defined(Py_REF_DEBUG) && defined(Py_LIMITED_API) && Py_LIMITED_API+0 >= 0x030A0000
// Stable ABI for limited C API version 3.10 of Python debug build
#if defined(Py_REF_DEBUG) && defined(Py_LIMITED_API)
// Stable ABI for Python built in debug mode
static inline void Py_DECREF(PyObject *op) {
_Py_DecRef(op);
}
Expand Down
2 changes: 0 additions & 2 deletions Lib/test/test_stable_abi_ctypes.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 0 additions & 9 deletions Misc/stable_abi.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2428,12 +2428,3 @@
added = '3.12'
[const.Py_TPFLAGS_ITEMS_AT_END]
added = '3.12'

[function._Py_IncRefTotal_DO_NOT_USE_THIS]
added = '3.12'
ifdef = 'Py_REF_DEBUG'
abi_only = true
[function._Py_DecRefTotal_DO_NOT_USE_THIS]
added = '3.12'
ifdef = 'Py_REF_DEBUG'
abi_only = true
2 changes: 0 additions & 2 deletions PC/python3dll.c
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.