PEP about abi4 etc. (placeholder)

[gvanrossum]

Sam Gross (@colesbury) has an elaborate proposal for abi4. He's going to draft a PEP.

Things I recall:

• noGIL builds need PyObject_HEAD to be 32 bytes (it's 16 bytes currently -- all on 64-bit platforms).
• There's a multi-dimensional compatibility matrix, involving at least the ABI version selected in the source code (Py_LIMITED_API), the wheel's abi tag (abi3 or abi4), the wheel's CPython version tag (e.g. cp312), the CPython version used to build the extension, the CPython used to load the extension.
• There's a possible hack involving weak linking (Mac/Linux) or "dlsym" (sp?) hacks (Windows).
• It's easy to get confused by the distinction between the "full" C API and the limited API or the stable ABI. And there's a "full" ABI as well (for "version specific" wheels).

[vstinner]

I have many ideas how what to put in an abi4. But when it comes to noGIL, I have one request: make PyObject structure empty. The PyObject members would be stored before PyObject* pointer, similar to the internal PyGC_Head structure. It implies to break the backward compatibility since C extensions can currently access directly to PyObject.ob_type and PyObject.ob_refcnt. C extensions would have to go through functions:

• Py_REFCNT(), Py_SET_REFCNT()
• Py_TYPE(), Py_SET_TYPE()

The main drawback is that currently, sizeof(PyObject) is leaked into a few places of the abi3:

• PyType_FromSpec(): [C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type python/cpython#83754 (comment)

• https://mail.python.org/archives/list/python-dev@python.org/message/PGKRW7S2IUOWVRX6F7RT6VAWD3ZPUDYS/

• https://vstinner.github.io/c-api-abstract-pyobject.html

I'm considering to write a first PEP only to make PyObject opaque: it doesn't imply to make PyObject empty, but only to prevent direct access to PyObject. IMO it's pre-requirement to be able to move PyObject members before PyObject*, to have an ABI which doesn't include PyObject members.