Supporting multiple ABI versions at once

[steve-s]

In order to be able to evolve the API, but keep ABI stability, it would be useful if one Python version could support multiple ABI versions at once. How would it look like:

• extension A is compiled with ABI 1.0
• Python wants to change something in the API in an ABI incompatible way, ABI 2.0 is created
• extension B is compiled with ABI 2.0, binary artifacts of extension A are not updated (there can be many reasons for that)
• still Python can load and run both A and B at the same time

[gvanrossum]

This is the kind of compatibility story that we never solved for the Python 2-to-3 transition. It set us back by a decade. So it would behoove us to do better: even though the scope here is smaller (just stable-abi-using extensions), the size of the affected ecosystem is likely even larger than it was then.

I suspect that this can only be solved properly by constraining ABI 2.0 in some way, or at least by some clever hacks.

For example, let's say we want to change PyList_GetItem to stop returning a borrowed reference. Just changing the function will cause unacceptable leaks. As a solution, we can rename the function to PyList_GetItem2, and deprecate the old version. (We could adopt any naming scheme we want to here, the 2 suffix is just an example.) We can even do something with macros so that calling PyList_GetItem from freshly compiled code will produce an error (either at compile time or at runtime) but calling it from a previously compiled wheel continues to do the right thing (returning a borrowed reference).

But now we're stuck with the ugly PyList_GetItem2 name. To solve that, secondary, problem, we can use macros to let you say PyList_GetItem and map that to PyList_GetItem2, but only when you define some other flag that says you know that PyList_GetItem returns a new reference. If you were to look in the DLL or .so file you'd see PyList_GetItem2, but in the source you see the clean name.

Still, for readers of extension module source code it's a bit confusing that sometimes PyList_GetItem returns a borrowed reference and other times a new one, so it may be better to uniformly rename all ABI calls, new and old? But to what? PyList_GetItem is really the nicest name for this functionality.

At this point it's a matter of choice, though -- we certainly can design an ABI 2.0 in such a way that it can coexist with ABI 1.0 at runtime, as long as the function names seen at the linker level are different. (Functions whose semantics are unchanged don't need to be different, though they could be for consistency. Another choice.)

[steve-s]

I believe that the PyList_GetItem example is only a tip of an iceberg. Even more problematic examples are those where complex interactions between several functions change. For example imagine overhauling the module initialization sequence, changing the way errors are reported, or changing the tp_traverse and tp_clear mechanism such that GC can run on a separate thread.

Another class of compatibility issues related to ABI stability is stability of some implicit contracts such as GIL vs. per interpreter GIL vs no GIL. For that one we should allow extensions to communicate to the runtime what expectations they have (i.e., declare some flags/options). This must be designed in a way that allows binary compatible evolution.

we certainly can design an ABI 2.0 in such a way that it can coexist with ABI 1.0 at runtime, as long as the function names seen at the linker level are different

That's one option. Another approach was taken by HPy, where there is no linking, the only "stable forever" contract is that the extension must expose symbols get_required_hpy_major_version_{ext name} and get_required_hpy_minor_version_{ext name}. HPy implementation proceeds according to what these functions return.

For now, it creates an HPyContext ABI compatible with given HPy version (a struct of pointers to API functions) and continues with module initialization passing the right HPyContext version when calling any functions exposed by the extension. Calling an API function means calling a pointer to that function retrieved from the HPyContext struct. The memory layout of HPyContext is the ABI here.

If another extension is loaded that requires different HPyContext version, we can create another instance of another version of the struct and pass that to the functions exposed by the other extension without affecting the first extension.

If we decided to ditch the HPyContext concept or change it fundamentally, we'd still be fine. The initialization sequence (after calling get_required_hpy_(major|minor)_version_{ext name}) for the new HPy version can do something different, but the initialization sequence for older HPy versions would stay the same.

Of course, another issue is how to support old/legacy extensions, which make some assumptions such as that there is GIL, and at the same time support new extensions/code that want to run with per interpreter GIL. Inherently we need a way to at least emulate the GIL for the old extensions while not actually doing it. This problem is orthogonal to how the API/ABI looks like. What can help with the HPy design is that the different HPyContex implementations can take different code-paths (one could even generate them at runtime) and one can, for example, allocate HPyContext per extension and store something useful in it, such as some lock, index to some table, etc.

There is a blog post about the HPy design with more details. Note that it was written before we introduced the get_required_hpy_(major|minor)_version_{ext name} and some of the implementation details are not up-to-date, but the overall design approach stays the same.

https://medium.com/graalvm/hpy-binary-compatibility-and-api-evolution-with-kiwisolver-7f7a811ef7f9

[encukou]

But now we're stuck with the ugly PyList_GetItem2 name.

Let's just accept that and move on?
A macro to rename PyList_GetItem2 to PyList_GetItem would get pretty confusing for non-C languages, which can't use macros. (We had a similar situation with PyArg_Parse -> _PyArg_Parse_SizeT. People cope, but it's pretty unfriendly.)
We can deprecate the old version, so it warns and (hopefully) doesn't show up in autocompletion. But I wouldn't reuse the same name to mean something different.

For the other issues: IMO, the new stable API should only contain functions (and types needed for their arguments). It should never allow you to get any complex type back (so if you create a module using PyModuleDef_v1, but Python uses PyModuleDef_v2 now, it doesn't need to remember/synthesize a PyModuleDef_v1). Similarly, it shouldn't give any function pointers (signatures also change).
Then, we should be fine with versioned names.

HPy's approach -- versioning the whole API -- seems good if you need to support alternative implementations, but IMO CPython would be better off versioning individual functions.

[steve-s]

HPy's approach -- versioning the whole API -- seems good if you need to support alternative implementations, but IMO CPython would be better off versioning individual functions.

I think there can be some situations where versioning of individual functions may not be sufficient as explained in my previous comment. On the top of that, with the HPy approach the extension doesn't even get any access at all to different API/ABI versions, so it cannot accidentally/intentionally (people can be creative sometimes) mix incompatible versions.

[erlend-aasland]

[...] IMO CPython would be better off versioning individual functions.

We already do that; for example Py_FinalizeEx() is a new variant of Py_Finalize(). One improvement could be to formalise replacement API naming. FWIW, SQLite uses _vX suffixes (sqlite3_prepare(), sqlite3_prepare_v2(), sqlite3_prepare_v3()).

[vstinner]

As a solution, we can rename the function to PyList_GetItem2

I proposed PyList_GetItemRef() :-) Or just use PySequence_GetItem() which already exists and returns a new strong reference.

We already do that; for example Py_FinalizeEx() is a new variant of Py_Finalize(). One improvement could be to formalise replacement API naming

Shortly after implementing PEP 445 which added the new PyMemAllocator structure, I changed its name to PyMemAllocatorEx when I added a new calloc pointer :-( I was afraid of someone using the structure without setting the calloc member because they didn't pay attention that the structure changed. It made me sad that the structure name had to change. What if tomorrow we add allocator functions taking an alignment argument? Change the structure name again? Cross fingers and hope that developers will read the doc?

[vstinner]

By the way, I never understood why PyModule_Create2() has a 2 suffix. Why should I not use PyModule_Create()? Is it going to bite me? Why was it kept? In terms of API migration, I'm not convinced by this example.

[vstinner]

I vaguely recall that some people tried running Python 2 and Python 3 in the same process. Honestly, that would be cool if it worked, especially if it would be easy to share objects betwen the two "spaces".

If you imagine doing the same with thread-safe ("nogil") C extensions with old (not thread-safe) C extensions, it would be cool to be able to use the two at the same time. But the ABI may be very different and so incompatible. Look also at PyObject.ob_refcnt changes caused by immortal objects (PEP 683). The ABI differences between recent and old Python is only growing.

It's very appealing to imagine that loading an old and a new ABI in the same process would work. The problem is that the Python ABI is badly designed in term in backward and forward compatibility. Honestly, IMO even the stable ABI is still leaking too many implementation details (like PyObject members).

The Linux glibc uses symbol versionning which is pretty cool. It helps to change the ABI and still support running old unmodified binaries. It relies on Linux ELF symbol versionning. If we want something similar in a portable way, we can just change the symbol name. Like: PyDict_GetItem (version 1), PyDict_GetItem2 (version 2), etc. The ABI and the API can use different names. For example, the API can use #define PyDict_GetItem PyDict_GetItem2 (the old API name uses the new ABI symbol).