C API exposes pointers to real Python objects

[steve-s]

The fact that PyObject* is a real pointer to some runtime data-structure can be (ab)used in many ways. Those may slowly become part of the contract of the C API even if not intended.

First problem is that by comparing the pointer one can check if two objects are the same. This forces the Python engine to keep mapping between its own internal objects and whatever is handed out to extensions, because "whatever is handed out to extensions" must be always the same for the same object.

It is trivial if the objects have stable address, but with a moving GC this is not the case. Both Java and .NET and many other systems have moving GC, so their objects do not have stable address, and explicit mapping must be maintained just for the sake of maintaining the expectations of the extensions that PyObject* is a real pointer even if opaque.

What I've seen in other languages is also this "nice" trick where people copy the definition of a C structure and then cast the opaque pointer they've gotten from the runtime to that struct so that they can access its internals. It can go as far as defining several versions of said struct for several versions of the interpreter and choosing one at runtime. Now, all the complications and breakages that this causes are the problem of people who decide to use such tricks (and maybe sometimes it is necessary, I am not judging). I am pointing this out as another example why not exposing pointers to the real data (or at least specify that what people get from API should be seen as opaque identifier of Python object, but not a pointer to it) would be better.

[vstinner]

IMO the C API long term goal should be to convert PyObject* to fully opaque handles, as HPy does. I designed PEP 620 around this concept. The implementation is a long journey full of changes breaking the backward compatibility. IMO it's worth it to keep Python competitive, solve some ABI issues, make the implementation of the C API easier in other Python impementations, etc.

[vstinner]

See also issue #31.

[vstinner]

I suggest to continue the discussion in capi-workgroup/api-revolution#7