proto_api.h not included in python packages

[maxnoe]

What version of protobuf and what language are you using?
Version: master/3.19/3.14, seems to affect all recent versions
Language: C++/Python

What operating system (Linux, Windows, ...) and version?

Linux

What runtime / compiler are you using (e.g., python version or gcc version)

Not really applicable

What did you do?

Trying to compile a python extension using pybind11_protobuf, the Header proto_api.h, which is inside the python package in this repo is not found.

It seems this header is not included when installing the python package, e.g. in wheels or the arch linux python-protobuf package.

What did you expect to see

Header is included.

What did you see instead?

See above.

[haberman]

proto_api.h will be going away soon, as we will be releasing a new implementation of the Python Protobuf library that is based on https://github.com/protocolbuffers/upb instead of C++ protos.

The new Python Proto library cannot support the existing proto_api.h interface, as Python objects will not have a proto2::Message* inside of them. Instead they will have a upb_Message*.

We will likely expose a new/different capsule eventually to allow pybind11 users to access the upb proto object. But this will not be in the first version.

[maxnoe]

What implications does this have for creating python bindings to a C++ library using the C++ protobuf library and not upb?

It sounds like it will require serialization/deseralization to transform c++ message instances into python upb instances.

That would be very unfortunate for us and would bring us back to the current state, which we had hoped to improve by using pybind11_protobuf

[StefanBruens]

We will likely expose a new/different capsule eventually to allow pybind11 users to access the upb proto object. But this will not be in the first version.

@haberman - do you have any way forward for making pybind11_protobuf buildable again? Something that is available now, not "eventually"?

As long as you don't have a replacement, you should not break dependent projects.

[haberman]

I believe the pybind11_protobuf build should be fixed by: pybind/pybind11_protobuf#165