or-tools should use released versions of pybind11_protobuf and pybind11_abseil

[StefanBruens]

Version: main/v9.9

Currently, both dependencies are pulled in using a "random" version from git.

Unfortunately, both projects have not released any version yet, but this would probably change when encouraged from a high profile, dependent project like or-tools.

In the past, the changing versions have already led to various build failures for or-tools itself. Depending on "official" versions would also make the dependencies more usable outside of or-tools, and reduce problems due to loading different versions of the python modules at the same time.

[Mizux]

related to:

• Create a release package to enable adding pybind11_protobuf to Bazel Central Registry pybind/pybind11_protobuf#139
  Already got an off dicussion with rwgk (Googler in charge of this project) currently no time and knowledge to do it....

[StefanBruens]

Apparently, the mess is even greater than I have imagined:

1. (latest) pybind11_protobuf depends on a "deprecated", internal header of protobuf: proto_api.h not included in python packages protocolbuffers/protobuf#9464
2. the "replacement" has no stable API or ABI, and does not plan to do any releases: https://github.com/protocolbuffers/protobuf/tree/main/upb

So or-tools added a new, hard dependency in 9.7 which was a dead end already at that time ...

These projects are all Google projects - do they even talk to each other?

[lperron]

everybody is doing its best with his resources and priorities.

Fixing pybind11 arcane dependencies it not high on that list.

and upb is integrated in the protobuf code.

[StefanBruens]

everybody is doing its best with his resources and priorities.

I just fail to understand why making someone else live harder is seen as totally fine, instead of coming up with something working out of the box. There are plenty of people who have a hard time building pybind11_protobuf, and thus or-tools.

Fixing pybind11 arcane dependencies it not high on that list.

Which part is arcane? or-tools dependency on it is fairly new. pybind11_protobuf may use a "deprecated" API, but as there are no alternatives it should be considered recent.

and upb is integrated in the protobuf code.

It does not matter if upb is integrated in the protobuf code. It does not provide the required APIs needed for generating the bindings. And as long as or-tools is using pybind11_protobuf upb is just irrelevant.

[StefanBruens]

@lperron Can you tell us why this was closed? None of the mentioned issues have been addressed.

[lperron]

Won:t fix. I have no way of changing any of this. Le mar. 4 juin 2024, 00:23, StefanBruens ***@***.***> a écrit :

…

@lperron <https://github.com/lperron> Can you tell us why this was closed? None of the mentioned issues have been addressed. — Reply to this email directly, view it on GitHub <#4204 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUPL3OA7HLCXYKA6THMZDLZFTUFJAVCNFSM6AAAAABG3GCZSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBWGIZDOMZRHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[StefanBruens]

@lperron - one first step would be to integrate back the patches carried here to both pybind11_abseil and pybind11_protobuf.

[lperron]

Agreed. Not that simple. Le jeu. 13 juin 2024, 03:00, StefanBruens ***@***.***> a écrit :

…

@lperron <https://github.com/lperron> - one first step would be to integrate back the patches carried here to both pybind11_abseil and pybind11_protobuf. — Reply to this email directly, view it on GitHub <#4204 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUPL3KDO2PXH7TJ7K4UVSDZHFUUBAVCNFSM6AAAAABG3GCZSWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRVGE4DONBYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mering]

I recently uploaded recent versions of pybind11_abseil and pybind11_protobuf to the Bazel Central Registry (BCR).

There probably won't be tagged releases of these projects as no such process exists and there are no resources to create such a process. Nevertheless, the BCR will probably contain "important" snapshots.

This is only about Bazel which is similar to what is used internally. I see a lot of patches concerning CMakeLists.txt. Are they necessary to make or-tools build with CMake? How does it even work as the patches are added in the Bazel WORKSPACE file?

[StefanBruens]

Agreed. Not that simple.

But not magic either. Went ahead and submitted patches for CMake builds to pybind11_protobuf.

[StefanBruens]

I recently uploaded recent versions of pybind11_abseil and pybind11_protobuf to the Bazel Central Registry (BCR).

There probably won't be tagged releases of these projects as no such process exists and there are no resources to create such a process. Nevertheless, the BCR will probably contain "important" snapshots.

This is only about Bazel which is similar to what is used internally. I see a lot of patches concerning CMakeLists.txt. Are they necessary to make or-tools build with CMake? How does it even work as the patches are added in the Bazel WORKSPACE file?

What process is needed? Tag a release, done.

Importing fairly random revisions from github repositories is just a nightmare. If you just care for one (your) single downstream package this may seem to work fine. As soon as a package starts to pull in different versions of the same package via various dependencies, hell breaks loose.

[mering]

@StefanBruens what value does a tag with a random tag number on a random revision provide you in addition to the already unique revision?

You seem to want something like semver with compatibility promises which requires a process to capture incompatibilities.

Version resolution is handled by Bzlmod for you.

[Mizux]

@mering most OSS google projects use/provide a CMake based build (re2, protobuf, or-tools, googletest, benchmark, abseil-cpp) to integrate nicely with the oss distro package manager ecosystem -> bzlmod is off topic here....
note: curious to know how you could generate a distro package from the bazel build with dependencies comming fromthe system package manager and do this with the dozen of major distro which are not always up to date...

Since we (google/or-tools) use pybind11 (and pybind_abseil & pybind_protobuf) for few of our or-tools C++ components wrapper, we now need a release tag for each of these dependencies for condaforge/homebrew/etc to be able to bump to our last release (and build our python package). Also outside Google you need a consensus on the version (i mean a commit hash) use among project.
The easiest way to do it is to perform a release on github (like all others google projects already did !) so we can have a signal and a consensus in the OSS community/package maintainers...

[StefanBruens]

@StefanBruens

Version resolution is handled by Bzlmod for you.

Not everyone uses bzlmod (actually, hardly anyone outside of google does ...).