Future of vendored libraries

[lonvia]

Some libraries we depend on are included in the osm2pgsql repository in the contrib directory. This kind of vendoring is usually not a good idea because it makes packaging more difficult. There was a good reason to add the libraries anyway, usually because we needed very new features not available in the distributions. Many of the reasons have now gone away and it is time to review the vendoring decision. This issue is to track the state of these libraries.

Protozero

Used by Libosmium. Widely available in versions that are okay for us.

• Debian: Buster has a version that's too old, buster has reached EOL already.
• macOS: Homebrew doesn't have Protozero as its own library but ships it with libosmium.
• Windows: Available with vcpkg.

After Trixie is released (expected in summer 2025) at the latest we can remove support for Buster at which point we can remove our copy.

Libosmium

Used all over for basic OSM code. Widely available in versions that are okay for us.

• Debian: Same as with Protozero.
• macOS: Current versions are in homebrew.
• Windows: Available with vcpkg.

CLI11

Used for parsing command line arguments. Included because it is missing in Debian versions before Bookworm.

• Debian: Version 2.1.2 available in Bookworm, not available in Bullseye or before.
• Fedora: Available since Fedora 30 from 2019, long EOL. Current Fedora versions still in support have version 2.
• macOS: Available in homebrew.
• Windows: Available with vcpkg.

We can probably remove our copy once Bullseye reached EOL.

fmt

Used all over the code for string formatting. Although widely available in distributions we have our own copy because fmt doesn't provide backwards compatibility between major versions and they are releasing a new major version about once a year.

Long-term this library should be replaced by using the equivalent functions from the C++ library introduced in C++20. But support in C++ libraries is lagging a bit, so it will be some time until we can switch.

Catch2

Used for unit tests. We are currently stuck at version 2, because version 3 is not available in Debian yet.

Note the confusing version numbering: "Catch" is version 1 from long ago, "Catch2" is version 2 and above. Version 1 and 2 are header only, "Catch 2 v3" is a proper library. Versions 1 and 2 are not maintained any more.

• Debian: Debian catch2 is version 2 in Bookworm, Trixie will have version 3.
• Fedora: Package catch2 is version 2, package catch seems always to be the "current" version, currently 2. Version 3 available since Fedora 38.
• macOS: Packages of version 2 are available as catch, version 3 as catch2.
• Windows: catch2 in vcpkg is version 3.

Catch2 is only a development dependency. That means we can be less strict regarding compatibility with Linux distributions. Compatibility with latest distros should be sufficient as long as we find a solution for the CI. We should start experimenting with Catch2-v3 once Debian Trixie is out and then decide about the future. We need in particular figure out how catch2v3 handles compatibility between minor versions or if we'd be stuck with the latest version.

[gdt]

I maintain the osm2pgsql package in pkgsrc, which is the official packaging system on NetBSD,but also operates on about 20 systems. (I use it to load geofabrik extracts of Massachusetts into postgis, for use in qgis). I am in general in favor of de-vendoring, and a quick check shows that many of these are not yet in pkgsrc (we do have catch2 3.7.1). I see that as a clue that they are not widely used -- adding packages (for upstreams with reasonable build systems) is easy, and if they were a dependency of something else they probably would have been added (we have about 20K packages). I'll have a look at the list and see what can be added.

It's not clear if the code can build with catch2 v3, or if there is a requirement for catch2 v2. I understand your point about "development dependency" but pkgsrc has the idea that one should be able to run tests for any package ("make test") without understanding them, as a user, and that this is important because of the wide variety of operating systems and CPU types.

I would suggest that Debian, Homebrew, vcpkg is a subset of packaging in general, covering only part of the landscape even if most users, and would suggest Fedora, maybe Suse, FreeBSD ports, pkgsrc as well for assessing.

It would be nice to edit in the upstream homepage/repo above, for each package.

(I understand where you are coming from about fmt. Waiting for now and stopping seems like a good plan.)

[gdt]

Trying to package CLI11, it tries to download catch2 during the build. Just pointing this out, as it's bad behavior from a packaging viewpoint. After providing catch2, it built examples, which is also wrong, but that was not hard to turn off. So I expect it can be packaged in under an hour in most packaging systems, by someone who had never heard of it but is skilled at packaging.