PostgreSQL Extension Network
Replies: 6 comments 30 replies
-
Other IdeasSome other ideas for the format, in either the short or long term:
|
Beta Was this translation helpful? Give feedback.
-
|
That link does not lead to wheel signing patterns but to a Postgres error on Stack Overflow; could you provide a working link? |
Beta Was this translation helpful? Give feedback.
-
|
Oops, paste fail. Fixed. |
Beta Was this translation helpful? Give feedback.
-
|
I think you also need to take into account different glibc versions and different standard C library implementations. So you probably also need a libc implementation field and a libc version number field also. In some distros one can even choose between MUSL and GlibC on many source-based distros for example. And there are binary MUSL-based distros also. |
Beta Was this translation helpful? Give feedback.
-
|
It is worth noting that Python's wheel format ended up having to add both these fields (a libc and a libc version) to avoid these sorts of problems. |
Beta Was this translation helpful? Give feedback.
-
|
Ah, perhaps the distribution name should be used instead of just "linux"? I don't see libc mentioned in the Python wheel spec; the format is: For the platform tag, the example is |
Beta Was this translation helpful? Give feedback.
-
|
See PEP 600 and the discussion at https://github.com/pypa/manylinux |
Beta Was this translation helpful? Give feedback.
-
|
Oh, interesting. Looks like As for the version, perhaps it makes sense to revise the |
Beta Was this translation helpful? Give feedback.
-
|
Okay, I've collapsed the os, os_version, and arch fields into a single notional field, "platform", which contains the same basic parts, but what the supported values will be will be defined later. I added a footnote referencing the use of gnu libc and musl libc versions, as well as Homebrew macOS versions, to get the point across, and updated examples to use |
Beta Was this translation helpful? Give feedback.
-
I suspect that restriction makes this very restrictive for higher-performance versions of extensions that depend on modern CPUs, like pgvector: The list contains only archetypes ( |
Beta Was this translation helpful? Give feedback.
-
|
We could include such info in the filename, yes, although with OCI distribution it doesn't matter (the file name becomes irrelevant). But I'm trying to follow precedents here. Perhaps I need to go back and review what RPM and Yum do, though I think it's mainly that different architectures are in different directories… |
Beta Was this translation helpful? Give feedback.
-
|
It looks like yum.postgresql.org doesn't use anything other than the basic architectures. I see:
Looking through the list of repos for RedHat 12 I don't see anything different. AFAIC they don't support any specific release of x86. That said, we can always expand the necessary annotations later, but I definitely would like to include them in |
Beta Was this translation helpful? Give feedback.
-
|
There is a discussion ongoing on -hackers about approximately this topic (updating the baseline x86-64 architecture), which is why I decided to call the same issue out over here, too. |
Beta Was this translation helpful? Give feedback.
-
|
Ah, useful thread, thanks. I'll happily follow whatever the community Linux registries standardize on. And if that means adding stuff like |
Beta Was this translation helpful? Give feedback.
-
|
The second part of the change I [mentioned upthread(#discussioncomment-9885707) is that it no longer specifies that the list of OSes and architectures will come from |
Beta Was this translation helpful? Give feedback.
-
I'm not so sure about these requirements. I don't have Python on my system, nor Ruby, and Perl is only installed because PostgreSQL requires it for development. Would Python/Perl/Ruby be an installation requirement for Trunk to work with this? That sounds like a rather large set of dependencies. |
Beta Was this translation helpful? Give feedback.
-
|
No. My plan is to build SDKs and a CLI in Rust. The only other tooling required to build a trunk package would be whatever an extension requires (PGXS, compiler, pgrx, etc.). |
Beta Was this translation helpful? Give feedback.
-
|
Oh, sorry, what I mean is: if an extension requires a Perl, Python, or Ruby pipeline, the PGXN trunk CLI should be able to use it to build the extension and the trunk file. |
Beta Was this translation helpful? Give feedback.
-
|
The exerpt was from
Which implies that the trunk file's contents may include scripts later executed on the client systems, using the client's Perl/Python/Ruby - and that's where my issue lies. Is that a misinterpretation on my part? |
Beta Was this translation helpful? Give feedback.
-
|
You are not. I've included this because a fair number of extensions ship that use these interpreters. dalibo/emaj, for example, installs both Perl and PHP scripts. I want to support that pattern. Ideally the installer CLI will either prompt the user to install the required interpreters, or else install them directly by shelling out to The same issue applies for extensions that require third-party libraries, BTW, like PostGIS. Third-party dependencies are gonna be a PITA to get right, but we gotta get em right. |
Beta Was this translation helpful? Give feedback.
-
|
I've added this sentence to the end of that section:
|
Beta Was this translation helpful? Give feedback.
-
|
For extensions that use external dependencies (simples ones like pgsql-http depending on libcurl, complex ones like postgis depending on geos/gdal/proj/etc) what is the expectation for how those dependencies should be bundled? Static linking only? Is there some way to place dynamic dependencies in the pgsql tree such that the deps in the tree are picked up preferentially over deps already on the system? |
Beta Was this translation helpful? Give feedback.
-
|
No worries, @df7cb. That's kind of what I'm trying to get at with option 4: use the OS packaging system, if available, to try to satisfy dependencies. Nice thing about the first three options, though, is that one doesn't have to depend on or wait for the packaging system to build the packages — especially since not all extensions are available in all the packaging systems. Anyway, we have options. |
Beta Was this translation helpful? Give feedback.
-
|
There's a reason pip, cpan, cran, gem, etc, exist and include facilities for shipping and building deps or just flat out shipping binaries. There's really no way to build a packaged postgis "trunk" that depends on system libraries and says nothing except "I'm a postgis trunk for linux ix86". Because that doesn't even begin to describe all the variability in what might be on the system (or, more likely, not on the system). Basically the "trunk" system is only useful for people who have a perfectly calibrated set of system packages, which is to say that people might as well not use "trunk" and should just be using the... existing RPM and DEB packages from PGDG. Like, why are we here at all? |
Beta Was this translation helpful? Give feedback.
-
|
Most of my experience of systems like this is CPAN, and there's quite a lot of variation in how external dependencies are handled. Some include the binaries (or sources), some rely on the system to provide dependencies (quite common IME), some require an "Alien" CPAN module to provide the dependencies, and some do all three and let the person installing the module choose. I suspect it's much the same for the other systems. I'm a little confused by this statement:
Because the first three options I mentioned above are about embedding compiled dependencies directly in a trunk distribution, or depending on other trunk distributions that include needed binaries. This is not unlike RPM, DEB, and other packaging models.
Not every extension is available via DEB and RPM. Sometimes they get behind. And not everyone uses Linux! So if you prefer to use DEB and RPM, that's great, I see no eroding of those systems. But there are so many more extensions and OSes to support, and I'd like to build a Postgres-based (which is to say, |
Beta Was this translation helpful? Give feedback.
-
|
Sorry I spoke inelegantly. Basically my thrust is that "just use system packaging" is antithetical to what seems (to me) to be the point of the "trunk" story. Once one accepts system packaging as the answer to dependencies, one is on a fast and straight road that leads (IMO) logically to "only use system packaging". Which isn't what this conversation is about. |
Beta Was this translation helpful? Give feedback.
-
|
Ah, thank you. Yes, that makes much more sense. And it's why I'm thinking much more, from the start, about the first three options, and expect to defer the fourth until it's needed for some reason — if ever. But I'd like to see how far I can get with pgsql-http, then PostGIS to prove those patterns. Should be interesting! |
Beta Was this translation helpful? Give feedback.
-
|
NOTE: Moved to a pull request for the RFCs Book. |
Beta Was this translation helpful? Give feedback.
-
RFC: Binary Distribution Format
This RFC specifies the binary distribution format for PGXN packages, also called the trunk format.1
A trunk is a ZIP-format archive with a specially formatted file name and the
.trunkextension. It contains a single distribution nearly as it would be installed by PGXS. Although a specialized installer is recommended, a trunk file may be installed by simply copying directories of files to destinations defined by pg_config.File Format
File name convention
The trunk filename is:
Definition of variables:
package: Package name, e.g.vector,postgis,pgAdmin.version: Distribution version in SemVer format without build metadata, e.g.,0.8.6or1.0.0-beta.pg: Major version of Postgres the binary was built against, e.g.,pg15,pg16.platform: The platform the binary was built for. Will be made up of one to three hyphen-delimited2 values for the OS, version information3, and CPU architecture. Examples:any,gnulinux-amd64,darwin-23.5.0.arm64,musllinux-1.2-amd64v3. The allowed values will be defined one or more separate RFCs.Examples:
pgtap-1.0.1+pg15-any.trunkpackagespgtapversion 1.9.1, compatible with Postgres 15 (any minor release) on any platform.pair-0.32.1+pg16-gnulinux-amd64.trunkpackagespairversion 0.32.1,compatible with Postgres 16 (any minor release) on GNU libc-based Linux for amd64 CPUs.
pair-0.32.1+pg16-darwin-23.5.0-arm64.trunkpackagespairversion 0.32.1, compatible with Postgres 16 (any minor release) on Darwin version 23.5.0 (macOS) for arm64 CPUs.Escaping and Unicode
The
+in the file name indicates the division between the package name and version and the package metadata. The package name and version must not include a+. This allows the file name, without the.trunkextension, to also function as a valid SemVer.Tools producing trunks should verify that the filename components do not contain
+, as the resulting file may not be processed correctly if it does.The package name should be lowercase.
The file name components should all be UTF-8.
The filenames inside the archive are encoded as UTF-8. Although some ZIP clients in common use do not properly display UTF-8 filenames, the encoding is supported by the ZIP specification.
Parsing
Parsing of the file name takes place in four parts:
For the file name, remove the
.trunkextension. If working with the directory name (prefix) extracted from the archive, there will be no.trunkextension.Split the name into two parts at the
+sign. The left part is the package name and SemVer. The right part is the platform specification.For the left part, split on the right-most dash. If the string to the right of the dash is a valid SemVer, then the left part is the package name. If the right string is not a valid SemVer, try again at the second right-most dash and check again. Continue until a valid SemVer is produced or else fail.
Split the right string on dashes. There will be between two and four values as follows:
pg16) andany.pg16), the OS (gnulinux,darwin, etc.), and the architecture (amd64,arm64, etc.)pg16), the OS (gnulinux,darwin, etc.), the OS version (23.5.0) and the architecture (amd64,arm64, etc.)Examples:
pgtap-1.0.1+pg15-anypgtap1.0.1pg15anypair-0.32.1-beta1+pg16-gnulinux-amd64pair0.32.1-beta1pg16gnulinuxamd64pair-0.32.1+pg16-darwin-23.5.0-arm64pair0.32.1pg16darin23.5.0arm64File contents
The contents of a wheel file should unpack into a directory with the same name as the file, but without the
.trunkextension. The contents of the directory are:trunk.jsoncontains metadata necessary to install the extension. The format wil be subject to a future RFC, but at a minimum will include the trunk format version, package version, dependencies, license, language and runtime (e.g., libc implementation and version), platform metadata, and Postgres version and build configuration. Trunk installers should warn if the trunk version is greater than the version it supports, and must fail if the Trunk version has a greater major version than the version it supports.digestscontains a list of (almost) all the files in the trunk and their secure hashes. Each line lists a single file and its checksum in the BSD digest format:{algorithm} ({filename}) = {checksum}. Every file exceptdigests--- which cannot contain a hash of itself --- must be listed in this file. The cryptographic hash algorithm must be SHA-256 or better; specifically, MD5 and SHA-1 are not permitted, as signed trunk files rely on the strong hashes indigeststo validate the integrity of the archive.pgsqldirectory contains one or more subdirectories named forpg_configdirectory configurations:bin,doc,html,include,pkginclude,lib,pkglib,locale,man,share, andsysconf. Each contains the files to be installed in the correspondingpg_configdirectory.pgsql/binand begin with exactly'#!{cmd}, wherecmdis the name of the interpreter, in order to enjoy script wrapper generation and shebang rewriting at install time. They must have no extension. The list of supported interpreters will depend on the features of the installer, but one can reasonably expect support for Perl, Python, and Ruby. If no appropriate instance of the given interpreter is present, the installer may abort the installation.README,LICENSE, andCHANGELOGmay optionally be in the directory. Each must be plain text or Markdown-formatted. In the latter case, they may use the extension.md.Makefile,configurefile or any other tool for building the package contents.During extraction, trunk installers verify all the hashes in
digestsagainst the file contents. Apart fromdigestsand its signatures, installation will fail if any file in the archive is not both mentioned and correctly hashed indigests.Details
Installing a Trunk
The following descriptions will use a trunk file named
pair-0.32.1+pg16-gnulinux-amd64.trunk. Trunk installation notionally consists of two phases:digestsitself, is listed indigestalong with it valid hash digest. If any file is missing or has an invalid digest, installation should fail. If a file listed indigestsis not present, installation should fail.trunk.jsonfile. Check that the distribution is compatible with:anyis allowed for any platform#!{cmd}to point to the correct interpreter. Fail if no such interpreter is present.pgsqldirectory.References
Footnotes
With much inspiration and from and gratitude to the Python Binary distribution format. ↩
Why hyphens? They allow the entire file name, between the package name and the
.trunkextension, to be a valid SemVer. ↩See for example PEP 600 defining Python wheel tags for different versions of GNU libc and PEP 656 defining tags for different versions of musl libc. See also how Homebrew uses macOS version names in file names for its packages. ↩
Beta Was this translation helpful? Give feedback.
All reactions