Python libraries all build using the same generic name

[thunderbiscuit]

The attempt at the publishing of the 0.28.0 release of bdkpython failed.

This is because the new, cleaner build workflow built libraries that all had the same name, and their upload to PyPI therefore failed (you can't publish more than one library with the same name). The library they all attempted to upload was

bdkpython-0.28.0-py3-none-any.whl

Whereas the names we're looking for (from previous versions) are similar to the following:

bdkpython-0.27.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
bdkpython-0.27.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
bdkpython-0.27.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

bdkpython-0.27.1-cp310-cp310-macosx_12_0_universal2.whl
bdkpython-0.27.1-cp39-cp39-macosx_12_0_universal2.whl
bdkpython-0.27.1-cp38-cp38-macosx_12_0_universal2.whl

bdkpython-0.27.1-cp310-cp310-win_amd64.whl
bdkpython-0.27.1-cp39-cp39-win_amd64.whl
bdkpython-0.27.1-cp38-cp38-win_amd64.whl

After exploring solutions, the cleanest approach seems to be what is described here.

The one slight issue with that is that for macOS, even if you build a universal wheel, it still names it according to the platform onto which is was built. This is fixed by specifying the command line flag --plat-name macosx_12_0_universal2.

I created this issue to explain how I came to the fix I used for the 0.28.0 release, and will link to this issue in the script that uses the trick.

[thunderbiscuit]

Follow up small issue on this run:

 ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Binary wheel 'bdkpython-0.28.1-cp310-cp310-linux_x86_64.whl' has an    
         unsupported platform tag 'linux_x86_64'.        

Attempting the --plat-name manylinux_2_17_x86_64.manylinux2014_x86_64 argument to match the library names of the previous versions.

[thunderbiscuit]

Follow up issue on this run:

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Binary wheel                                                           
         'bdkpython-0.28.2-cp310-cp310-manylinux_2_17_x86_64_manylinux2014_x86_6
         4.whl' has an unsupported platform tag                                 
         'manylinux_2_17_x86_64_manylinux2014_x86_64'.     

Note that our old libraries for Linux were called with the following convention:

bdkpython-0.27.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
bdkpython-0.27.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
bdkpython-0.27.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Which includes the double platform tag manylinux_2_17_x86_64.manylinux2014_x86_64 (notice how they are separated by a dot). When using the --plat-name manylinux_2_17_x86_64.manylinux2014_x86_64 argument to the build command, the name of the file actually created replaces the dot with an underscore. The filenames become something like this:

bdkpython-0.28.2-cp310-cp310-manylinux_2_17_x86_64_manylinux2014_x86_64.whl

This last part is not a valid platform name, hence the error.

Why the double platform naming?

After wondering about whether the naming could be simplified I figured I'd ask ChatGPT what the naming structure represented and why both were needed (or not). This is the answer:

Me:

Can you tell me more about the naming convention for this python library name? mylib-0.27.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

ChatGPT:

Yes, sure! Here is an explanation of the naming convention for the Python library name you mentioned: mylib-0.27.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

1. mylib-0.27.1: This part of the name specifies the name and version of the library. In this case, the library name is mylib and the version is 0.27.1.
2. cp310-cp310: These two tags indicate the Python version and the ABI (Application Binary Interface) compatibility tag for which the wheel is built. In this case, it specifies that the wheel is built specifically for Python 3.10 and is compatible with the ABI3 of Python 3.10.
3. manylinux_2_17_x86_64: This part of the name indicates the Linux platform for which the wheel is built. Many Linux distributions use the same base image, known as "manylinux", which is a reference to a set of compatibility standards for Linux-based platforms that provide binary compatibility across different versions of Linux distributions. In this case, the wheel is built for a Linux platform that supports the manylinux_2_17 standard, which means it's compatible with most modern Linux distributions.
4. manylinux2014_x86_64: This tag specifies the version of the manylinux standard that the wheel is built for. The manylinux_2014 standard is a specific version of the manylinux standard that was released in 2014. This tag is included for compatibility with older Linux distributions that may not support the newer manylinux_2_17 standard.
5. .whl: This is the file extension for Python wheels, which are a standard format for packaging and distributing Python libraries.
   So, the entire name mylib-0.27.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl indicates that this is a wheel file for the mylib library version 0.27.1, built for Python 3.10 and its ABI3 compatibility, and targeted for a Linux platform that supports the manylinux_2_17 standard with compatibility for the older manylinux2014 standard.

Me

Ok cool. But what exactly is the manylinux_2_17 standard?

ChatGPT:

The manylinux_2_17 standard is a set of compatibility standards for Linux-based platforms that provide binary compatibility across different versions of Linux distributions. It is a reference to a specific version of the manylinux standard.
The goal of the manylinux standard is to provide a consistent environment for building and distributing binary wheels that can be installed on a wide range of Linux distributions. This is important because Linux distributions can have different versions of system libraries, which can cause incompatibilities when installing binary packages.
The manylinux_2_17 standard was introduced in 2017 and is based on the glibc version 2.17, which is a widely used C library in Linux. This standard requires that the compiled code of a wheel package is linked against glibc version 2.17 or later and that the package doesn't link against any other system library. Additionally, the standard specifies other compatibility requirements such as the minimum kernel version and support for the UTF-8 character encoding.
The manylinux_2_17 standard is widely adopted by many Linux distributions, including CentOS 7 or later, Debian 8 or later, Fedora 24 or later, and Ubuntu 14.04 or later. By adhering to the manylinux_2_17 standard, Python packages can be distributed as binary wheels that are compatible with a wide range of Linux distributions, reducing the need for users to compile packages from source code.

Proposed solution

From the above answer, it feels to me like using the manylinux_2_17 uniquely (i.e. dropping the manylinux2014_x86_64 platform tag) is probably a good solution, since it will be compatible all the way back to Debian 8, which is... old, and so is GLIBC 2.17. We don't have, at the moment at least, reason to expect users would run bdkpython on particularly old operating systems, and if they ever need to we can adjust at that point, or ensure they are able to build the library themselves for their particular requirements (this approach also applies well to the PyPy releases, which we removed in favour of simply keeping the CPython ones.

[thunderbiscuit]

These fixes worked for the 0.28.3 release (see workflow here).

[thunderbiscuit]

I am now building separate libraries with different names for the macOS arm64 and x86_64 architectures. The library names are as follows:

# arm64
bdkpython-0.30.0.dev0-cp310-cp310-macosx_11_0_arm64

# x86_64
bdkpython-0.30.0.dev0-cp310-cp310-macosx_11_0_x86_64