Drop support for manylinux1 images January 1st, 2022

[mayeut]

CentOS 5 support - which is used as the base image for manylinux1 - has been EOL for almost 4 years.
There's now very little need for manylinux1 compatible wheels (less than 0.1% of glibc systems downloads on PyPI in October 2020, c.f. #542 (comment))

manylinux1 will not receive any new features from now on (e.g. no CPython 3.10).
Support will end January 1st, 2022.

What this means on January 1st, 2022:

• the manylinux1 branch will be removed (a tag will be created to keep history)
• existing images will be kept in quay.io

Amend on January 22nd, 2022:

• as long as automated updates are working, the branch will be kept around
• once automated updates start failing, what was first planned on January 1st, 2022 will happen.

[henryiii]

Just to be complete, since the “very little need” is misleading, the reason 99% users get manylinux1 is not glibc version, but pip version. The version of pip in the most popular distro (Ubuntu), the most popular version (18.04) is pip 9. Even if you install Python 3.8 (it has 3.6 by default), you still get Pip 9. It’s quite unsupported to overwrite the system package manager’s pip, too. Making a new venv defaults to pip 9 even if you do update pip manually (such as a user install). So even though manylinux2014 is supported with a new enough pip, using the default tooling, only manylinux1 is.

This issue should be fixed from recurring in the future by the perennial manylinuxes, once distros ship that.

Edit: forgot about CentOS 7!

[mayeut]

@henryiii,

the reason 99% users get manylinux1 is not glibc version

Can you provide the source of this stat ?

The stats in the linked comment suggested less than 25% last October
Yes those stats are probably biased by the fact that CI accounts for a number of them

I'm hopping that by the end of the year, things will have evolved in the right direction. Some major packages like cryptography already moved to manylinux2010 only and I guess this will get more users using an upgraded pip.

You'll still be able to use manylinux1 after end of support. Everything will be in a frozen state, that's all. No new features and no more updates.

[mayeut]

I updated the stats by looking at February downloads on PyPI:

SELECT t0.cpu, t0.num_downloads, t0.pip_version, t0.glibc_version FROM (SELECT
  COUNT(*) AS num_downloads,
  details.installer.version AS pip_version,
  details.distro.libc.version AS glibc_version,
  details.cpu
FROM `the-psf.pypi.downloads*`
WHERE
  details.installer.name = 'pip' AND
  details.system.name = 'Linux' AND
  details.distro.libc.lib = 'glibc' AND 
  _TABLE_SUFFIX
    BETWEEN FORMAT_DATE(
      '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 28 DAY))
    AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
GROUP BY details.installer.version, details.distro.libc.version, details.cpu
ORDER BY num_downloads DESC) AS t0 LIMIT 16000; 

When looking at the data for all x86 cpus, this gives:

When looking only at 32-bit x86 (which might lessen the CI bias a bit):

[henryiii]

I didn’t mean 99% of users can only get manylinux1, I meant that 99% of the times that manylinux1 is selected when manylinux2010 is available, it’s due to pip, not glibc limitations (the stat is easily computable from your stats above). I know there are a lot of (non-CI) Ubuntu 18.04 users because build broke under Ubuntu 18.04’s setup and we had a stream of reports. Trying to upgrade based on Python version assuming the embedded version was a minimum (as seemly indicated in the manylinux readme) didn’t fix the problem because python3-pip and python3-venv are not connected to the main Python version. That’s also how I found out about the python3.8 package and the fact it’s still tied to pip 9.

My point is this is a huge number of users on an OS with three years left to go, many of whom have no idea about virtual environments or other such things, and they will all suddenly be compiling when pip installing if a project drops manylinux1. I expect that’s why manylinux1 is still used by 75% of the cibuildwheel projects even though the default is manylinux2010.

manylinux1 does have to die, and the best way is for everyone to drop it at the same time. But wanted to clarify it’s not only the “less than 0.1% of glibc systems” stat that matters.

[henryiii]

In your data, you can see CentOS 7 and Ubuntu 18.04 clearly listed with pip 9; there’s also a mystery 2.26 distro with pip 9.

[mayeut]

Thanks for the explanation of your stat. Indeed 99% of systems not manylinux2010 compliant systems are not compliant only because of pip.

many of whom have no idea about virtual environments or other such things, and they will all suddenly be compiling when pip installing if a project drops manylinux1

Again, projects do not have to drop manylinux1 and would still be able to use the last. The image just won't evolve as there won't be use for it to evolve unless some CPython 3.10 appears with pip <= 19.x

Yes that means on packager side, there would be a need to split the build depending on CPython version being built.

I guess python3.8 was added after in ubuntu 18.04 and it shares pip with the default python 3.6 so 9.x

[mayeut]

In your data, you can see CentOS 7 and Ubuntu 18.04 clearly listed with pip 9

yes

there’s also a mystery 2.26 distro with pip 9.

glibc 2.26 with pip 9 is probably amazonlinux:2 (opensuse/leap:15.2 uses 10.0.1)

[henryiii]

Seeing Python version there could also be interesting.

[mayeut]

Seeing Python version there could also be interesting.

I dropped the patch version on everything not to get too many rows.

https://docs.google.com/spreadsheets/d/1ZV-xf-2qXkPUdPp8D8i4qRq6Jo1GSNlJQBGuLyVkiGQ/edit?usp=sharing

SELECT t0.cpu, t0.num_downloads, t0.python_version, t0.pip_version, t0.glibc_version FROM (SELECT
  COUNT(*) AS num_downloads,
  REGEXP_EXTRACT(details.python, r"^([^\.]+\.[^\.]+)") as python_version,
  REGEXP_EXTRACT(details.installer.version, r"^([^\.]+\.[^\.]+)") AS pip_version,
  REGEXP_EXTRACT(details.distro.libc.version, r"^([^\.]+\.[^\.]+)") AS glibc_version,
  details.cpu
FROM `the-psf.pypi.downloads*`
WHERE
  details.installer.name = 'pip' AND
  details.system.name = 'Linux' AND
  details.distro.libc.lib = 'glibc'
 AND 
  _TABLE_SUFFIX
    BETWEEN FORMAT_DATE(
      '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 28 DAY))
    AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
GROUP BY pip_version, python_version, glibc_version, details.cpu
ORDER BY num_downloads DESC) AS t0 LIMIT 16000; 

[henryiii]

Didn't polish off the spacing and clean the really small segments, but here's a basic pip & manylinux version pie chart for each Python version. Stuck 2.6 in to make the grid nicer. :)

from hist import Hist
import pandas as pd
import matplotlib.pyplot as plt

data = pd.read_csv(
    "results-20210227-133657 - results-20210227-133657.csv",
    usecols=("cpu", "num_downloads", "python_version", "pip_version", "glibc_version", "policy"),
    converters={
        "python_version": str,
        "pip_version": lambda x: int(x.split(".")[0]),
        "glibc_version": lambda x: int(float(x.split("-")[0]) % 1 * 100),
    },
)

h = Hist.from_columns(
    data,
    ("cpu", "python_version", "pip_version", "policy"),
    weight="num_downloads",
)

fig, axs = plt.subplots(2, 3, figsize=(12, 8))
for i, py in enumerate(["2.6", "2.7", "3.6", "3.7", "3.8", "3.9"]):
    ax = axs.flatten()[i]
    ph = h.project("python_version", "pip_version")[py, :]
    ph.plot_pie(ax=ax, normalize=True, autopct='%1.0f%%', pctdistance=.8)
    ax.set_title(f"Python {py} {int(ph.sum()) // 1000000:,} M")

plt.tight_layout()
plt.show()

fig, axs = plt.subplots(2, 3, figsize=(12, 8))
for i, py in enumerate(["2.6", "2.7", "3.6", "3.7", "3.8", "3.9"]):
    ax = axs.flatten()[i]
    ph = h.project("python_version", "policy")[py, :]
    ph.plot_pie(ax=ax, normalize=True, autopct='%1.0f%%', pctdistance=.8)
    ax.set_title(f"Python {py} {int(ph.sum()) // 1000000:,} M")

plt.show()

Quick observation; Pip 9 is frighteningly large for 3.7 (6%) and even for Python 3.8 (3%). But even then, nothing like 3.6 (17%), so dropping Python 3.6 might be good point for most projects to drop manylinux1.

[mayeut]

Thanks @henryiii for those.
The end of support date planned is after CPython 3.6 EOL.
The plan might be revisited depending on the evolution of statistics (Edit: added those to https://mayeut.github.io/manylinux-timeline/)

c.f. also https://discuss.python.org/t/blog-post-about-manylinux-and-the-future-of-manylinux1/5734

My point is this is a huge number of users on an OS with three years left to go, many of whom have no idea about virtual environments or other such things, and they will all suddenly be compiling when pip installing if a project drops manylinux1.

I agree with you however, I think the can be guided into learning those and that would be beneficial for everybody.
Projects dropping manylinux1 should help their user understand the issue and that's what at least cryptography and PyArrow are doing for exemple by updating the docs, adding a message when building from source, communicating by other means...

Let's see how it evolve in the coming months.

[jschueller]

for those who still want to use manylinux1 a bit more, its possible to add python 3.10 on top of the deprecated image using pypa scripts:
https://github.com/openturns/docker-images/blob/master/manylinux1_x86_64/Dockerfile
or just pull https://hub.docker.com/r/openturns/manylinux1_x86_64

[lordmauve]

Talking in the Pygame Discord, I came up with an idea that may help projects decommission manylinux1.

The problem that Pygame (and projects that depend on it, like mine) has faced with wheels is that when wheels are unavailable pip falls back to trying to build an sdist. For Pygame this fails on Linux much more than it succeeds: you need a ton of dependencies. And if you have some but not all of the dependencies you can succeed in building a Pygame that is missing some capabilities (reduced set of sound and image file formats). Desupporting wheels for any platform/interpreter means that users will file issues saying that the package doesn't install on their system.

I suggested filling slots we do not support in the compatibility matrix of wheels with deliberately bad wheels that refuse to install, but with a message that suggests which direction to travel to get a supported wheel (upgrade pip, upgrade Python, etc.)

AFAIK wheels cannot themselves crash pip with an error so I suggested having empty wheels that have a dependency on an sdist-only package that contains the message (I made one).

[henryiii]

Here's the October 2021 data:

Based on data https://github.com/mayeut/manylinux-timeline, code at https://gist.github.com/4f0715e64df7cf267d2d2490b5facee9.

[cbrnr]

Quick question regarding which manylinux package a project should provide by default. Apart from missing Python 3.10 support, would it be sufficient to only provide a manylinux1 wheel? And if Python 3.10 support is required, then would it be OK to just provide a manylinux2010 wheel? What are the reasons to provide newer wheels such as manylinux2014? I noticed that in one of my projects we build a manylinux2014 wheel, but at the end a manylinux1 wheel is automatically added (which is identical to the 2014 wheel except for a different version tag).

[henryiii]

You can provide the lowest one, it will have the best compatibility. If you can build it with a newer image, that's even better - since the manylinux1 images are going to be discontinued soon. The final output depends on what glibc features are used. It doesn't make sense to try to build one older than your dependencies, though - so if you require NumPy, then why make a manylinux2010 Python 3.10 wheel? Anyone pre-2014 would have to build NumPy from SDist just to use your wheel before manylinux2014.

[cbrnr]

You can provide the lowest one, it will have the best compatibility.

OK, this makes sense.

If you can build it with a newer image, that's even better - since the manylinux1 images are going to be discontinued soon.

I'll use the lowest supported one then.

The final output depends on what glibc features are used. It doesn't make sense to try to build one older than your dependencies, though - so if you require NumPy, then why make a manylinux2010 Python 3.10 wheel? Anyone pre-2014 would have to build NumPy from SDist just to use your wheel before manylinux2014.

How do I determine which manylinux to use from my dependencies?

For example, sleepecg requires NumPy, and I'm building a manylinux2014 wheel. However, this workflow produces three additional manylinux1 wheels that seem to be copies of the 2014 ones. I have no idea what's up with those automatically generated manylinux1 wheels.

[mayeut]

@cbrnr,

I have no idea what's up with those automatically generated manylinux1 wheels.

This is the behavior of auditwheel<4. Some issues/comments about this behavior are linked in pypa/auditwheel#281 (comment)

The behavior changed in auditwheel>=4 in pypa/auditwheel#289 (1 wheel but still would mention manylinux1 compatibility in you case).

Given the project in the example does not target python 2.7 or 3.5, I'd highly recommend updating cibuildwheel to the latest version c.f. also https://cibuildwheel.readthedocs.io/en/stable/faq/#automatic-updates in order to get newer manylinux images by default, support 3.10, configuration in pyproject.toml, ...

How do I determine which manylinux to use from my dependencies?

To my knowledge, if you want to match the ones used for your dependencies, the easiest is to check manually what's been deployed in PyPI.

If you use cibuildwheel 2.2+, given wheels provided by numpy based on versions in https://pypi.org/project/oldest-supported-numpy, the pyproject.toml would look like (only focusing on x86_64 linux):

[tool.cibuildwheel]
# default
# cp3.10+, NumPy has manylinux2014 wheels or more (can't predict the future)
# use manylinux2014 image
manylinux-x86_64-image = "manylinux2014"
# do not mark wheel as compatible with previous policies (if any) with --only-plat
repair-wheel-command = "auditwheel repair --only-plat -w {dest_dir} {wheel}"

# override conf for older pythons
[[tool.cibuildwheel.overrides]]
select = "cp3{7,8,9}-*"
# cp3.{7,8,9}, NumPy has manylinux1 wheels
# still use manylinux2014 image given it builds a manylinux1 compatible wheel for this specific project
manylinux-x86_64-image = "manylinux2014"
# enforce manylinux1 is produced
repair-wheel-command = "auditwheel repair --plat manylinux1_x86_64 -w {dest_dir} {wheel}"

[cbrnr]

Thanks @mayeut, upgrading to the latest version solved the issue!

[mayeut]

The active support for manylinux1 is now over.
Given the time supporting the image has dropped considerably with update automation, I don't see a reason why actually stopping those updates. So, as long as there's no need for active support, those updates will continue to happen.
Once they start breaking, the latest commit will be tagged and the branch removed.

[dralley]

@mayeut Should this be closed now?

[henryiii]

manylinux1 is still receiving updates last I checked (2-17-2024).

[mayeut]

The update workflow started to fail.
A tag has been created from the last commit of the manylinux1 branch and the branch has been deleted.