drop 32-bit support?

[njsmith]

@ionelmc was asking in #pypa whether anyone actually uses 32-bit linux, so I ran some queries, and it looks like currently the answer is:

• numpy: 99.5% of manylinux downloads are 64-bit
• lxml: 99.8% of manylinux downloads are 64-bit

So no, almost no-one uses 32-bit linux. I figured I'd open an issue to write down this observation, and raise the question of whether (or when) we should stop bothering to make 32-bit manylinux images.

---

Noting for future reference:

SELECT
  COUNT(*) AS downloads,
  REGEXP_EXTRACT(file.filename, r'.*-(manylinux[^-]+)') AS manylinux
FROM
  TABLE_DATE_RANGE([the-psf:pypi.downloads], TIMESTAMP("20171120"), CURRENT_TIMESTAMP())
WHERE
  file.project = 'lxml'
GROUP BY
  manylinux
ORDER BY
  downloads DESC
LIMIT
  1000

[ionelmc]

To give some context: I was just having some issues with the 32bit image while making my customized image, I didn't notice that I need to run yum install with linux32 when building it.

[rdb]

Not every project has the same audience. For our project, 45% of the manylinux downloads are 32-bit. There's no chance we can afford to drop 32-bit support.

[njsmith]

45%? Whoa. What project is that?

…

On Nov 24, 2017 4:15 AM, "rdb" ***@***.***> wrote: Not every project has the same audience. For our project, 45% of the manylinux downloads are 32-bit. There's no chance we can afford to drop 32-bit support. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#128 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAlOaPfDwEk0w3i4Q2wIplYSjKINKv9bks5s5rNfgaJpZM4Qpo5X> .

[rdb]

Hmm, I just noticed that your query includes bandersnatch downloads. Excluding those, I get the more reasonable number of 16%, which sounds far less surprising, although still enough reason to keep supporting 32-bit.
The project is panda3d, for what it's worth.

[njsmith]

I pulled some more stats on this, since the discussion has come up again int he context of manylinux2 (where we don't necessarily have good 32-bit compilers). Here's a table showing 32- and 64-bit manylinux downloads for the last ~5 weeks, split up by project, sorted by the 32-bit / 64-bit ratio:

https://docs.google.com/spreadsheets/d/1Oq2x01Fh3c3ZOEaJkBZk_c9yGu3vWNXISTV6BA8_2qA/edit?usp=sharing [Edit: see below for a more accurate spreadsheet]

Query:

WITH
  downloadinfo AS (
  SELECT
    COUNT(*) AS downloads,
    file.project as project,
    REGEXP_EXTRACT(file.filename, r'.*-manylinux1_(.*)\.whl') AS manylinux_variant
  FROM
    `the-psf.pypi.downloads*`
  WHERE
    _TABLE_SUFFIX >= '20180101'
    AND REGEXP_CONTAINS(file.filename,
      r'.*-manylinux1')
  GROUP BY
    manylinux_variant,
    file.project)
SELECT
  m32.downloads / m64.downloads AS ratio,
  m32.downloads as downloads32,
  m64.downloads as downloads64,
  m32.project
FROM
  downloadinfo AS m32,
  downloadinfo AS m64
WHERE
  m32.project = m64.project
  AND m32.manylinux_variant = 'i686'
  AND m64.manylinux_variant = 'x86_64'
ORDER BY
  ratio DESC

[njsmith]

Hmm, this also includes mirror downloads – I'm not sure off the top of my head how we restrict to only pip...

[njsmith]

Okay, that was easy :-). I added a details.installer.name = 'pip' to the WHERE clause, and the results look very different. Here's the updated spreadsheet:

https://docs.google.com/spreadsheets/d/1kPU-O9EoiNer5aYFXmRwu-g2bv8Yih332kB9xtkiR5o/edit

[markrwilliams]

It turns out that devtools-7 supports compiling 32-bit binaries on 64-bit architectures. I haven't been able to cross compile a 32-bit Python yet because a) I didn't build a patched 32-bit glibc RPM and b) even a CentOS 6.9 in a compliant, vsyscall-full virtual machine doesn't seem to provide a 32-bit binutils, so I can't get a readelf to satisfy CPython's cross compilation chain.

However, it seems like this ought to be possible, in which case we can just have a 64-bit image that can build both 32-bit and 64-bit wheels...

[markrwilliams]

It turns out that devtoolset-7 doesn't ship a complete 32-bit cross compilation toolchain; it only includes a subset of 32 bit libraries and doesn't include supporting 32-bit programs like pkgconfig.

Building 32-bit Pythons on the 64-bit doesn't appear to be possible without non-trivial effort. Maybe somebody else can find an easier way to do it.

[jcfr]

Building 32-bit Pythons on the 64-bit doesn't appear to be possible without non-trivial effort.

That should be more straightforward using https://github.com/python-cmake-buildsystem/python-cmake-buildsystem

You can even compile CPython to a broad set of target.

And if cross-compilation beyond 32-bit is of interest, these collection of docker images may be relevant to you https://github.com/dockcross/dockcross