Piwheels may have more downside than upside on 64-bit Raspberry Pi OS

[pettyalex]

Hello,

I ran into a pip installation failure while trying to install hail, and it made me question why piwheels.org is included as a default pip index on 64-bit Raspberry Pi OS. Piwheels.org is extremely valuable for 32-bit Raspbian, but does not at this time build 64-bit wheels at all. For pure python noarch -none-any wheels, we can get those from pypi.org, which I believe to be more reliable than piwheels.org.

The specific failure I saw repeatedly was:

(venv) (py312) alex@rpi400:~/scratch $ pip install boto3
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting boto3
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /simple/boto3/boto3-1.34.73-py3-none-any.whl
  Using cached https://www.piwheels.org/simple/boto3/boto3-1.34.73-py3-none-any.whl (136 kB)
Collecting botocore<1.35.0,>=1.34.73 (from boto3)
  Downloading https://www.piwheels.org/simple/botocore/botocore-1.34.73-py3-none-any.whl (12.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━ 10.8/12.0 MB 105.4 kB/s eta 0:00:12
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    botocore<1.35.0,>=1.34.73 from https://www.piwheels.org/simple/botocore/botocore-1.34.73-py3-none-any.whl#sha256=7b560473efb5030aa0ec7655d296508511e25b59518878b50f05becd8d6fb5d5 (from boto3):
        Expected sha256 7b560473efb5030aa0ec7655d296508511e25b59518878b50f05becd8d6fb5d5
             Got        ec2fc578de897461e0f4dcd7e501d56aafd3403237fced5ef30766b3269d2c08

I believe that this failure is because of either an internal timeout in pip, or network issues with piwheels.org, previously documented here: piwheels/packages#32 .

If piwheels.org does not build 64-bit wheels, maybe it should not be used by default on 64-bit Raspberry Pi OS? Perhaps it could be kept as a default for 32-bit but removed from 64?

[lurch]

ping @bennuttall

[bennuttall]

This is covered in the FAQ under "I got a hash mismatch when trying to install a package from piwheels". There's a workaround there: https://www.piwheels.org/faq.html

It looks like pip was doing a retry, so it may have been struggling to download the file at the time.

In recent years, pip changed its behaviour to download from extra indexes before PyPI. See pypa/pip#6394 - It doesn't usually make any difference, but if you want to disable piwheels on your 64-bit image, just run rm /etc.pip.conf.

[XECDesign]

Sounds like piwheels should still be used for arm64 then. Thanks, Ben!

[pettyalex]

@bennuttall Does Piwheels offer ARM64 wheels? Is there any situation where piwheels will offer a compiled wheel that PyPI doesn't have on ARM64?

If it doesn't, then what's the value of Piwheels being enabled by default on 64-bit Raspberry Pi OS?

Edit: I didn't intend that to come off so harshly, and appreciate what Piwheels has done for Python on 32-bit Pis, but I do want to understand what it offers to 64-bit Pis, and question if perhaps you could get the load off your servers if you don't have wheels to offer 64-bit pis.

[bennuttall]

We don't currently build aarch64 wheels, but we intend to support for this at some point. Plus, there's a small benefit to having pure Python wheels when PyPI only has sdist.

what's the value of Piwheels being enabled by default on 64-bit Raspberry Pi OS?

Generally speaking I'd say it doesn't make a difference, and when we get around to adding aarch64 support, people would already have the configuration.

@XECDesign could choose to separate the configuration for armv7 and aarch64 pip packages, but it's not really worth the effort, and it'd have to be undone if/when we add support for aarch64.

We've served nearly 500M wheels, and only had reports of the issue you've described maybe 5 times. It could easily happen with PyPI too.

[XECDesign]

Alright, then if it generally does no harm and may provide a benefit in the future, I'll leave it as is. If a user wants to disable it, the option is there.

[clement-escolano]

We've served nearly 500M wheels, and only had reports of the issue you've described maybe 5 times. It could easily happen with PyPI too.

You can add a 6th time from my part :-)