Do basic syntax checks on the RECORD file when installing a wheel

[SpecLad]

For now, only check requirements specified in the original RECORD specification (https://packaging.python.org/en/latest/specifications/recording-installed-packages/#the-record-file), without the added constraints from the wheel specification.

Some of the warnings ("invalid digest size" and "duplicate line") should really be errors, but I was able to find some popular projects that have these problems (any projects built with sphinx-theme-builder for the former, zeroconf for the latter). To avoid breaking stuff, keep these as warnings for now.

Fixes #6198
Fixes #5913
Un-fixes #6165 (the underlying issue in wheel has been fixed 3 years ago, so the workaround should no longer be needed)

[SpecLad]

To minimize potential breakage, I downloaded one wheel for each of the most popular 5000 PyPI packages (4399 wheels in total, because some packages had no wheels), and attempted to parse the RECORD file in each of them using the parse_record function from this PR. There were no hard errors, and the warnings were as follows:

• catboost has unnecessary padding in the digest (see File hashes are encoded incorrectly in the wheels' RECORD files catboost/catboost#2282).
  • Fixed in catboost 1.2 (2023-05-01)
• pantsbuild.pants has both unnecessary padding and the wrong alphabet (see Invalid entry in the RECORD file in pants's wheels pantsbuild/pants#18111).
  • Fixed in pants 2.15.0 (2023-02-25).
• furo, pydata-sphinx-theme, sphinx-book-theme have unexpected digest lengths. All of these are due to Hashes generated in the wrong format pradyunsg/sphinx-theme-builder#13. This is the reason why I made this a warning and not an error.
  • Fixed in furo 2023.03.27 (2023-03-27).
  • Fixed in pydata-sphinx-theme 0.13.2 (2023-03-29).
  • Fixed in sphinx-book-theme 1.0.1 (2023-03-31).
  • Underlying issue fixed in sphinx-theme-builder 0.2.0b2 (2023-03-27).
• zeroconf has identical duplicate entries (see Windows wheels have duplicate RECORD entries python-zeroconf/python-zeroconf#1125). This is why I made this a warning (although it's a pretty harmless condition, so it might not need to be an error at all). Caused by Duplicate lines in the wheel RECORD file when a native extension is built python-poetry/poetry#7505.
  • Fixed in zeroconf 0.47.4 (2023-03-20).
  • Underlying issue fixed in poetry 1.4.1 / poetry-core 1.5.2 (2023-03-19).

[pradyunsg]

Hehe, I'm pretty sure some of this is my fault. ^>^

I'll take a closer look at this sometime next week, but I wanna explicitly say: Thank you for doing this and filing this PR!

[uranusjr]

Is there similar logic (the record model part) in installer? I wonder if this is a good chance to vendor the lib in.

[pradyunsg]

Is there similar logic (the record model part) in installer?

Yes.

I wonder if this is a good chance to vendor the lib in.

We should -- although that'd be blocked on pypa/installer#153 and pypa/installer#156.

[uranusjr]

I’m mostly thinking we can probably reuse stuff in installer that don’t exist in pip (hash-checking etc), pip logic going into installer doesn’t need to block that.

[SpecLad]

I looked at the RECORD checking logic in installer, and it's a lot more lax than what I have implemented here. It does not reject or warn about any of these test cases:

'a.py,foobar=abcde,'
'a.py,sha256=~!@#$%,'
'a.py,sha256=abcde,'
'a.py,,-1'
'a.py,sha256=4OHi4-Tl5ufo6err7O3u7_Dx8vP09fb3-Pn6-_z9_v8=,'
'a.py,sha256=4OHi4+Tl5ufo6err7O3u7/Dx8vP09fb3+Pn6+/z9/v8,'
'a.py,sha256=abcdef,'
'a.py,,123\na.py,,456'
'a.py,,123\na.py,,123'

[pradyunsg]

Are you calling validate_record (it's on the main branch) or record.validate(data=...)?

[SpecLad]

Are you calling validate_record (it's on the main branch) or record.validate(data=...)?

No, because my patch only checks the RECORD file itself, while validate_record checks the data in the wheel, so it wouldn't be a fair comparison.

Now, most of the aforementioned test cases would fail if you were to validate them against wheel contents, but I think it's a good idea to check the syntax before trying to validate the contents, since in that way you can fail faster and give better error messages.

[SpecLad]

So how do we move forward from here? Is there anything you'd like me to do?