packaging.tags.compatible_tags() includes cpxx-none-any tags if interpreter is provided

[tom-bowles]

Pyxll is published with a cp39-none-any wheel, which pip installs successfully with cpython 3.9 on windows. packaging.tags.sys_tags() doesn't include that tag. (The practical impact of which, for me, is that I can't install pyxll using poetry, which uses packaging.tags.sys_tags() to check wheel compatibility).

Pip has TargetPython.get_tags(), which gives the following for me

cp39-cp39-win_amd64
cp39-abi3-win_amd64
cp39-none-win_amd64
cp38-abi3-win_amd64
cp37-abi3-win_amd64
cp36-abi3-win_amd64
cp35-abi3-win_amd64
cp34-abi3-win_amd64
cp33-abi3-win_amd64
cp32-abi3-win_amd64
py39-none-win_amd64
py3-none-win_amd64
py38-none-win_amd64
py37-none-win_amd64
py36-none-win_amd64
py35-none-win_amd64
py34-none-win_amd64
py33-none-win_amd64
py32-none-win_amd64
py31-none-win_amd64
py30-none-win_amd64
cp39-none-any
py39-none-any
py3-none-any
py38-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any

Whereas packaging.tags.sys_tags() gives

cp39-cp39-win_amd64
cp39-abi3-win_amd64
cp39-none-win_amd64
cp38-abi3-win_amd64
cp37-abi3-win_amd64
cp36-abi3-win_amd64
cp35-abi3-win_amd64
cp34-abi3-win_amd64
cp33-abi3-win_amd64
cp32-abi3-win_amd64
py39-none-win_amd64
py3-none-win_amd64
py38-none-win_amd64
py37-none-win_amd64
py36-none-win_amd64
py35-none-win_amd64
py34-none-win_amd64
py33-none-win_amd64
py32-none-win_amd64
py31-none-win_amd64
py30-none-win_amd64
py39-none-any
py3-none-any
py38-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any

The only difference is the absence of cp39-none-any. Should those be the same? Pip makes use of packaging.tags, so should it use sys_tags() or is there a reason for the different implementation?

[brettcannon]

Duplicate of #311 . Please read the full issue if you would like to consider submitting a PR to change the semantics.

[pradyunsg]

Pip has TargetPython.get_tags()

What version of pip are you using? For over two years now, pip has been using packaging.tags for determining platform tags.

See https://pip.pypa.io/en/stable/news/#v20-0

[tom-bowles]

@pradyunsg - I'm using the latest pip, 22.0.3. It makes use of a vendored packaging.tags but it doesn't call sys_tags(). It has its own function get_supported() in https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/compatibility_tags.py that calls cpython_tags(), generic_tags() and compatible_tags() from packaging.tags, but does something a bit more complex with them than what sys_tags() does.

I'll look into what a PR would take tomorrow, but my question remains about why this apparent disparity exists between what pip will install and what packaging.tags.sys_tags() returns. This manifests for me as pyxll installing fine with pip but not with poetry. Maybe that's a bug in poetry (i.e. maybe it should be doing what pip does rather than calling sys_tags()) but to my untrained eye it looks like an inconsistency between pip and packaging. Either that or (more likely) I've misunderstood something...

[tom-bowles]

Would a PR that makes sys_tags() do what get_supported() does in pip be of any value?

[tom-bowles]

Actually, from a quick scan I can't immediately see why get_supported() (with default parameters) and sys_tags() would return different results. They do though, as I pasted above. It's late here. I'll poke around a bit more tomorrow.

[brettcannon]

Maybe that's a bug in poetry (i.e. maybe it should be doing what pip does rather than calling sys_tags()) but to my untrained eye it looks like an inconsistency between pip and packaging.

It's an inconsistency and not strictly a bug with anyone. Wheel tags are only standardized so far as their format and some basic details. But support for e.g. cp39-none-any isn't specified or required from any standard (what is here is based on a lot of research of what is used on PyPI and what I could make sense when I centralized the code for handling wheel tags here).

As mentioned in the issue I linked to, we can update packaging.tags, but someone has to write the PR and justify the design decisions they made for where to insert something like cp39-none-any.

I would also question why pyxll has such wheels? What are they doing which makes them reliant on CPython but not on the ABI? Unfortunately they don't have an issue tracker we can ask on.

[pradyunsg]

If you can post the output of pip debug -v in a GitHub Gist and share that here, that’d be appreciated.

[tom-bowles]

@pradyunsg https://gist.github.com/tom-bowles/7c0bcc830c8f3c31b6f9f15969c2440f

[pradyunsg]

The issue here is that pip always calls packaging.tags.compatible_tags with an interpreter. That means, after #466, pip started accepting cpXX wheels, since that was being generated by compatible tags. The change needed to bring consistency here would be a purely pip-side change, dropping the interpreter argument when running without a custom --implementation.

[pradyunsg]

https://github.com/pypa/pip/blob/d35bedc5cbb9f19217f6a42a68ab3c7f010fe694/src/pip/_internal/utils/compatibility_tags.py#L159

[pradyunsg]

Would a PR that makes sys_tags() do what get_supported() does in pip be of any value?

pip has that logic because we’ve historically had a —platform and flags like that, in the CLI. I don’t think that logic is particularly useful for other clients, who don’t need that sort of overrides.

[tom-bowles]

The source of the difference is that when pip calls compatible_tags() it always passes the interpreter, so for cp39 it calls compatible_tags(None, "cp39", None), whereas sys_tags() only passes in the interpreter name if it's pypy.

Would a PR that makes sys_tags() always pass the interpreter in, with the justification for including cp39-none-any and its position in the order being "that's what pip does and at least one popular package relies on it" be desirable? I get that tags aren't fully standardised, but isn't consistency between packaging and pip as to which tags are compatible with a given system useful in its own right? Alternatively, should pip not pass this in (and break pyxll)?

I don't have any insight into pyxll as it's closed source, unfortunately. It looks like they've had a single cpxx-none-any wheel for years. My best guess is they build a single wheel that requires cpython but works on both 32- and 64-bit Windows. If so, maybe that should be two wheels. It doesn't work on non-Windows platforms.

[tom-bowles]

Ah, missed your reply @pradyunsg - hadn't refreshed while I was writing mine. We found the same thing.

So your position is that this should be raised as an issue with pip, and that they should drop cp39-none-any, breaking pyxll? If so I'll raise an issue. Might be nice to tell the pyxll folks about it...

[tom-bowles]

Hmm, are you sure pip only started accepting these wheels with #466? That was in october 2021, but pypi has pyxll packages going back to 2020 that have (only) cpxx-none-any wheels: https://pypi.org/project/pyxll/4.5.1/#files

[tom-bowles]

Pip 20.2, from before that change, does appear able to install pyxll

D:\venvs>tags_oldpip\Scripts\pip.exe install pyxll
Collecting pyxll
  Using cached pyxll-5.4.0-cp39-none-any.whl (54 kB)
Installing collected packages: pyxll
Successfully installed pyxll-5.4.0
WARNING: You are using pip version 20.2; however, version 22.0.3 is available.
You should consider upgrading via the 'D:\venvs\tags_oldpip\Scripts\python.exe -m pip install --upgrade pip' command.

[pradyunsg]

What's the output of pip debug (or get_compatible_tags, if the debug command wasn't added) then?

[pradyunsg]

Well, let's move the rest of the discussion to pip's issue tracker. @tom-bowles Would you be willing to file a bug report against pip for this?

[tom-bowles]

@pradyunsg Yes - will do later today.