-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
twine check
should guard against python_requires
conflicting with the wheel tags
#739
Comments
I'd like to voice my -1 -- universal just means "pure python and works on any version" and presumably will hold the same meaning when C-incompatible python 4.x rolls around. For the same reason that |
@asottile that's what I was thinking until @gaborbernat pointed out that it's not exactly true. It is currently |
It is true, or at at least what the documented point of the metadata isn't conflicting, |
This is hinting pip to attempt downloading the wheel under Python 2. @pradyunsg seems to think that this could cause problems with the old resolver (combined with |
Also, I think that it's not the config for the tool producing wheels that should be linted but the resulting metadata in the wheel. |
To achieve that intent the wheel tag this should map would need to be just |
It would only cause problems for pip versions that don't understand what I'd rather see is if python2 is truly dead then |
I like this idea! let's push to make that happen |
I mean, either universal should start meaning Can we get some opinion from at least:
Thanks! |
I think pypa/wheel#396 is on the right track, except that I would rather abort the operation if such a discrepancy is detected because otherwise this could go unnoticed by the users. |
As packaging BDFL, the wheel spec doesn't mention "universal" at all, so I'd consider that an implementation detail of the wheel package. From pip's POV, we consume the tags, not "universal" directly. I'd be happy if pip were to add a check that the tags and |
Yep, |
This will break a lot of CIs, rightfully though. |
As a precaution we could check some of the most popular packages to make sure they have their settings right. I could probably write a script for this. |
What information are you looking out for here? I mean, almost all of @asottile packages will fail essentially, and I'm sure even I have a few lying around (though I don't mind if the CI starts failing). You could in theory get https://hugovk.github.io/top-pypi-packages/, and download each wheel, check if the wheel tag is conflicting with python-reqires. But say 60% fail, what would you do then? |
Pre-built wheels are not a problem, since I would lessen the impact by submitting PRs to projects that have misconfigured their packaging setup. |
I should also say that right now, there's nothing in any spec that disallows publishing a wheel with (for example) a Python tag of Wheels where the tags and requires-python describe different but overlapping sets of versions is a much harder problem, though (for a start, there's no set of tags that's equivalent to an open-ended requires-python like And that's about all that I think needs to be said from a standards perspective. |
That would mean you'd need a brand new pip version (and brand new packaging version), which would never work on Python 2 since that's been dropped by both packages. Not against a "py" tag, but it's come too late for this. If it was implemented, it probably would need a new setting;
It's very clearly documented that this means Python 2 + Python 3, from the first release note (wheel 0.10, "Define a new setup.cfg section [wheel]. universal=1 will apply the py2.py3-none-any tag for pure python wheels."), two years before the change linked above (which also still mentions
He can change one package (
I did. Didn't get a happy response. pre-commit/pre-commit#1832. "general word of advice: don't touch infrastructure or tooling for a project you don't maintain unless prompted by an issue asking explicitly to do so" and "the crusade to remove this flag is not correct and I closed this change because it is not welcome here". |
technically that could be installable since |
All older (not c) python3 tags install:
|
I would also like to deprecate the |
There's an idea to set the lowest Python supported as the tag via Requires-Python here: pypa/wheel#336 , by the way.
I don't see a strong point for producing I'm not against getting more packages using the |
Drat, that'll teach me to try to be clever and invent a "better" example (I was originally going with py2 rather than py37, which would also be uninstallable, I believe - although now I'm nervous even about that 🙂) And actually, there's no specification that mandates what compatibility tags a given distribution must say it supports, so from a purely theoretical standpoint, it's even more complex than that... Let's just go with "it's possible to create wheels that are uninstallable" 😉 |
@asottile Do you still object to the removal of |
correct |
Why? When it's clearly wrong to have this in a py3 only project? |
I don't see why a user should be expected to make or understand some random For Python 4, we have Requires-Python and the knowledge that people are going to want to write code that runs in both Python 3 and 4 at the same time. From the 3.0 release notes:
|
The |
|
Yes, and setuptools is likely to adopt the bdist_wheel command into itself in the future. |
Do you have any documentation that confirms that's how "universal" was intended? I don't recall it being like that - it was originally intended specifically to indicate that a project was "universal" as in "could be run on any Python installation" and as such translated to
Um, It's certainly true that things have changed, though, and the reality nowadays is that "universal" probably isn't a useful designation for real projects, and certainly there are unlikely to be many projects that still work on "all versions of Python". But I'd suggest that we stick to discussion what's useful or worth supporting now, and not try to read too much into the history (or people's recollections of it). |
Well, as you pointed out, it scopes the wheels to Python 2 and 3 only which is not really universal. But realistically, any project has a minimum supported Python version which they should indicate with |
It was when we designed the flag. We didn't think sufficiently far ahead, but the intention was "everything".
But what I support and what it works on may be two different things. And I may prefer not to block people on older systems from using my code, just because I won't support them. I feel like there's an assumption underlying this that all projects are somehow required to hit a certain bar of "professionalism" here, and to be blunt, that just isn't the case.
How do you plan on fixing all the existing projects that don't do that? I really don't see any benefit in a rule like this that only applies to new uploads. To give a concrete example, pip's resolver, if faced with the latest version of a package saying that it doesn't support my Python version, will quite happily go back through all the older versions (potentially hundreds of them) looking for one that supports my installation. And ultimately, what it will find won't necessarily be one that supports my version, it's just as likely to find something that's old enough that it was uploaded before the rule requiring We need to remember that PyPI don't have the resource (or, as far as I'm aware, the desire) to be a curated repository of packages, so there's a lot of stuff on there that's of questionable value or is simply abandoned. Imposing new rules without considering how we apply them to existing projects is just going to result in rules that have so many exceptions that people of necessity will ignore them. Also, what's to stop projects just saying Anyway, this is going a long way off topic. I'm going to stop responding to this issue unless explicitly pinged, as I don't think this subthread is going anywhere useful. I suggest we get back to the original question, of whether |
The guess about intentions are based on the problem it was trying to solve (memory, the current docs, and old release notes); I just downloaded wheel 0.9.7 and setuptools 0.7.2, and ran bdist_wheel on a simple project. The produced wheel was Note that @asottile's projects are Python 3.6+, and have
For 1, since setuptools, pip, packaging, and most other Python 3+ pure Python libraries have dropped it, I don't think that's true anymore. It's also very clearly documented everywhere it shows up in wheel and packaging.python.org's guides that it means Python 2 + 3, and pure Python wheels do not need universal=1. For Python 4, I don't think we could reuse For macOS, "Universal" meant PowerPC+Intel. "Fat" meant Intel32+Intel64. And now "Universal2" means Intel+AppleSilicon. I think we could avoid another special flag to bdist_wheel by using Requires-Python, but say we didn't; it probably would have to be |
Note, I do know a few projects that "secretly" supported a version but set the requires one higher. It's not really that helpful of a practice, I think, but it is a rare case where they want to have a |
How about this concrete proposal:
This would mostly be implemented through pypa/wheel#336. This has a small consequences to the point of this thread:
Edit: added missing behavior when both are given. |
imo attempting to match a wheel tag to python_requires is kind of a lost cause, what would you do for |
Three possibilities. I'm also personally fine with Similarly, CERN still has running Scientific Linux 6 machines that have pip 8, FFI. It's not truly dead; I'm very glad at least pip correctly removed py2 when updating to 21. 7% of Python 2 users have pip < 9. |
These sorts of discussions really should be made on the wheel issue, not here. |
IMO twine printing warning when there's some sort of obvious mismatch like the valid-but-uninstallable wheel example that @pfmoore gave above, or a wheel that's tagged for Python 2 but can't be installed there due to python_requires is a good idea. It's probably a good idea to check if the last version of pip that supported Python 2 does not work with py, because if it does, then there's not much discussion to be had here. :)
FWIW, I think this issue's OP isn't to suggest that we should try to have them match precisely though. Rather, it's about being tagged for a Python version that the package can't be installed on. And this whole thing about inferring tags should likely move to the wheel issue tracker. FWIW, I'd like to put the We can add |
👋 (relatively new) Twine maintainer here. It's been interesting to watch this discussion, but I've only skimmed it. Can somebody write (or point to) a concise proposal for an enhancement to Twine that has (or can get) the support of the folks involved? |
@bhrutledge I'd say there does not exist a proposal which everyone who has been involved in this discussion would be supportive of. I'll try to summarize:
I think the best we could do is have a check that can never be considered an error because the confidence in that check would be so low that is also only run when provided a flag that warns if you have All of the above is to say: There's almost no alignment on what any of this means (the current maintainers of various pypa projects seem to have different interpretations of things compared to the PEPs when they were written) and tools have grown behaviour based on what was available. Twine taking any action here seems like it would result in endless churn and heartburn for us. Finally as a total aside, it seems all members of PyPA can tag issues on this project despite not being maintainers and thus can give an issue a sense of being accepted as a feature/enhancement. That's kind of crappy and doesn't let the actual maintainers guard their "no" appropriately. |
Oh, sorry about that. I believe that being in the "Gardeners" team grants this right. That team is meant to give other PyPA folks the ability to do some harmless housekeeping but it seems like not everybody's aware of this.
FWIW I think I'm okay with a "no" if that's what people here think is best. I don't have a strong opinion and only raised this issue because I overlooked this in my projects and didn't even notice until Bernát told me.
This solution sounds compelling. I'd definitely +1 it if that's something others can agree on. It is good to have an optionally enabled warning that could become an error with an explicit |
To clarify this point slightly, the wheel tags say which Python version(s) a wheel is compatible with, so pip ignores any wheel that has tags saying it isn't compatible with the target system. This is not really a "hint", it's mandated behaviour - you must not install a wheel whose tags declare that it's incompatible. The only way we use tags to choose which wheel to install, is in the sense that we choose the "most compatible" wheel where multiple otherwise-equivalent wheels are available. "Universal" (whatever that means in practice) will likely be low on that scale, by nature of what it means to be "universal". Again, that's mandated by the standard so all installers will do this, not just pip.
If my comments have had that effect, I'm sorry. I have no authority here and my comments are only intended to be informative. I only got involved because @gaborbernat pinged me for a view on what the standards said, and I've tried my best to confine my comments to clarifications on what the standard-mandated behaviour is (which frankly, is only peripheral, as the issue is about a check, which is not something the standards have anything to say on). FWIW, I entirely support the project maintainers having the final say, and I do not want any comments I've made here to be seen as exerting any pressure on them. |
The discussion here got a little far afield, but the focus is not on The thing that twine check could (and personally, I think should) to is make sure that the Requires-Python metadata slot and the Python tags (which are both standards controlled) do not conflict. Something with Whether |
The Issue
With the sunset of Python 2, distribution maintainers should stop producing universal wheels with (ones with
py2.py3
tag).The solution is to drop
universal = 1
or turn that into 0 (pypa/packaging.python.org#726) but it's easy to miss this bit. So this results in people adding things likepython_requires = >= 3.6
but still having thatpy2
tag in the wheels which is rather useless (and, well, incorrect).The same could happen to platform-specific wheels specifying ABI3 with tags like
abi3-py36
while havingpython_requires = >= 3.7
, for example.I propose improving the
twine check
command to detect such inconsistencies.The text was updated successfully, but these errors were encountered: