Speculative: --only-binary by default? #9140
Comments
|
We could attempt to make the default more intelligent (or maybe just more magical). Basically have the implicit default be that if a wheel is found at all for some project, that project defaults to only allowing wheels.
…Sent from my iPhone
On Nov 16, 2020, at 12:40 PM, Paul Moore ***@***.***> wrote:
What's the problem this feature will solve?
A lot of users are reporting issues when there's no Python 3.9 binary for projects they need, and pip tries to build from source and fails with an obscure error (because the user doesn't have a compiler, or isn't set up to build the relevant packages).
Describe the solution you'd like
Pip shouldn't try to build from source if the user isn't prepared to deal with build errors. As it's not possible to know the user's level of expertise, we should err on the side of caution, and by default only allow wheels to be installed. Users who know they need to install from source and have checked that they can do so, can explicitly say so using a new --allow-source flag, which acts as an "opt-in" to source builds.
Alternative Solutions
Improve the error messages when a source build fails. This is hard, because the details of what went wrong are entirely the responsibility of the build backend.
Additional context
I don't realistically think this can be added without a lot of disruption, but given that significant numbers of projects ship wheels these days, maybe it isn't as unthinkable as it once was. I do think it's worth discussing the implications, if only as a thought experiment, and I don't know where else we could do that apart from here.
One big problem area is that we can't distinguish between "pure Python" projects that are shipped only as sdists, but which only need Python to build, and complex projects that need a compiler. So restricting to wheels only would require an explicit opt-in for some projects which currently install with no issue.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
But we can’t know what “all projects” means before deciding whether to set the flag, since dependency information is inside the sdist/wheel 🙃 |
|
@uranusjr I'm suggesting making |
|
Oops, my previous response was toward @dstufft’s “intelligent” suggestion. Sorry for the confusion. To express my thoughts in more words, I think the “only wheel unless some project needs to compile from source” would be very difficult to implement since the two parts in the logic depend on each other. I would much prefer @pfmoore’s original suggestion of having |
|
The logic isn’t hard and has nothing to do with dependency information.
Current logic is roughly:
1. Fetch a list of links from the index for project X.
2. Filter said list of links using the value of —only-binary (among other things like platform tag).
3. Return list of links for use in the dep solver.
The proposed change only slightly changes the logic in step 2 slightly, such that unless the user has explicitly configured only-binary, we will set the value of it implicitly by inspecting the entire list of links we’ve discovered for project X, and determining if there is a wheel file or not.
This is simple, and would prevent the breakage that Paul is currently seeing, projects which generally make wheels available, but that haven’t for this version of Python / OS / Whatever.
It wouldn’t change anything for projects which don’t ship wheels at all, some of which will be pure Python, some of which will be compiled code, but in any case there’s no “upgrade to Python 3.9 and suddenly start compiling code” problem for these projects since they are consistent in what they require.
The biggest issue with this that I see is in the effort of being smarter about our default to not break certain kinds of projects, we make it easier for projects to accidentally break their users. If my project historically did not upload wheels, and then I start uploading wheels with version 3.1, all previous versions suddenly stop working without opting in to some flag. This is done without any obvious change by the user (upgrading versions of pip is an obvious change, but some thing I install starting to upload wheels is not).
We could work around that problem by trying to reduce the blast radius of the implicit “wheels only” setting, by saying that we will only filter out non wheel links by default that are of the same version of a wheel we’ve found. Thus if we find an sdist for 1.0, 2.0, 3.0, and 3.1 and we find a wheel for 3.1, when we filter the list of links, we will filter it so it has the sdists for 1.0, 2.0, and 3.0 and the wheel for 3.1.
This makes it so that as soon as you upload a wheel for a given version, you’re effectively signaling that not only should a wheel version be preferable, but that the sdist should only be used if explicitly configured to by the user.
…Sent from my iPhone
On Nov 17, 2020, at 4:48 AM, Tzu-ping Chung ***@***.***> wrote:
Oops, my previous response was toward @dstufft’s “intelligent” suggestion. Sorry for the confusion.
To express my thoughts in more words, I think the “only wheel unless some project needs to compile from source” would be very difficult to implement since the two parts in the logic depend on each other. I would much prefer @pfmoore’s original suggestion of having --only-binary :all: unless the user explicitly allows source distributions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Maybe we simply make |
I like it? I think something like 98% of packages on PyPI have wheels in the latest release, so I don't think this is catastrophically bad.
IMO one of the improvements we should make here is adding a sentence like: "This failure occurred while trying to generate [a wheel / metadata] for packageName. This is not an error in pip." This also applies to the proposed approach here too -- clearer error messaging would be good. :) |
I suspect the number would be significantly lower if you count percentage of downloads instead. There are a bunch of popular pure-Python projects that don’t bother with wheels because the effect is minimal. |
I'm pretty sure that's a figure I gave you, and I found the bug in my calculation a bit later 🙁 I need to re-do the sums, but I think it's a lot lower than that, unfortunately.
The number's a lot lower without doing the sums incorrectly 🙂 Sorry about that. I don't have download information, but I'm re-doing the numbers right now, and I'll see what things look like if you factor in "uploaded a file in the last 12 months" as well. I might try getting download numbers from the BigQuery data for offline analysis. Downloads per project, per year (month?) might be sufficiently interesting, if I can work out how to get that relatively easily in a CSV format or similar. To confirm, my query has just completed. Comparing "number of projects that distribute sdists but no wheels for their latest version", vs "number of projects that distribute wheels for their latest version", the numbers are almost identical (124508 vs 124782). Looking at projects which have released at least one file in the last year, the values are 32890 and 66635. So half of all projects, 2/3 of projects active in the last year, have wheels. As I say, I think that however we did this, it would result in a lot of breakage. |
pradyunsg
added
state: needs discussion
|
It's a backwards incompatible change, so regardless it's going to break someone. The goal behind my proposal is to limit the blast radius, so that we limit the breakage, either to specific projects, or to specific versions within a project. I think there's two questions here too:
I'm not sure about the long term "right" answer. I can see an argument that we want to encourage wheels where possible.. but I also think that there are some projects that simply cannot be shipped as wheels, and maybe will never be able to be shipped as wheels. We need to figure out if going wheel only by default will end up being worth it, or if we will push too many projects out of viability. For the second one, I think having the default by to filter out sdists, for any project version that has any wheels uploaded, solves the main driver to this proposal, without breaking projects that are not shipping wheels (or used to ship wheels, but found out that was problematic). That could be useful as a stepping stone for getting to a wheel only default (for instance, we could provide warning when installing from sdist then), or it could be a reasonable end state that solves the surprising accidental sdist install, without dropping support for sdist only projects by default. |
|
I think we could do a lot better if we could somehow identify which projects are "hard" to build from source. I feel like blocking sdists that build into universal wheels is going a bit far. In the most general sense, that's basically impossible, but maybe we could add metadata somewhere (in the simple index?) to mark "pure Python" projects? I agree it's not clear what the best long term answer is. We're seeing a lot more people using Python nowadays who honestly don't want to, or know how to, deal with building stuff from source. For those people, pip downloading a sdist that needs a compiler to build is almost certainly just a source of problems. But they are also precisely the sorts of user who won't know enough to add |
|
I wonder if we can leverage PyPI in some way to encourage wheels, or to at least surface better information to highlight which projects don't ship wheels? This might be a better question for discourse? I dunno. |
|
I've got a big chunk of downloaded data from PyPI that I am querying to get a better feel for this sort of stuff. The biggest problem is the vast amount of (to be polite) "limited value projects" on there - without some form of insight, it's hard to know for sure whether it's OK to ignore a project called "0html" or "django-3-jet-zupit" - especially when it comes up in the same query as "090807040506030201testpip"... |
|
What if PyPI automatically builds the simplest pure Python wheels? There’s recent interest to detect malicious source distributions on PyPI, and the wheel it would produce as the side effect should be able to be reused. |
|
Any more thoughts here? I especially like the idea
The metadata option also seems reasonable, then the scientific python community could mark NumPy, Scipy, tensorflow, pytorch as "prefer binary by default" and save a lot of CI and cloud resources. |
|
I like the idea as well, maybe with a twist: Versions with only sdist are excluded, unless there are no wheels available at all prior to that version. Use django-grappelli as an example, this means that
|
|
For another data point here is an issue filed by a python3.5 user of cffi where they cannot build with the sdist, and changing the default would have helped them. |
|
Please edit the title binary-only -> only-binary. I always have to check |
|
FWIW, that tells me that we should add an alias for that option. |
uranusjr
changed the title
|
+1 for a solution via either package metadata or via a simple rule like " Otherwise it has the risk of becoming a
This does not seem like a good idea. Not only is it harder to understand, it also partially defeats the purpose here. If a package has a very old source-only release (e.g., from the pre-wheels era) then that will be will be found the moment there's no suitable wheel for a user. In your particular example, |
|
Makes sense. I think it's quite difficult to gauge the actual impact here, since people here all care much about Python packaging (for apparent reasons) and likely push for wheels in projects we are involved. So I feel the only way to go forward is to actually try to implement this (maybe as a There are probably still some implementation details we need to sort out. Should we go with |
|
I would propose an alternate path forward. Rather than changing the default behavior of As has been mentioned above, pip currently mixes two different things (building and installing from source and installing from pre-built binaries) and I think it is a mistake to tilt pip even more in favor of being a binary-only package manager. By adding a new CLI entry point it is possible to make what ever changes are needed to make pip behave like a binary package manager without having to worry about breaking an existing users. I think another issue here is a disagreement as to what exactly wheels are for. I have always considered (and I may be the only one to hold this position) the sdist the canonical source of truth for what the released version of the package is on pypi with the wheels are provided for the convenience of the user (the linux wheel spec is "manylinux" which suggests it is a best-effort rather than authoritative artifact!). I think making pip more-binary package-manager like by default will only re-enforce the expectation that projects will (promptly) provide a wheel for your platform / Python version / Python implementation and one not existing is a "bug". There was a discussion on the numpy mailing list about the ever expanding number of platforms that projects are expected provide wheels for becoming un-sustainable (the latest beta-release of Matplotlib has 21 wheels and we are not yet covering the full Python version/Python implementation/arch/OS matrix https://pypi.org/manage/project/matplotlib/release/3.5.0b1/). If pip is going keep going down the path of binary packaging, I think there needs to more discussions about how filling out the build matrix can be lifted from the projects to some centralized build service like the homebrew, conda-forge, and the Linux distributions do already. Separating the wheels into their own channel/management chain would also make it easier to manage things like updating version pinning on the wheels post-facto (e.g. putting an upper bound on something or banning known-bad version combinations), re-building with updated versions of non-Python dependencies (xref h5py/h5py#1942), or dealing with CVEs much easier. |
|
How can we make the abstract discussion here more concrete? I see a couple of subjects being mixed together
I apologize if I missed some of the topics here, please feel free to add to the table. The next question is who will do the work ... |
|
I’m dropping a link to the RFC proposing to disable install scripts by default for NPM, which would have roughly the same effect as making |
|
As an outsider lurker, I think making When installing from sdists, pip could print a big fat warning like or with the wording subject to reflection of course, but you get the idea. |
|
Some more thoughts. I think pip should try to be much more informative in its error messages than it is currently. Granted, building the package is under the sole responsibility of the build backend. However, 95% of problematic cases should be covered by a few simple heuristics.
Current message: Wished warning: Because if there are wheels specific to some Python versions but no cross-Python (
Current message: Wished warning:
Overall, I have to say that pip's current error messages are sometimes quite uninformative; I would recommend improving them first and seeing if the current problem of zillions of people reporting that pip doesn't work persists. Yes, it's true that many people won't read them — but many will, too. |
I don't disagree with you, and it would be nice if this could be done. But I hope you don't think that we haven't been trying to improve things here. If anyone has any good ideas on how to improve things, we'd love to work with them on this. But the important point is thinking about how improved messages can be implemented - it's often easy to think "it would be nice if pip could tell me XYZ" but when you look at the code, it becomes impossible to even see how pip can ever know that XYZ is the case. To give a specific example, I have no idea how we could usefully deliver the warnings you suggest. Information on what wheels are available is only available in the finder, and when the finder is called, we have no assurance that we'll ever do a build of that package. Furthermore, if we do select a source-only candidate, there's still no certainty that we'll do a build - we'll call the backend hook to prepare the metadata as part of the resolution process, but the backend might very well not do a build at that point, it might be able to calculate metadata without needing to do a build (for example, setuptools has the All of which is to say that your suggestions are really useful feedback, and match a lot of other suggestions we've seen (including a number we received from the user interface work that was funded a few years back). But unless someone comes along to explain how we can implement these ideas in practice, they will never become anything more than "nice to have" suggestions, I'm afraid. |
|
Just found this discussion whilst looking for options that help protect me and the inexperienced users I regularly have contact with against the apparently growing trend of malicious packages that act through arbitrary code execution on package install (I wrote about that here referencing the 2022 W4SP stealer amongst other resources). Am I right in thinking the A carefully worded opt-in flag that can be set as an environment variable would seem to give users who decide it's worth the risk a minimal-fuss option to continue as before (maybe |
|
Using Making it a default is generally considered to be a good idea, as well, but we need to plan the transition, and that's what this is stalled on. Breaking every package that hasn't uploaded wheels isn't really a good move... |
|
To be clear, IDK if this was just a typo, or a misunderstanding. It may be surprising if you're used to thinking of 'binary' meaning executable, as in the |
|
I think the confusion comes from thinking about install time versus run time. Both can run arbitary code at run time (when the package is imported). But at install time with binary you only need to download it, you don't need to execute it, because it's already been pre-built (which could include compiled executables). Whereas non-binary is source code and you need to build it which can involve running arbitary code. |
|
Btw, as real world evidence, in the now defunct rip project they started off with only supporting wheels (binary) and it made lots of resolutions impossible, and I wasn't able to test it against any real world work project I had until it started to support sdists. |
|
@takluyver you're right, it was a typo on my part - I did mean |
|
I think there's another approach here that's more straightforward to implement:
The inspiration here is there are a few projects that already do not provide sdists at all because the build process is so difficult that they don't want users to try. That's the current workaround for this issue, and while it's deeply unfortunate that sdists don't exist, the nice thing about that workaround is that it's opt-in on a per-project basis. This proposal merely formalizes that workaround and makes the sdists available if you really want them. Existing projects that provide only sdists, or that do not reliably provide wheels, would continue to upload normal sdists, and they would be resolved like normal. No changes to build tools are strictly required (as with the proposals to add metadata); all you need to do is rename the file. We certainly could make this nicer but it's not necessary. This is backwards-compatible with existing versions of pip (and other installers): for projects that provide discourage sdists, they will just not recognize the discouraged sdists as a file type they can use and they'll gracefully degrade to same behavior as if sdists are not uploaded at all. And for projects that don't, the current behavior (including attempting to build normal sdists) is correct. (We can probably come up with a better name than "discouraged" ... "manual"? "intentional"?) Does this seem like a reasonable approach? If so I can suggest it on the forums, since it's mostly not a pip change. (Thanks @zooba for mentioning this issue to me in another forum thread.) |
Doing that would require a new standard, and hence a PEP. It's not something pip (or PyPI) would adopt unless it was standardised. Personally, I don't think it's something we should try to standardise, but if you want to, then feel free to develop a PEP for it. I'd rather we simply made |
I think the only unavoidable complication is how to handle packages that have an sdist but no wheels at all. In that case, I'd prefer the default to be to use the sdist, but that's a complication (and if I specify |
|
Maybe it needs a |
Which is what already happens, so it's slightly stronger than that in that the presence of any binary options at all prevents choosing the source option. I don't have a problem with the naming, just that the obvious interpretation isn't what we need here. |
Well, from the discussion on this thread, that is quite complicated itself, right? I think the least complicated proposal is Did I miss something in the discussion above that makes this change easy to implement? I'm happy to write the code if so, I've got a bit of free time and I care about this problem, but I didn't see any designs that looked uncomplicated - but I did see comments about fundraising and a careful rollout process with community outreach. :) My proposal does not change any existing behavior, and is therefore safe to roll out immediately without any coordination.
Yes, agreed this is a PEP and not just a pip issue. I started a discussion here: https://discuss.python.org/t/preventing-unwanted-attempts-to-build-sdists/54169 |
Yes. I just think it's less complicated than adding a new type of sdist into the mix. |
|
I will say: I think using a rollout coupled with a specific unreleased-at-start-time Python version will be a better mechanism to do this. |
I haven't looked at the code so maybe this is not straight-forward but here is an example of the current error message with pip 24.1: Click to see pip error output:$ pip install python-flint
Collecting python-flint
Downloading python-flint-0.6.0.tar.gz (3.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 4.6 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: python-flint
Building wheel for python-flint (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for python-flint (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [178 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-313
creating build/lib.linux-x86_64-cpython-313/flint
copying src/flint/__init__.py -> build/lib.linux-x86_64-cpython-313/flint
creating build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/__init__.py -> build/lib.linux-x86_64-cpython-313/flint/flintlib
creating build/lib.linux-x86_64-cpython-313/flint/flint_base
copying src/flint/flint_base/__init__.py -> build/lib.linux-x86_64-cpython-313/flint/flint_base
creating build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/__init__.py -> build/lib.linux-x86_64-cpython-313/flint/types
creating build/lib.linux-x86_64-cpython-313/flint/functions
copying src/flint/functions/__init__.py -> build/lib.linux-x86_64-cpython-313/flint/functions
creating build/lib.linux-x86_64-cpython-313/flint/utils
copying src/flint/utils/flint_exceptions.py -> build/lib.linux-x86_64-cpython-313/flint/utils
copying src/flint/utils/__init__.py -> build/lib.linux-x86_64-cpython-313/flint/utils
creating build/lib.linux-x86_64-cpython-313/flint/test
copying src/flint/test/test.py -> build/lib.linux-x86_64-cpython-313/flint/test
copying src/flint/test/__init__.py -> build/lib.linux-x86_64-cpython-313/flint/test
copying src/flint/test/__main__.py -> build/lib.linux-x86_64-cpython-313/flint/test
running egg_info
writing src/python_flint.egg-info/PKG-INFO
writing dependency_links to src/python_flint.egg-info/dependency_links.txt
writing top-level names to src/python_flint.egg-info/top_level.txt
reading manifest file 'src/python_flint.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'CHANGES'
adding license file 'LICENSE'
writing manifest file 'src/python_flint.egg-info/SOURCES.txt'
copying src/flint/pyflint.c -> build/lib.linux-x86_64-cpython-313/flint
copying src/flint/pyflint.pxd -> build/lib.linux-x86_64-cpython-313/flint
copying src/flint/pyflint.pyx -> build/lib.linux-x86_64-cpython-313/flint
copying src/flint/flintlib/__init__.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/acb.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/acb_calc.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/acb_dft.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/acb_dirichlet.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/acb_elliptic.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/acb_hypgeom.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/acb_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/acb_modular.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/acb_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/arb.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/arb_fmpz_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/arb_hypgeom.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/arb_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/arb_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/arf.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/arith.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/bernoulli.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/dirichlet.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/flint.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpq.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpq_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpq_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_factor.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_lll.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_mod.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_mod_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_mod_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_mod_poly_factor.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_mod_vec.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_mpoly.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_poly_factor.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/fmpz_vec.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/mag.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/mpoly.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/nmod.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/nmod_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/nmod_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/nmod_poly_factor.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/nmod_vec.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/partitions.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flintlib/ulong_extras.pxd -> build/lib.linux-x86_64-cpython-313/flint/flintlib
copying src/flint/flint_base/flint_base.c -> build/lib.linux-x86_64-cpython-313/flint/flint_base
copying src/flint/flint_base/flint_base.pxd -> build/lib.linux-x86_64-cpython-313/flint/flint_base
copying src/flint/flint_base/flint_base.pyx -> build/lib.linux-x86_64-cpython-313/flint/flint_base
copying src/flint/flint_base/flint_context.c -> build/lib.linux-x86_64-cpython-313/flint/flint_base
copying src/flint/flint_base/flint_context.pxd -> build/lib.linux-x86_64-cpython-313/flint/flint_base
copying src/flint/flint_base/flint_context.pyx -> build/lib.linux-x86_64-cpython-313/flint/flint_base
copying src/flint/types/acb.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb_mat.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb_mat.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb_poly.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb_poly.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb_series.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb_series.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/acb_series.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb_mat.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb_mat.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb_poly.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb_poly.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb_series.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb_series.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arb_series.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arf.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arf.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/arf.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/dirichlet.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/dirichlet.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/dirichlet.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq_mat.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq_mat.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq_poly.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq_poly.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq_series.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq_series.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpq_series.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mat.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mat.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mod.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mod.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mod.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mod_mat.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mod_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mod_mat.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mod_poly.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mod_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mod_poly.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mpoly.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mpoly.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_mpoly.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_poly.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_poly.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_series.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_series.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/fmpz_series.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod_mat.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod_mat.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod_mat.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod_poly.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod_poly.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod_poly.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod_series.c -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod_series.pxd -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/types/nmod_series.pyx -> build/lib.linux-x86_64-cpython-313/flint/types
copying src/flint/functions/showgood.c -> build/lib.linux-x86_64-cpython-313/flint/functions
copying src/flint/functions/showgood.pyx -> build/lib.linux-x86_64-cpython-313/flint/functions
copying src/flint/utils/conversion.pxd -> build/lib.linux-x86_64-cpython-313/flint/utils
copying src/flint/utils/typecheck.pxd -> build/lib.linux-x86_64-cpython-313/flint/utils
running build_ext
building 'flint.pyflint' extension
creating build/temp.linux-x86_64-cpython-313
creating build/temp.linux-x86_64-cpython-313/src
creating build/temp.linux-x86_64-cpython-313/src/flint
gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/home/oscar/.pyenv/versions/3.13t-dev/include/python3.13t -c src/flint/pyflint.c -o build/temp.linux-x86_64-cpython-313/src/flint/pyflint.o
src/flint/pyflint.c:1254:10: fatal error: flint/mag.h: No such file or directory
1254 | #include "flint/mag.h"
| ^~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for python-flint
Failed to build python-flint
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (python-flint)The primary problem with this error message is that it does not even hint at the root cause of the problem or the likely solution from an end user perspective. It is unlikely that the solution for a user trying to install something is that they read the build log output and then do something to make the build succeed on a subsequent run. Of course some users can fix this but the error message UI should be designed firstly for users who are assumed to be unable to fix the build. The root cause of the install failure is the fact that there are no wheels available for the given Python version, platform and architecture which in this case is CPython 3.13 free-threaded on Linux x86_64. The likely solution for an end user is to use a different Python version like CPython 3.12 or a non-free-threaded build of 3.13. Or the solution might even be to use a different OS like WSL on Windows. The opening message says nothing about why it is downloading the sdist rather than installing from a wheel: It should at least say something like:
It could also list the available wheels that were rejected which would likely help some people to understand the problem. The final error message is:
It should probably emphasise the absence of prebuilt wheels as the problem and then read more like:
Even better would be to say which wheels are available for the given version of python-flint here at the end or even to summarise common issues:
I don't know if the whole subprocess business makes it difficult to even know that python-flint 0.6.0 is the build that failed. If it is possible then I think it would be worthwhile to query PyPI to find the list of wheels for the purpose of giving a better error message at this point. |
This is somewhat similiar to better error message requests when no version is found on the specific platform, e.g. #12234. But in the case that it should mention this for sdist builds. I assume there would be general support for someone making a PR That improved error messaging like this. |
💯 |
Having looked at the code I can see what you mean. The general organisation of where data is stored in the different classes is not really what I had imagined. I would have imagined grouping the links into versioned releases first and then keeping the links in a release together until a version is selected but that is not how it is structured. It took a while for me to find where the list of wheels appears and is filtered for those with non-matching platforms. At the stage where they are discarded it is not yet known what version is going to be selected so the entire list of all "links" for all versions of a distribution are downloaded and those without matching platform tags are immediately filtered out. After the filtering done by The only easy way I can see is that the finder would need to store every link ever seen rather than discarding non-matching ones so that the information is available for subsequent use: pip/src/pip/_internal/index/package_finder.py Lines 750 to 753 in 47ffcdd I don't have much of a sense of how reasonable/unreasonable it is to store all of them in terms of memory use. An alternative is retrieving the information from PyPI again which is perhaps not unreasonable at the point of exiting with an error. A modified version of |
Would it be reasonable to do this? It seems like it might also be necessary to do something like this in order to be able to implement some of the suggestions above like "use sdist only if the project never had wheels for any release before the one being installed". Possibly that sort of logic cannot be implemented now because non-matching wheels are discarded before any attempt to think about which version to install. |
I don't know. Someone would probably need to try it and detemine the impact on code complexity and performance. Even though I'm very much in favour of improving the error messages, I'm more concerned that we don't over-complicate the code structure (while it's far from easy to follow, the basic structure of separating the "find" step from the "resolve" step is pretty clean). |
|
Do you need every discarded wheel or just that there was at least one discarded wheel? |
I want every available wheel for the version that was ultimately selected for building e.g. if building matplotlib 10.3 fails than I want the names of all wheels for matplotlib 10.3. The problem is that at the time of discarding the wheels we don't know which version of matplotlib is going to be selected (although I guess the most recent version is likely). Note that we are just talking about discarding the filenames of the wheels vs storing all filenames in memory (the files themselves are not needed). All filenames are loaded from PyPI but most are immediately discarded e.g. for being the wrong platform or Python version. If we do store the filenames in memory though then we probably end up storing the names of all wheels for all versions of all distributions that the finder looks at during installation. I don't have much of a sense for whether that is sometimes a very big list that you would not want to store in memory. Thinking about this later though I think it is cleaner just to retrieve the desired filenames from PyPI again at the point of build failure. That way it has no impact on successful installation and doesn't need to complicate the happy codepath. |
What's the problem this feature will solve?
A lot of users are reporting issues when there's no Python 3.9 binary for projects they need, and pip tries to build from source and fails with an obscure error (because the user doesn't have a compiler, or isn't set up to build the relevant packages).
Describe the solution you'd like
Pip shouldn't try to build from source if the user isn't prepared to deal with build errors. As it's not possible to know the user's level of expertise, we should err on the side of caution, and by default only allow wheels to be installed. Users who know they need to install from source and have checked that they can do so, can explicitly say so using a new
--allow-sourceflag, which acts as an "opt-in" to source builds.Alternative Solutions
Improve the error messages when a source build fails. This is hard, because the details of what went wrong are entirely the responsibility of the build backend.
Additional context
I don't realistically think this can be added without a lot of disruption, but given that significant numbers of projects ship wheels these days, maybe it isn't as unthinkable as it once was. I do think it's worth discussing the implications, if only as a thought experiment, and I don't know where else we could do that apart from here.
One big problem area is that we can't distinguish between "pure Python" projects that are shipped only as sdists, but which only need Python to build, and complex projects that need a compiler. So restricting to wheels only would require an explicit opt-in for some projects which currently install with no issue.
The text was updated successfully, but these errors were encountered: