pip install killed due to excessive memory consumption (due to new pip resolver) #7768
Comments
|
What happens when you run the build locally? Does it succeed after some time or does it fail with a |
|
Eurgh so when you just try to install myst-nb (one of the key requirements) on its own, it takes ages and then you get: |
|
The above is when myst-nb is set with the requirement: This is weird because there has barely been any version changes between the two:
jinja2 is already a dependency of sphinx anyway and markdown-it-py/mdit-py-plugins have no dependencies of their own. So it should just resolve to exactly the same, except with updated versions of myst-parser and markdown-it-py |
|
Jeez, then I pin a some of the package versions, and I get this: markupsafe fixed this ages ago, pypa/setuptools#2017 (comment), but now the pip resolver is digging up these old faulty versions 😬 |
|
Ah I've figured out one of my issues: myst-nb -> myst-parser~=0.13.0 -> markdown-it-py~=0.6.0 but also myst-nb[testing] -> jupytext>=1.16.0 -> markdown-it-py~=0.5.3 A clear inconsistency, but one that pip did not identify 😢 So yeh, to be very unappreciative of your work @pradyunsg lol; it seems that when the pip resolver works its great 👍, but if there are any inconsistencies in the requirements it is prone to send it off on a "wild goose chase", in the case of RTD, often ending in an excessive memory consumption error 😞 (I completely understand that these dependency resolvers are not an easy thing to achieve though) |
|
out of interest @pradyunsg does the pip resolver use the PyPi JSON API, like poetry claims to do https://python-poetry.org/docs/faq/#why-is-the-dependency-resolution-process-slow? |
|
Hmm... Maybe you have a dependency conflict somewhere? I'll need to take a closer look at this, but I'm moving houses tomorrow. Please feel free to ping me over the weekend if I've not taken a look? pip doesn't use the JSON API because it's not standardised, and thus cannot be used on non-PyPI indexes. There's an effort to standardise it, but that's been a little slow given the all-volunteer nature of things. |
Do you know the reason of why this is only happening in Read the Docs for your setup? It seems that pip is cycling over the dependencies and downloading all versions of the same package (i.e ipython 7.x down to 5.x) and finally RTD kills this build because of "Build exited due to time out". |
Just because RTD uses small VMs for the build, with limited memory. This is generally fine, but when pip starts downloading lots of different versions it can eat up that memory fast. Just to clarify, in the specific case I linked to above, it was "solved" by making some changes to the dependency versions, albeit these fixes were not at any stage obvious from the pip install failures. But in other cases, the dependencies have all been correct, and it has required additional pinnings which are not strictly necessary (for example aiidateam/aiida-tutorials#278 (comment)) |
Hrm... In the particular build that you mentioned in the description, https://readthedocs.org/projects/markdown-it-py/builds/12576084/, that build was done in a 8Gb RAM VM. That build was killed due to time out (the message at the command output is incorrect) because it took more than 2011s to complete. We can increase the building time for that project if you want as a test. |
Wait you're saying that this is not correct "Command killed due to excessive memory consumption"?
as mentioned, this particular project is "fixed" now, so I don't need that thanks |
We add that message when we think it was killed due to memory, but it's not 100% guaranteed that it was a memory issue all the times. In particular, when the build has both messages: time out and memory, I'd trust more the time out one if the build took a lot of time as this case. However, if the exit code is 137 (our custom code for DOCKER OOM) there are more chances that it was a real OOM instead time out. |
|
BTW, if this doesn't happen on a previous version of pip, a workaround can be to downgrade pip on rtd #7752 (comment) |
|
Is there anything actionable on this issue or we should close it? |
|
hey @humitos no thanks I guess in the end just something to bear in mind if others report similar issues; but mainly a pip issue. |
no-response
bot
removed
the
Needed: more information
Details
I have opened this issue on pip itself (pypa/pip#9318) but I'm opening it here as well, to track its progress, since it is only on RTD that I have encountered it, and I'm sure I will not be the only RTD user affected by this.
Also perhaps there may be an RTD specific aspect to the fix.
Example RTD build: https://readthedocs.org/projects/markdown-it-py/builds/12576084/
The text was updated successfully, but these errors were encountered: