pip download --no-binary :all: --no-build-isolation importlib_medata fails to fetch a proper sdist or takes a long time to complete

[pombredanne]

Description

This fails with a pinned version:

$ pip download --no-binary=:all: --dest wheels/   --no-build-isolation importlib_metadata==4.11.1
Collecting importlib_metadata==4.11.1
  Using cached importlib_metadata-4.11.1.tar.gz (42 kB)
  Preparing metadata (pyproject.toml) ... done
  WARNING: Requested importlib_metadata==4.11.1 from https://files.pythonhosted.org/packages/82/10/600b88188a4e94562cfdf202ef1aca6fedda0723acae8a47376350ec0d5d/importlib_metadata-4.11.1.tar.gz#sha256=175f4ee440a0317f6e8d81b7f8d4869f93316170a65ad2b007d2929186c8052c, but installing version 0.0.0
Discarding https://files.pythonhosted.org/packages/82/10/600b88188a4e94562cfdf202ef1aca6fedda0723acae8a47376350ec0d5d/importlib_metadata-4.11.1.tar.gz#sha256=175f4ee440a0317f6e8d81b7f8d4869f93316170a65ad2b007d2929186c8052c (from https://pypi.org/simple/importlib-metadata/) (requires-python:>=3.7): Requested importlib_metadata==4.11.1 from https://files.pythonhosted.org/packages/82/10/600b88188a4e94562cfdf202ef1aca6fedda0723acae8a47376350ec0d5d/importlib_metadata-4.11.1.tar.gz#sha256=175f4ee440a0317f6e8d81b7f8d4869f93316170a65ad2b007d2929186c8052c has inconsistent version: filename has '4.11.1', but metadata has '0.0.0'
ERROR: Could not find a version that satisfies the requirement importlib_metadata==4.11.1 (from versions: 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.21, 0.22, 0.23, 1.0.0, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.6.1, 1.7.0, 2.0.0, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 3.0.0, 3.1.0, 3.1.1, 3.2.0, 3.3.0, 3.4.0, 3.5.0, 3.6.0, 3.7.0, 3.7.1, 3.7.2, 3.7.3, 3.8.0, 3.8.1, 3.8.2, 3.9.0, 3.9.1, 3.10.0, 3.10.1, 4.0.0, 4.0.1, 4.1.0, 4.2.0, 4.3.0, 4.3.1, 4.4.0, 4.5.0, 4.6.0, 4.6.1, 4.6.2, 4.6.3, 4.6.4, 4.7.0, 4.7.1, 4.8.0, 4.8.1, 4.8.2, 4.8.3, 4.9.0, 4.10.0, 4.10.1, 4.11.0, 4.11.1)
ERROR: No matching distribution found for importlib_metadata==4.11.1

This works but gets a very old version after trying to fetch almost all versions:

$ pip download --no-binary=:all: --dest wheels/   --no-build-isolation --no-deps importlib_metadata
Collecting importlib_metadata
  Using cached importlib_metadata-4.11.1.tar.gz (42 kB)
  Preparing metadata (pyproject.toml) ... done
Discarding https://files.pythonhosted.org/packages/82/10/600b88188a4e94562cfdf202ef1aca6fedda0723acae8a47376350ec0d5d/importlib_metadata-4.11.1.tar.gz#sha256=175f4ee440a0317f6e8d81b7f8d4869f93316170a65ad2b007d2929186c8052c (from https://pypi.org/simple/importlib-metadata/) (requires-python:>=3.7): Requested importlib_metadata from https://files.pythonhosted.org/packages/82/10/600b88188a4e94562cfdf202ef1aca6fedda0723acae8a47376350ec0d5d/importlib_metadata-4.11.1.tar.gz#sha256=175f4ee440a0317f6e8d81b7f8d4869f93316170a65ad2b007d2929186c8052c has inconsistent version: filename has '4.11.1', but metadata has '0.0.0'
[....]
  Using cached importlib_metadata-0.7.tar.gz (16 kB)
  Preparing metadata (pyproject.toml) ... done
Saved ./wheels/importlib_metadata-0.7.tar.gz
Successfully downloaded importlib_metadata

or this works without --no-build-isolation but takes a very long time to complete which makes it impractical to use.

$ time pip download --no-binary=:all: --dest wheels/   --no-deps importlib_metadata==4.11.1
Collecting importlib_metadata==4.11.1
  File was already downloaded /home/pombreda/tmp/pipo/wheels/importlib_metadata-4.11.1.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Successfully downloaded importlib_metadata

real	0m46.409s
user	0m42.389s
sys	0m2.196s

Expected behavior

The sdist should be downloaded alright, reasonably fast and never fail.

BUT instead of reading the sdist PKG-INFO metadata (which IMHO should not even ever be opened), pip opens the setup.cfg which is using setuptools_scm and fails.... because there is no version there. And fails because it somehow would need setuptools_scm to read a PKG-INFO file?

Instead the command (without --no-build-isolation) eventually works but takes forever because it is somehow running something... I would expect a mere read of the metadata not to require running anything for an sdist. I would expect reading metadata not even br needed for a --no-deps download.

So with all these, it becomes rather hard to reliably or practically fetch ONLY the sdist: either it takes a long time, or it fails, or is incorrect (returning a very old version like in the first example above)

pip version

pip 22.0.3

Python version

3.9

OS

linux

How to Reproduce

pip download --no-binary=:all: --dest wheels/ --no-build-isolation importlib_metadata==4.11.1

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Can you try running with -v or even -vv and identify where the slowdown occurs?

I'm pretty sure it's an issue with importlib metadata or setuptools-scm that's causing problems here.

This message is a symptom of that problem:

Discarding https://files.pythonhosted.org/packages/82/10/600b88188a4e94562cfdf202ef1aca6fedda0723acae8a47376350ec0d5d/importlib_metadata-4.11.1.tar.gz#sha256=175f4ee440a0317f6e8d81b7f8d4869f93316170a65ad2b007d2929186c8052c (from https://pypi.org/simple/importlib-metadata/) (requires-python:>=3.7): Requested importlib_metadata from https://files.pythonhosted.org/packages/82/10/600b88188a4e94562cfdf202ef1aca6fedda0723acae8a47376350ec0d5d/importlib_metadata-4.11.1.tar.gz#sha256=175f4ee440a0317f6e8d81b7f8d4869f93316170a65ad2b007d2929186c8052c has inconsistent version: filename has '4.11.1', but metadata has '0.0.0'

I'll take a note that we need to improve this warning.

/cc @jaraco for the kind of weird failures that setuptools is causing due to having a default version instead of failing. This is the sort of thing that I'm concerned about, in pypa/setuptools#2329 (comment).

[pradyunsg]

Also, if the issue here is that you don't have setuptools-scm installed, then you should get better behaviour from pip in main, with --no-build-isolation (a clear error message).

[pombredanne]

@pradyunsg Thanks for the quick reply... not that the PKG-INFO metadata are correct... it's only the setup.cfg which is of partially filled in.
See the attached log.

The build isolation seems to create some complete virtual env with pip/setuptools/setuptools_scm ... from scratch and seem to build everything from source, may be it is mining bitcoins? 👼 😇 ... which is not needed since I do not require a build but only a mere sdist download, and I do not require the deps either ... so there is a lot of work likely done extra when I would likely only expect a lookup to PyPI to get the URL to download at most before the download proper and nothing else beyond this.
pip-log.txt

[pombredanne]

And here is a verbose log with --no-buidl-isolation
pip-log-no-bi.txt

At least this fails fast. In all cases setuptools_scm has been involved.

[pombredanne]

Also, if the issue here is that you don't have setuptools-scm installed, then you should get better behaviour from pip in main, with --no-build-isolation (a clear error message).

@pradyunsg I tend to think that setuptools-scm should never be needed when I merely download a source distribution. No build should ever be attempted in the first place IMHO.

[pradyunsg]

Well, pip download pkg will download pkg and its dependencies.

Discovering the dependencies needs the metadata to be generated (pip doesn't read PKG-INFO files, only METADATA, if I remember correctly -- there's also a PEP standardising metadata format for sdists that hasn't been implemented yet).

And, if you're thinking you can circumvent that by passing no-deps, you're right but right now "download" and "generate metadata" are highly coupled in pip -- there's an open issue for no-deps pip download and metadata generation as well.

[pombredanne]

@pradyunsg ok, fair enough. How can I help there? Practically pip download is essentially useless in its current incarnation as it never can reliably download all the things you ask it to, even if it takes a long time. I have medium size pinned requirements files with about 100 specs and this never completes correctly. So the alternative for me is either: help fix it here or craft my own effective and working downloader. What would you recommend?

[pombredanne]

@pradyunsg I am not sure why you labeled this R: wrong project and type: not a bug
If pip cannot download existing sdists at all, is this not a bug?