pip download behaves differently for editable and non-editable VCS requirements

[sbidoul]

Environment

• pip version: 20.3.3
• Python version: irrelevant
• OS: irrelevant

Description

Assume the following requirements.txt:

-e git+https://github.com/pypa/packaging#egg=packaging

pip download -r requirements.txt does a VCS export, hence does not include the .git directory in the resulting zip file.

Without -e, the same pip download command produces a zip file that contains the .git directory.

Analysis

This VCS export mechanism was presumably done as a performance optimization. However it has correctness issues if the build backend requires the VCS information to prepare metadata (#9337).

Possible approaches:

1. Remove the VCS export pip feature: this will be correct but may create performance issues in some scenarios that rely on it (with the presence of a possibly large .git directory in the resulting zip file and a larger download), this is done in Remove VCS export feature #9274.
2. Do a VCS clone for preparing metadata (as it is currently done for non-editable VCS requirements), then do a VCS export (2a) to prepare the zip file, possibly using the local clone as a source (2b), or using the build backend to create a sdist (2c). This will have some performance implications, but produce smaller archives as output (i.e. without the .git directory), and produce the same result with and without -e.
3. Keep the discrepancy and do not fix pip download of editable VCS requirements fails if requirement needs VCS information to prepare metadata #9337, reintroduce the distinction between pip wheel and pip download in order to fix pip wheel -e git+https:// does a git export instead of git clone #9273.

[sbidoul]

I'm inclined to release (1) and then implement one of the variants in (2) if the output zip size issue proves to be a problem in practice. I could live with (3) although it's not very clean.

[sbidoul]

Noting that variants (2a) and (2b) will still produce crippled zip files if the build backend relies on VCS information.

[uranusjr]

I wonder if shallow clone would be useful here. If not, reverting is the right approach IMO. Yes, this is create a performance issue, but we have other approaches to fix that (in-place build etc.), and people really should use pre-built distributions if they care about performance IMO.

[sbidoul]

For reference, shallow clones were discussed in #2432 and were rejected.

[uranusjr]

#2432 rejected shallow clones for pip install because it would break setuptools-scm. But for pip download, the worse case scenario is the user need an additional git fetch, which I feel is an acceptable middle ground.

[uranusjr]

I want to add that I’m by no means particularly in favour of any approach. It is perfectly fine IMO to always git clone the whole thing. Feel free to choose the solution you feel is best 🙂

[sbidoul]

Even for download, shallow clones will create issues with setuptools_scm (variants of #9337). FWIW, my group resolved that particular performance issue with git-autoshare. I think if pip ever needs to support shallow clones, that would be via a per-requirement option or something in the URL fragment, which would apply to all commands.