pip install [git repo] hangs on clone step on Windows with large repositories

[Mikuana]

Environment

• pip version: 20.2
• Python version: 3.7 and 3.8
• OS: Windows

Python packages are available as git repositories, hosted in AWS CodeCommit, with authentication handled via AWS CLI integration with git credential helper.

Description

Pip version 20.2 causes VCS pip install from AWS CodeCommit to freeze during the clone step. This does not occur with all our repos, but occurs consistently for the ones with the problem.

My team has a suite of python packages that we develop for our internal use. These packages are available as git repositories, and we install them using the pip VCS integration feature.

pip install git+https://git-codecommit.us-east-2.amazonaws.com/v1/repos/MyDemoRepo

This solution has worked for us, without issue, until we upgraded to pip version 20.2. I have tested that all minor versions between 19.1 and 20.1 work as expected.

On version 20.2, with specific repositories, when the install gets to the clone step it stalls there, and the process never completes. The temporary folder is initialized, and the hidden .git repo appears to be populated and in-tact, but the rest of the repo is empty and the process never completes.

Adding the verbose switch does not provide any detail as to why the process is hanging. Examination of logs on the side of the remote server suggest that the git command is being processed correctly.

Expected behavior
Using pip install git on a CodeCommit repo will complete or raise an error for all repositories.

How to Reproduce
Reproducing may be difficult, since the issue only occurs with some of our packages, but we don't know why. Our code is not public, so I can't expose it directly. However, the steps are as follows:

1. Install python 3.7 and pip 20.2 on Windows
2. Install AWS CLI
3. Get an AWS API Token
4. Configure .gitconfig credential helper to use AWS CLI for CodeCommit URL
5. Start installing python packages with pip install git+https://; we're not sure why some work and some don't, so this last step is a little, uninformative...

Output

pip install git+https://git-codecommit.us-east-2.amazonaws.com/v1/repos/MyDemoRepo

Collecting git+https://git-codecommit.us-east-2.amazonaws.com/v1/repos/MyDemoRepo
  Cloning https://git-codecommit.us-east-2.amazonaws.com/v1/repos/MyDemoRepo to /tmp/pip-req-build-rpozcsdw

[uranusjr]

The only significant change to VCS recently is #8817 but that doesn’t seem that related since you’re just fetching the default branch. Do the stuck repos have submodules or something?

[Mikuana]

No, there's no sub-module, or anything exotic in terms of the git structure. It's a fairly vanilla setup, with a .gitignore. The file structure is like this:

| repo
| ------- module/
|--------.gitignore
|--------LICENSE
|--------README.md
|--------setup.py

[zhangk1551]

Same problem for python 3.6 & pip 20.2.4.

[pradyunsg]

Can folks confirm if this occurs with pip 20.3?

[Mikuana]

Confirmed that the same issue occurs with pip 20.3

[pradyunsg]

Thanks for the confirmation!

Someone needs to provide us with a reproducer here, or contribute the fix, because pip's contributors currently have no way to test this.

[Mikuana]

I'll see if I can further isolate the issue. I've done some testing with a few different variables. So far the only variable that has seemed to make a difference is running the command from Ubuntu instead of Windows. It's possible this is an OS dependent thing, but whether that's a purely OS issue, an OS-pip issue, or an OS-git issue, I don't know yet.

[Mikuana]

I believe I've isolated the issue down to the way that Git for Windows behaves when working with large repos. For some reason, Git on Windows outputs to stderr instead of stdout when reporting on the download of large repos. In combination with the changes made to subprocess calls in v20.2, for some reason this results in the entire subprocess hanging. Essentially, when the VCS install subprocess runs, it's looking for a response on stdout, but if the repo is large enough everything gets output to stderr instead, and the whole thing hangs. It doesn't seem like this should happen (there may be an underlying bug in Git or in the subprocess module), but everything stops at the stdout read.

I've tested a fix based on v20.2.4 that first checks stderr, then checks stdout. I can't open a pull though, because there's no branch for that version, just a tag.

I've got a link to the comparison below. What's the process to fully test this and get it merged as a patch on 20.2 and 20.3?

https://github.com/pypa/pip/compare/pypa:20.2.4...Mikuana:v20.2-vcs-bugfix?expand=1

[Mikuana]

@pradyunsg would it be possible to get a branch created from the 20.2.4 tag so that I can open a pull request directly against it? Then we can discuss the rest of the details in the PR instead of in this issue.

[uranusjr]

But you won’t be able to get the patch into a new pip version if you don’t create the patch against 20.3 (or master). What are the problems preventing a branch from 20.3 being possible?

[Mikuana]

None that I know of. I was planning to do that next. I was just thinking it would be best to patch the minor version where the issue first appeared, then work the change into the rest of the history from there.

That way, if someone is doing a large VCS install, it will work in 20.1, 20.2, and 20.3.

[pradyunsg]

Feel free to make a PR against the current master. We usually don't backport fixes as follow CalVer, and don't usually cut bugfix releases for older versions.

I don't see why this issue needs to be treated differently anyway, so I reckon a PR to master is sufficient. ^>^

[Mikuana]

@pradyunsg sounds good. I think there's an case to be made for patching 20.2, but I'm certainly not going to argue against your normal process. I'll set up a pull against master and reference it here.

[Mikuana]

PR to master opened in #9234

[sbidoul]

@pradyunsg Should we reopen this to track the backport to 20.3.4 ? Or is it a better way to track it ?