New resolver: Every time I run pip install it clones remote repositories, regardless if requirement already satisfied.

[skofgar]

What did you want to do?

I'm have a requirements.txt where I'm pulling dependencies, some of which pull directly from GitHub/GitLab repositories.

Example:

pip install git+https://github.com/psf/requests.git#egg=requests

I'm aware that requests, can just be installed with pip install requests, but the problem happens with a bunch of private repositories, that I cannot share.

Problem
My problem is, that in the previous resolver the dependency would only get installed on the first run of pip install ... and all consecutive runs would say "Requirement already satisfied". However, with the new resolver, it always clones the repository, making our builds significantly slower. Also sometimes we pre-install private repo requirements, so that later runs of pip install -r requirements.txt don't require the credentials, but that also fails now.

Output

Example runs:

> pip install git+https://github.com/psf/requests.git#egg=requests
Collecting requests
  Cloning https://github.com/psf/requests.git to /private/var/folders/ss/nyrxzdcs66x433hfl0_c7sb976qv3q/T/pip-install-9u1iggu3/requests_e94dd8fbb0ec4c3192e3320bb267ba89
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/Caskroom/miniconda/base/envs/testenv/lib/python3.6/site-packages (from requests) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/Caskroom/miniconda/base/envs/testenv/lib/python3.6/site-packages (from requests) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/Caskroom/miniconda/base/envs/testenv/lib/python3.6/site-packages (from requests) (1.26.2)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/Caskroom/miniconda/base/envs/testenv/lib/python3.6/site-packages (from requests) (2020.11.8)

consecutive runs:

pip install git+https://github.com/psf/requests.git#egg=requests
Collecting requests
  Cloning https://github.com/psf/requests.git to /private/var/folders/ss/nyrxzdcs66x433hfl0_c7sb976qv3q/T/pip-install-pwog_t5w/requests_15051cb7b9dd44e4984969dceaa734be
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/Caskroom/miniconda/base/envs/testenv/lib/python3.6/site-packages (from requests) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/Caskroom/miniconda/base/envs/testenv/lib/python3.6/site-packages (from requests) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/Caskroom/miniconda/base/envs/testenv/lib/python3.6/site-packages (from requests) (1.26.2)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/Caskroom/miniconda/base/envs/testenv/lib/python3.6/site-packages (from requests) (2020.11.8)

Additional information

PIP Version:

> pip --version
pip 20.3 from /usr/local/Caskroom/miniconda/base/envs/testenv/lib/python3.6/site-packages/pip (python 3.6)

[uranusjr]

See #5780. Summary: The previous behaviour was inconsistent and people want either way for many given flag combinations, so we’re going back to the drawing board and trying to come up with something that makes more sense. Can you take a look at the conclusion made toward the end of the thread, and see if it makes sense for you? We really need user feedback to gauge if our plan would work.

[uranusjr]

Closing this so discussion would go the the linked thread.

[skofgar]

Thanks @uranusjr, I'm reading through the discussion now