non-downloadable/installed dependencies shouldn't fail upgrades

[qwcode]

if some pkgb depends on pkga (i.e. has it specified in install_requires), an upgrade of pkgb shouldn't fail if a download isn't found for pkga (e.g. if it's not in PyPI), if pkga is installed locally.

to recreate, create pkga and pkgb in local directories with setup.py's like so

pkga/setup.py

from setuptools import setup
setup(name='pkga',
      version='0.1')

pkgb/setup.py

from setuptools import setup
setup(name='pkgb',
      version='0.1',
      install_requires=['pkga'])

and run

> pip install -e pkga/
> pip install -U -e pkgb/
Could not find any downloads that satisfy the requirement pkga in ./pkga (from pkgb==0.6.1)
Collecting pkga (from pkgb==0.6.1)
  No distributions at all found for pkga in ./pkga (from pkgb==0.6.1)

[qwcode]

issue came out of discussion in #2472

[msabramo]

I feel like there are 2 possible approaches to upgrading.

1. Download/do the minimal amount required to satisfy the constraints. Be lazy and don't hit the network unless necessary. This is better for speed and greater ability to work offline, but you end up with older versions of packages and maybe more variation in people's environments, which could lead to people having problems that are fixed in later versions.
2. Try to always upgrade to the latest. This tends to get people to the latest and to have more similar envs to each other and so maybe less bugs because folks are more likely to be on the same page. It does cause more downloads though so it makes pip slower and less likely to work when offline.

I think pip largely implements Approach 2.

I wonder if it would be useful to have pip support changing the strategy to Approach 1. But the fear I have is that this would make the code more complex, especially if no refactoring is done before to isolate the strategy from everything else.

Maybe it's worth documenting pip's overall strategy (if it isn't already) so we can at least point people to a URL to explain why they're seeing the behavior they're seeing?

[remram44]

I'm very OK with 2) so long as failing to contact or find pkg on PyPI for recursive deps is not a failure.

[msabramo]

@remram44: Apologies if this already came up - I'm on a phone so it's not so easy to browse around and check stuff...

I wonder if it would help if you added --find-links .. Maybe that would encourage pip to find pkga locally, though it might still try PyPI. I guess you could also try something like --find-index . --no-index an you might want to put those in a requirements.txt to make things easier.

See https://pip.pypa.io/en/latest/user_guide.html#fast-local-installs

[msabramo]

Ok, the 2 approaches I meant have been thoroughly documented already in the link I mentioned and in #59

[msabramo]

@remram44 : Would #59 help you?

[remram44]

#59 would make the current upgrade be upgrade-all, which would probably still fail if some packages in the dependency tree are not on PyPI. Which would still be wrong. #59 unrelated IMHO.

[qwcode]

@msabramo I think pip has to do #2, but that doesn't mean it should suffer from the bug in the description. pip should just know that not finding any downloads is ok for an installed dependency, as long as it fulfills the requirement.

[msabramo]

@qwcode: I agree that not finding any downloads for a dependency shouldn't fail the whole upgrade.

[qwcode]

fixed in PR #2538

[piotr-dobrogost]

Does it work for every level of dependency tree or only for top level?

[qwcode]

you mean the PR fix? should be any level