install --target should offer option to supplement existing packages

[jaraco]

I'd like to replace the Setuptools setup_requires and tests_requires dependency resolution with something that relies on pip. To this end, I've created rwt, which relies heavily on pip install --target to install the requisite packages.

One major difference between Setuptools setup_requires behavior and pip install --target is that pip install --target installs all requirements unconditionally while easy_install will only install those packages not already satisfied by the current environment (working set).

For the purposes of rwt and probably other use cases, if pip had a similar option for --target that would only install those packages not already satisfied by the current environment, that would be preferable, especially in cases where one is seeking to augment or override a small number of packages from an otherwise large set already mostly available.

[jaraco]

Consider this simple case:

$ python -m pip install requests
Collecting requests
  Downloading requests-2.12.1-py2.py3-none-any.whl (574kB)
    100% |████████████████████████████████| 583kB 2.1MB/s 
Installing collected packages: requests
Successfully installed requests-2.12.1
$ python -m pip install -t tmp requests_toolbelt
Collecting requests_toolbelt
  Using cached requests_toolbelt-0.7.0-py2.py3-none-any.whl
Collecting requests<3.0.0,>=2.0.1 (from requests_toolbelt)
  Using cached requests-2.12.1-py2.py3-none-any.whl
Installing collected packages: requests, requests-toolbelt
Successfully installed requests-2.12.1 requests-toolbelt-0.7.0
$ PYTHONPATH=tmp python
Python 3.6.0b3 (v3.6.0b3:8345e066c0ed, Oct 31 2016, 18:05:23) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__file__
'/Users/jaraco/tmp/requests/__init__.py'

Requests is installed in system site-packages, then installed again in ./tmp, even though the version in system site packages is sufficient for the purposes of this invocation.

[jaraco]

The requested behavior would be particularly useful in cases where the dependencies are already satisfied and thus would require no work to satisfy the install operation.

[chrahunt]

@jaraco, do you still need this post-PEP 517? Maybe something different?

[bloer]

I just ran into this and was about to submit a new issue. Based on #4389, it seems the intent of --target is to have a fully standalone directory with all dependencies. It seems that similar behavior can be obtained via either --prefix and --root, which handle the dependencies in the requested way.

Maybe all this could be solved by adding some documentation for how those three options differ, and what are the intended use cases?

[jaraco]

I could still use this functionality. pip-run (formerly rwt) does still wish to only install the packages not already satisfied. Currently, pip-run uses a rough heuristic to determine which packages are already installed and strip those out of the command-line (but not from requirements.txt), which sort-of works but isn't very reliable. Better would be for pip to do its usual resolution and only install the dependencies not already met.

It seems that similar behavior can be obtained via either --prefix and --root.

I seem to think I tried that, but it doesn't behave like --target. In particular, --prefix adds lib/python3.8/site-packages/ (or similar) to the target... and --root adds Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ (or similar).

Additionally, I notice that when a package is upgraded, either due to -U --upgrade-strategy=eager or because a more specific version is indicated, --prefix and --root invocations will uninstall the non-satisfying package, which is undesirable.

What I really want is something like --target, which installs the packages and their metadata in one nice, flat directory with all of the dependency resolution that a normal install provides.

[jaraco]

I should say, though, the importance of this case is substantially diminished now that pip-run is no longer trying to fulfill the setup_requires case. At this point, it's a nice-to-have.

[pradyunsg]

At this point, it's a nice-to-have.

Given the lack of activity here, I'm gonna go ahead and close this off for now. While this would be nice-to-have; there's not enough interest in actually doing this (based on the lack of responses here and that there's alternatives) but if folks feel like that this is still relevant, please do say so. :)

[uranusjr]

Actually, is #10110 basically the same thing?

[pradyunsg]

Yup.

[jaraco]

This issue is still needed and still a blocker to remove the suboptimal hack in pip-run that attempts to do the same thing (jaraco/pip-run#51). Is there a reason the duplicate is left open while this report is closed? It makes it more difficult to track if the issue is a moving target.

[pradyunsg]

@jaraco please don't comment on locked issues in this issue tracker, which you're able to do due to being an admin on the pypa organisation.