Implement PEP 376 REQUESTED

[sbidoul]

I'm opening this to discuss the opportunity to implement https://www.python.org/dev/peps/pep-0376/#requested. I'm willing to work on it, if there is agreement about its usefulness.

What's the problem this feature will solve?

It is currently not possible to discover, in the database of installed distribution, which distributions were installed by the user and which distributions were installed as dependencies.

Implementing the REQUESTED file would enable use cases such as smart uninstall (e.g. informing the users about possibly unneeded distributions) #5823

Describe the solution you'd like

Generate the REQUESTED file in pip install for top level requirements, i.e. provided by the user on the command line and packages provided via requirements files (-r option).

In a second stage, the freeze command could evolve a mechanism to output top level requirements on the one hand, and constraints for dependencies on the other hand, enabling a pip freeze | pip install loop that would preserve REQUESTEDness of distributions.

Alternative Solutions

Some higher level tools such as poetry implement this via their own requirement/lockfile format. pip-tools seems to achieve a similar goal with comments in the requirements file.

Having this standardized metadata piece implemented by pip would simplify the creation of such higher level tools, in addition to enabling additional pip use cases.

Note: flit, when used as an installer, implements the REQUESTED file.

[pfmoore]

I'm +1 in principle. If only because pip should be a good citizen and implement accepted standards.

I suspect there are a number of edge cases that need to be sorted out, but these could probably be handled as follow-ups.

One example, which I mention here so it's recorded somewhere:

1. User installs parent which installs child as a dependency.
2. User later wants to use child in its own right - how to say "mark child as requested, but don't change anything else?
3. Later still, user wants to drop the "requested" marker on child, and have it go back to being considered as "just" a dependency of parent.

Note that in the case of step (2), the user will likely actually want to uninstall child if parent is no longer installed, but only update the requested flag if parent is still present.

So I'm OK with implementing REQUESTED as defined in the PEP, I'm just not entirely sure if it will deliver as much value as users will actually expect.

[Shlokatadistance]

I think the above method is rather intuitive and can be helpful. A certain number of dependencies do greatly affect user experience and can be detrimental for development if they are accidentally uninstalled, because for a rather naive user, the process can be rather complicated.
However, I think we should consider the following cases:

1. Since we are also sorting which were installed as dependencies and requirments, we also need to develop another mechanism which would enable us to do so. Reading the data from terminal is not simplistic, so pip needs to incorporate this mechanism.
2. When working in virtual environments, many packages are installed as additions to one particular package. In certain cases, these are necessary to one another, so I am not sure how would we solve the issue, as mentioned by @pfmoore of updating the flags, because for one to function, the other is needed.

[pradyunsg]

User later wants to use child in its own right - how to say "mark child as requested, but don't change anything else?

pip install child?

user wants to drop the "requested" marker on child, and have it go back to being considered as "just" a dependency of parent

Maybe we need a pip mark command for handling this? Or... we can have a flag in pip uninstall to say "don't uninstall but mark as installed-as-dependency". I kinda like this second one, if we can come up with a good name.

[Shlokatadistance]

How about we simply make a new feature on this? as @pradyunsg said
pip install child
how about we add a flag -r to this, then the parent package, and then it installs a list of dependencies of the package?
I experienced this while installing dlib on my system, there were some other packages it needed first to function, and it was kind of aggrevating to install each part by part
This will give the user a better idea of the child packages or the parent packages existing, in the package he means to modify

[Shlokatadistance]

Following up from the previous, I also dont think it will too difficult to implement, we just need to automate the redirect of pip at this stage, and this will solve our purpose.

[uranusjr]

Right of the bat, I’ll probably be the most pessimistic voice in this thread. I am -1 for two reasons. I share Paul’s position that the value of implementing this feature is not obvious, but more importantly, I do not think REQUESTED is the best solution to the problem. It would be both much easier to have one single file describing the dependency graph than having each dist-info holding its own REQUESTED; the latter can easily become very inefficient to modify when you have multiple dependency layers.

Also, how do we deal with --target and --prefix? Or more generally, how do we deal with inconsistencies and broken dependecy links, in the world that not all tools respect and maintain this value? It is possible if pip maintains a central file describing the environment (whether it’s worthwhile is another problem), the sky comes crashing down if the user setup.py install a package at any point with REQUESTED.

[pfmoore]

I do not think REQUESTED is the best solution to the problem

This is closely tied to whether pip is a high-level or low-level tool. High-level tools like pipenv and poetry (and cargo, etc in other languages) work with a specification of "the environment we want", which is a high-level, easy to manage description of the overall final state.

Pip works at a much lower level - basically package-by-package (with packages triggering others as dependencies, but pip still treats each one mostly independently). I think REQUESTED is trying to solve the environment-management problem with a package-level solution.

I'd be fine with a proposal to remove REQUESTED from PEP 376 altogether. I'm also fine with pip implementing it just so that we comply with the standard. I don't honestly think that the other suggested benefits like "smart uninstall" will be any easier to achieve if we have this, though.

[sbidoul]

Pip works at a much lower level - basically package-by-package (with packages triggering others as dependencies, but pip still treats each one mostly independently).

REQUESTED is simply a way to mark individual packages as being manually installed. Similar to what apt-mark manual does in debian. As such, it remains a package-by-package information.

how do we deal with --target and --prefix

Wrt this very specific goal of marking manually installed package, --target and --prefix do not make any difference, unless I'm missing something.

[chrahunt]

I mentioned tracking what is installed directly in #7807 (comment). I think it's a good idea.

User later wants to use child in its own right - how to say "mark child as requested, but don't change anything else?

pip install child?

This seems reasonable to me, and is what I would expect. apt does something similar too - when you install something that's already installed it gets marked as manually installed.

3\. Later still, user wants to drop the "requested" marker on `child`, and have it go back to being considered as "just" a dependency of `parent`.

We don't necessarily have to decide this in the initial implementation - users can delete the REQUESTED file manually.