Install repo or path with extras without knowing project name

[jaraco]

I'd like to be able to install a project from a file system path or URL and install extras for it without having to know the project name.

It's possible to install without knowing the project name if extras aren't needed:

pip install git+https://github.com/jaraco/pytest-perf

And it's possible to install with extras if the project name is known:

pip install 'pytest-perf[testing]@git+https://github.com/jaraco/pytest-perf'

But if one wants to install a project with its extras without knowing its name, there appears to be no syntax for that.

Perhaps pip could support something like

pip install '[testing]@git+https://github.com/jaraco/pytest-perf'

Or allow a special sentinel to stand in for the actual project name.

pip install '*[testing]@git+https://github.com/jaraco/pytest-perf'

[uranusjr]

Personally I am -1. Pip supports bare URLs only because it had it before PEP 508 was introduced. Now we have PEP 508, pip should try to reduce non-standard syntax instead of adding more. The bare URL format is still useful since it is convenient and pretty trivial to parse, but we should avoid doing anything more complicated than that.

[pfmoore]

I agree with @uranusjr here. If there's sufficient justification for this, then it should be proposed as an extension to PEP 508. But if it is, I'd be against it there as well - having to deal with projects that don't have proper names is a source of difficult to handle edge cases, and I'd much rather we were moving towards a principle of "every project must have a well-defined, easily determined name" and this suggestion seems to be moving in the opposite direction.

[jaraco]

In the pytest-perf project, I'm trying to install the "project under test". I don't know the project name in advance. It's whatever repo is checked out at .. I'd like a way to install the git repo (at a given rev) at . with extras, and I'd like to be able to do that without knowing the distribution name in advance.

I could, of course, in order to determine the name, copy the pip logic that clones the repo and checks it out to the desired revision, then run build on it, then load the metadata to determine the name, and then delete that clone, but that seems grossly inefficient, especially considering that pip is about to do the same exact operation. Both cloning and building metadata are very expensive operations in many cases.

Even the current implementation, which installs from a remote URL, in order to produce a PEP 508 URL with extras, infers the project name from the stem of the remote URL. That approach is likely to work, but I'm sure there are cases where it does not.

I guess in this particular use-case, one could determine the distribution name of the project under test using the local checkout, but that's still an expensive operation and troublesome. I can imagine a scenario where the user doesn't want to use the local checkout, but would want to use some other reference repository or branch.

Since it's possible to install a file system path without specifying the project name (i.e. .[extras]), I'd really like to see an equivalent way to install a URL without specifying the project name. Is there a reason behind this disparity (why does the distribution name need to be known in advance for a URL but not for a path)?

having to deal with projects that don't have proper names is a source of difficult to handle edge cases

I'm not sure how this concern applies here. I want to install a project with a proper name. I just don't know the name in advance.

Now we have PEP 508, pip should try to reduce non-standard syntax instead of adding more. The bare URL format is still useful since it is convenient and pretty trivial to parse, but we should avoid doing anything more complicated than that.

The syntax I'm suggesting is based on PEP 508, specifically the non-bare URL form, but providing a way not to have to over-specify the name. I agree that PEP 508 maybe should be amended. I raise the issue here to express the use-case that I'd like to see supported.

[pfmoore]

Is there a reason behind this disparity (why does the distribution name need to be known in advance for a URL but not for a path)?

Probably a combination of legacy reasons (the non-URL forms aren't standardised and just sort of grew over time) plus performance (it's not as costly to get the name from a local directory - setup.py egg_info is assumed to be cheap, whereas a general URL involves a download, potentially a VCS clone, etc).

However, the more important point is that newer developments (like the new resolver) are heavily impacted by the cost of determining the name (and version) of a project, so while we've had to maintain existing cases where "unnamed" projects are supported, we'd rather avoid adding more such cases.

I'm not sure how this concern applies here.

My concern revolves around the resolver "candidate" implementation, where candidates need to have a name, and lazily resolving the name is potentially problematic (especially in conjunction with extras, which are a whole additional world of pain 🙂). But it is only a concern, I'm not saying there's an actual problem here, just that someone needs to verify that there isn't before we decide to accept a feature like this...

I agree that PEP 508 maybe should be amended. I raise the issue here to express the use-case that I'd like to see supported.

OK, I guess that if PEP 508 gets revised, then pip will support that change. But I'm saying that pip shouldn't implement this independently of such a standard. Is that sufficient? If so, then I'd argue that you'd be better making this case on Discourse as part of that proposal.