Enhancement: Add a general option for --repository (-r), and integrate with --index[-url]

[sanscore]

Repositories Proposal

Add a new CLI argument to specify a "repository"; -r, --repository <repo|url>.

Use Case

I would like to be able to specify a specific "repository" (index, if you prefer) to more easily limit where pip will search for packages. The idea is similar to setuputils/distutils ability to specify to which repository a package is uploaded.

One nice aspect of this idea is that it helps prevent information leakage when trying to install a package (or search for a package) intended for local/enterprise Pypi instances.

Another benefit, limiting the number of indexes searched to one should speed up the pip package discovery process. Though, I doubt that many users have more than two indexes.

Also, I want to be able to define shell aliases for pip; ex: pip-local. With the current setup, would have to set many aliases for each pip subcommand; alias pip-local-install='pip --trusted-host myhost.fqdn install --index-url http://myhost.fqdn/simple/, etc. (In writing this document, I've come to suspect that this desire could be achieved with inline Env Var, but I would argue that is not intuitive; ex: alias pip-local='PIP_TRUSTED_HOST=host.fqdn PIP_INDEX_URL=http://host.fqdn/simple pip'. I have yet to test it.)

Design Details

The ultimate goal of this enhancement is to elevate --index-url from pip install --index-url, however the incongruity with pip search --index complicates that possibility.

Another goal is to bring consistent terminology (index/repository) and, potentially, configuration compatibility with Distutils' .pypirc.

There's the obvious potential backwards compatibility issues which I have attempted to take into account. The design is to make -r repo supersede any invocations of --index or --index-url. Without -r <repo>, pip's behavior should be consistent.

Any notion of "default" repo has been conscientiously omitted from this write-up because my initial speculation is that it would confuse users and complicate matters too much. For instance, having a "default" would force the addition of a --no-repository option which (IMHO) is ugly and
unnecessary.

In the following sections I will describe pip -r repo interactions with other command line arguments only for convenience. The same behavior should be assumed regardless of how config settings were set; CLI argument, Env-var, or INI setting, etc.

pip invokation

pip invoked without -r

When pip is invoked without -r, then pip will behave identically as it does presently.

pip -r,--repository <repo>

• repo will identify either:

  • A [repo] defined in pip.conf, or
  • A url; i.e http://pypi.python.org/pypi, Index Discovery explained below.

• When pip is invoked with -r, then pip will exclusively search and install from that repository.

pip search --index Integration

pip -r repo search

• When pip is invoked with -r repo and without --index url, then the repository's url is assumed to be identical to the repository url and not be an issue.
• When pip is invoked with -r repo and with --index url, then the --index url is ignored.

pip -r <repo> install Integration

--index-url

• When pip is invoked with -r and without --index-url url, then the repository's url is modified to find the correct link. Presently, I'm aware of a few scenarios that can occur; Assume https://myrepo.mydomain/pypi-local is a working Pypi repo.

  1. Do nothing. Check the repo-url as is.
  2. Replace the end-of-path w/ /simple/; ex: https://myrepo.mydomain/simple/
  3. Append /simple/; ex: https://myrepo.mydomain/pypi-local/simple/
  4. (optional, Implementation-Detail) Discovered Index-Url Cache. After the proper Index-Url is discovered from above, add it to a configuration cache to short circuit future iterations of this process.

• When pip is invoked with -r repo and with --index-url url, then the index-url url is assumed to be correct ignored.

--extra-index-url

When pip is invoked with -r repo and with --extra-index-url url, then the extra indexes should be ignored.

--no-index

When pip in invoked with -r repo, TBD.

Comment: I have to plead ignorance on this argument as I have not yet needed to use it. I suspect that it's nonsensical to specify both pip -r repo and install --no-index, but I can't say so with authority.

But, my initial assumption would be to ignore -r repo.

pip -r <repo> list Integration

pip -r <repo> download Integration

pip -r <repo> wheel Integration

Same as: pip -r repo install

Configuration

pip.conf

Global

In the [global] section, like [distutils] in .pypirc, add an index-servers = [repos] option to specify custom repositories.

Potential Changes:

• Repos listed in index-servers could be appended to --extra-index-urls.

Custom Repos

Custom [repo] sections that allow one to define:

• repository: url, like.pypirc, later this is assumed to be the same url that will be used by pip search --index url.
• index-url: url, specifies the exact url for pip install --index-url.

Behavior Changes

• [global] behavior will be unaffected when -r repo is NOT present.
• With -r repo present:
  • [global] index-url will be ignored in favor of the [repo] index-url setting or index-url discovery process.
  • With -r repo, [global] extra-index-urls will be ignored.

Environment Variables

Add Env Vars for:

• PIP_REPO
  • When present, pip will assume pip -r ${PIP_REPO}.
• PIP_REPO_INDEX_URL
  • Dependent on presence of PIP_REPO.
  • With $PIP_REPO present, pip will assume:
    • --index-url ${PIP_REPO_INDEX_URL} for install, et.al.
    • --search ${PIP_REPO}
  • Without $PIP_REPO, $PIP_REPO_INDEX_URL is ignored.

External Integrations

.pypirc

Assuming the logic behind "pip.conf Configuration" is implemented, pip could gather repository information from .pypirc.

Related Issues

• search command uses --index, while install uses --index-url #589

[prokopst]

+1 to this proposal. Note I'm not involved in pip, I'm merely a reporter.

Yesterday with my colleague we hit the problem with --index-url vs. --index for search command. Docs say:

The names of the settings are derived from the long command line option, e.g. if you want to use a different package index (--index-url) and set the HTTP timeout (--default-timeout) to 60 seconds your config file would look like this:

[global]
timeout = 60
index-url = http://download.zope.org/ppix

So when global option index-url is set, search does not respect that, because index-url is actually an endpoint and search expects a different one. That's obviously confusing and it has been raised several times - #589, #3738, #4136. I think it's worth to invest into it and I would be glad to help.

A few notes to the proposal:

Another benefit, limiting the number of indexes searched to one should speed up the pip package discovery process. Though, I doubt that many users have more than two indexes.

I like the fact that it forces to use single repository (if I understand correctly). Multiple indexes make no sense to me, it should be responsibility of the repository to act as a proxy for other repository (including the official PyPI).

For example artifactory (universal repo for virtually anything, including python packages) does it nicely with virtual repositories and I've seen a custom solution doing the same. In the enterprise environment it improves resilience when an official repo is down for several hours.

When pip is invoked with -r repo and with --index url, then the --index url is ignored.
...
When pip is invoked with -r repo and with --index-url url, then the index-url url is ignored.
...
When pip is invoked with -r repo and with --extra-index-url url, then the extra indexes should be ignored.

I suggest to follow the principle of least astonishment and raise an error in such situations. Don't ignore, fail instead.

Moreover I would go further and actually deprecate both --index-url and --index and issue a warning. If different endpoints required explicitly, I would introduce options for search and for install endpoints.

PIP_REPO and PIP_REPO_INDEX_URL

I would follow the kiss principle and not allow PIP_REPO_INDEX_URL. Less code, less documentation, less confusion.

Assuming the logic behind "pip.conf Configuration" is implemented, pip could gather repository information from .pypirc.

Yes please! One config to store credentials would be great.

[nuess0r]

I'm in favour of this enhancement.

The two independent index and index-url stole today 1 h of my lifetime. The wording is not self explaining and the default use case for most users will be to do both actions on the same URL. Therefor most users have a unnecessary cluttered config file.