pip search does not honor index-url in ~/.pip/pip.conf

[Ralnoc]

• Pip version: 9.0.1
• Python version: 2.7.12
• Operating System: All

Description:

Currently the pip search command does not honor the index-url or the configuration structure in the ~/.pip/pip.conf

pip search should be updated so that it uses the same logic for handling index-url. So I can define a global path in ~/.pip/pip.conf to apply to both of them or have a subsection of [install] and [search] where I can have a unique index-url for each action.

This is extremely important for anyone using artifactory for a pypi repo.

An optimally supported ~/.pip/pip.conf would be:

[index]
index-url = https://artifactory.example.com/artifactory/api/pypi/pypi-repos/simple

[search]
index-url = https://artifactory.example.com/artifactory/api/pypi/pypi-repos

What I've run:

When I run pip install:

Collecting okta-aws-auth
  1 location(s) to search for versions of okta-aws-auth:
  * https://artifactory.example.com/artifactory/api/pypi/pypi-repos/simple/okta-aws-auth/
  Getting page https://artifactory.example.com/artifactory/api/pypi/pypi-repos/simple/okta-aws-auth/
  Looking up "https://artifactory.example.com/artifactory/api/pypi/pypi-repos/simple/okta-aws-auth/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): artifactory.example.com
  "GET /artifactory/api/pypi/pypi-repos/simple/okta-aws-auth/ HTTP/1.1" 200 None
  Updating cache with response from "https://artifactory.example.com/artifactory/api/pypi/pypi-repos/simple/okta-aws-auth/"
  Analyzing links from page https://artifactory.example.com/artifactory/api/pypi/pypi-repos/simple/okta-aws-auth/
    Found link https://artifactory.example.com/artifactory/api/pypi/pypi-repos/packages/okta_aws_auth/1.0.4/okta_aws_auth-1.0.4-py2-none-any.whl#md5=77be86fc6ce86f3354e3afc02a3111d8 (from https://artifactory.example.com/artifactory/api/pypi/pypi-repos/simple/okta-aws-auth/), version: 1.0.4
  Using version 1.0.4 (newest of versions: 1.0.4)
  Looking up "https://artifactory.example.com/artifactory/api/pypi/pypi-repos/packages/okta_aws_auth/1.0.4/okta_aws_auth-1.0.4-py2-none-any.whl" in the cache
  No cache entry available
  "GET /artifactory/api/pypi/pypi-repos/packages/okta_aws_auth/1.0.4/okta_aws_auth-1.0.4-py2-none-any.whl HTTP/1.1" 200 9528
  Downloading https://artifactory.example.com/artifactory/api/pypi/pypi-repos/packages/okta_aws_auth/1.0.4/okta_aws_auth-1.0.4-py2-none-any.whl
  Downloading from URL https://artifactory.example.com/artifactory/api/pypi/pypi-repos/packages/okta_aws_auth/1.0.4/okta_aws_auth-1.0.4-py2-none-any.whl#md5=77be86fc6ce86f3354e3afc02a3111d8 (from https://artifactory.example.com/artifactory/api/pypi/pypi-repos/simple/okta-aws-auth/)
  Updating cache with response from "https://artifactory.example.com/artifactory/api/pypi/pypi-repos/packages/okta_aws_auth/1.0.4/okta_aws_auth-1.0.4-py2-none-any.whl"
  Caching due to etag
Requirement already satisfied: awscli in /usr/local/lib/python2.7/site-packages (from okta-aws-auth)
Requirement already satisfied: bs4 in /usr/local/lib/python2.7/site-packages (from okta-aws-auth)
Requirement already satisfied: boto3 in /usr/local/lib/python2.7/site-packages (from okta-aws-auth)
Requirement already satisfied: requests in /usr/local/lib/python2.7/site-packages (from okta-aws-auth)
Requirement already satisfied: pytz in /usr/local/lib/python2.7/site-packages (from okta-aws-auth)
Requirement already satisfied: colorama<=0.3.7,>=0.2.5 in /usr/local/lib/python2.7/site-packages (from awscli->okta-aws-auth)
Requirement already satisfied: botocore==1.4.67 in /usr/local/lib/python2.7/site-packages (from awscli->okta-aws-auth)
Requirement already satisfied: s3transfer<0.2.0,>=0.1.9 in /usr/local/lib/python2.7/site-packages (from awscli->okta-aws-auth)
Requirement already satisfied: rsa<=3.5.0,>=3.1.2 in /usr/local/lib/python2.7/site-packages (from awscli->okta-aws-auth)
Requirement already satisfied: docutils>=0.10 in /usr/local/lib/python2.7/site-packages (from awscli->okta-aws-auth)
Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python2.7/site-packages (from bs4->okta-aws-auth)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python2.7/site-packages (from boto3->okta-aws-auth)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python2.7/site-packages (from botocore==1.4.67->awscli->okta-aws-auth)
Requirement already satisfied: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in /usr/local/lib/python2.7/site-packages (from s3transfer<0.2.0,>=0.1.9->awscli->okta-aws-auth)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python2.7/site-packages (from rsa<=3.5.0,>=3.1.2->awscli->okta-aws-auth)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python2.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.4.67->awscli->okta-aws-auth)
Installing collected packages: okta-aws-auth

Successfully installed okta-aws-auth-1.0.4
Cleaning up...
Starting new HTTPS connection (1): pypi.python.org
"GET /pypi/pip/json HTTP/1.1" 200 72955

when I run pip search:

Starting new HTTPS connection (1): pypi.python.org
"POST /pypi HTTP/1.1" 200 None
django_saml2_auth (1.1.4)  - Django SAML2 Authentication Made Easy. Easily integrate with SAML2 SSO identity providers like Okta
loktar (31)                - ci
okta (0.0.3)               - Okta client APIs
oktaauth (0.2)             - Authenticates from the CLI
oktavia (0.5.0)            - High performance pure Python/JavaScript search engine
pyramid_okta (2016.0.38)   - Bindings for the pyramid webframework and the Okta authorization system.

[therealgambo]

Having a quick look at this, I believe this line is where this is happening.
https://github.com/pypa/pip/blob/master/pip/commands/search.py#L36

I agree that if you have defined a different index-url in your pip.conf, this should be the preferred default over pypi.

[xavfernandez]

Well, it's simply because the search API endpoint is completely uncorrelated with the PEP503-defined simple API endpoint.

One is specified with the --index-url option (and --extra-url) and defaults to https://pypi.python.org/simple/.
The other is specified with --index option and defaults to https://pypi.python.org/pypi.

Cf #589 for the corresponding issue.

[Ralnoc]

@xavfernandez Got it. So in the end, configuring the index in ~/.pip/pip.conf will force search to use the new repository. (I just tested and verified this)

• index-url - changes setting for install and all other actions that use the PEP503 compliant simple API endpoint
• index - changes the setting for search and any actions that uses the layout required by search.

That works, though it isn't overly intuitive. It seems that having them both use the index-url setting then overriding the global with the action specific settings so that you can have the unique URL for search versus install would be more intuitive from an implementation perspective.

If nothing else, adding a 'Configurations Directives' section to the pip user guide's configuration section that details the viable configuration directives and what they are for would help alleviate much of this confusion.

[reillysiemens]

It seems to me that having the install subcommand use --index-url and the search subcommand use --index leads to a rather inconsistent end-user experience. Furthermore, why doesn't search also support something like --extra-url?

I might be speaking in ignorance here and maybe this isn't the best place to voice this concern, but it seems like both searching and installation ought to have the ability to specify multiple locations and the order in which they're queried using a common configuration option. Sadly we don't all have the luxury of placing our packages at https://pypi.python.org/pypi. 😞

[dstufft]

This is essentially a duplicate of #395 since that is the reason why we have different names for these configurations. Sharing the name will only make this more confusing sadly :(

[shiroyuki]

@dstufft So, there is no fix to the issue that pip command ignores pip.conf?

[shiroyuki]

Just update: apparently, when I run pip with sudo, it will only try to find pip.conf for root, not the user that runs sudo. Sorry, @dstufft. It is false alarm. :)

[eromoe]

My dockerfile

FROM python:2

RUN echo '[global]' >> /etc/pip.conf
RUN echo 'index-url = https://mirrors.aliyun.com/pypi/simple' >> /etc/pip.conf


COPY . /mnt

WORKDIR /mnt

RUN rm /etc/pip.conf
RUN pip install -e basin-textminer/
RUN pip install -r annotator-server/requirements.txt

ENV PYTHONUNBUFFERED 0

ENV HOST '0.0.0.0'
ENV PORT 12000

CMD python annotator-server/run.py

Still access files.pythonhosted.org

ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Rea
d timed out.                                                                       
The command '/bin/sh -c pip install -e basin-textminer/' returned a non-zero code: 

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.