pip 1.3.1: pip no longer walks dirs in file:// indexes

[jenisys]

Installing from a local, simulated index-server directory seems to be broken in pip-1.3.1 (but was working with pip-1.2.1). I am normally using this feature for tox-based testing after preparing this functionality with pip install --download=$HOME/.pip/downloads -r requirements.txt and creating the simple directory structure w/ symlinks to the archives in the base directory.

I prepare the problem with the following steps:

$ mkdir -p $HOME/.pip/downloads
$ pip install --download=$HOME/.pip/downloads argparse
$ dir2pi  $HOME/.pip/downloads   #< Create the index-server like substructure with pip2pi.

When I run pip-1.3.1, it fails to find the package:

$ pip install --index=file:///Users/xxx/.pip/downloads/simple -U argparse
Downloading/unpacking argparse
  Cannot fetch index base URL file:///Users/xxx/.pip/downloads/simple/
  Could not find any downloads that satisfy the requirement argparse
No distributions at all found for argparse
Storing complete log in /Users/xxx/.pip/pip.log

When I run pip-1.2.1 on the same directory everything works fine:

$ pip install --index=file:///Users/xxx/.pip/downloads/simple -U argparse
Cannot fetch index base URL file:///Users/xxx/.pip/downloads/simple/
Unpacking /Users/xxx/.pip/downloads/simple/argparse/argparse-1.2.1.tar.gz
  Running setup.py egg_info for package argparse
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    ...
    no previously-included directories found matching 'env27'
Installing collected packages: argparse
  Found existing installation: argparse 1.2.1
    Uninstalling argparse:
      Successfully uninstalled argparse
  Running setup.py install for argparse
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    ...
    no previously-included directories found matching 'env27'
Successfully installed argparse
Cleaning up...

[qwcode]

I'm seeing this work for 1.3.1, and we have tests that work against local file:// indexes in our test suite.

1. not that it's the issue, but the parameter is technically --index-url. I guess optparse is allowing --index

2. I notice in both your examples, you're getting "Cannot fetch index base URL file:///Users/xxx/.pip/downloads/simple/", which tells me something is not quite right here. you shouldn't be getting that.

can you run the examples again with -v -v (i.e. doubly verbose)

[jenisys]

I assume the "Cannot fetch index base URL ..." message comes from the fact that no "index.html" file exists (but I haven't followed the call-chain in the debugger). But that was no problem in the past because the packages were still discovered.

MY RESUMEE from the debug dumps below:
pip-1.2.1 seems to look into the directory structure when no index.html file exists and finds the archive there.
pip-1.3.1 seems to bail out if no index.html file exists.

OK, here are the dumps.
pip-1.3.1 first:

$ pip install --index-url=file:///Users/xxx/.pip/downloads/simple -v -v -U argparse
Getting page file:///Users/xxx/.pip/downloads/simple/argparse/
 file: URL is directory, getting file:///Users/xxx/.pip/downloads/simple/argparse/index.html
Could not fetch URL file:///Users/xxx/.pip/downloads/simple/argparse/: <urlopen error [Errno 2] No such file or directory: '/Users/xxx/.pip/downloads/simple/argparse/index.html'>
Will skip URL file:///Users/xxx/.pip/downloads/simple/argparse/ when looking for download links for argparse in ./lib/python2.6/site-packages
Getting page file:///Users/xxx/.pip/downloads/simple/
 file: URL is directory, getting file:///Users/xxx/.pip/downloads/simple/index.html
Could not fetch URL file:///Users/xxx/.pip/downloads/simple/: <urlopen error [Errno 2] No such file or directory: '/Users/xxx/.pip/downloads/simple/index.html'>
Will skip URL file:///Users/xxx/.pip/downloads/simple/ when looking for download links for argparse in ./lib/python2.6/site-packages
Cannot fetch index base URL file:///Users/xxx/.pip/downloads/simple/
URLs to search for versions for argparse in ./lib/python2.6/site-packages:
* file:///Users/xxx/.pip/downloads/simple/argparse/
Getting page file:///Users/xxx/.pip/downloads/simple/argparse/
 file: URL is directory, getting file:///Users/xxx/.pip/downloads/simple/argparse/index.html
Could not fetch URL file:///Users/xxx/.pip/downloads/simple/argparse/: <urlopen error [Errno 2] No such file or directory: '/Users/xxx/.pip/downloads/simple/argparse/index.html'>
Will skip URL file:///Users/xxx/.pip/downloads/simple/argparse/ when looking for download links for argparse in ./lib/python2.6/site-packages
Could not find any downloads that satisfy the requirement argparse in ./lib/python2.6/site-packages
Downloading/unpacking argparse
No distributions at all found for argparse in ./lib/python2.6/site-packages
Exception information:
Traceback (most recent call last):
  File "/Users/xxx/zzz/.tox/py26/lib/python2.6/site-packages/pip-1.3.1-py2.6.egg/pip/basecommand.py", line 139, in main
    status = self.run(options, args)
  File "/Users/xxx/zzz/.tox/py26/lib/python2.6/site-packages/pip-1.3.1-py2.6.egg/pip/commands/install.py", line 266, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/Users/xxx/zzz/.tox/py26/lib/python2.6/site-packages/pip-1.3.1-py2.6.egg/pip/req.py", line 1025, in prepare_files
    raise not_found
DistributionNotFound: No distributions at all found for argparse in ./lib/python2.6/site-packages

Storing complete log in /Users/xxx/.pip/pip.log

Now the dump for pip-1.2.1:

$ pip install --index-url=file:///Users/xxx/.pip/downloads/simple -v -v -U argparse
Getting page file:///Users/xxx/.pip/downloads/simple/argparse
 file: URL is directory, getting file:///Users/xxx/.pip/downloads/simple/argparse/index.html
Could not fetch URL file:///Users/xxx/.pip/downloads/simple/argparse: <urlopen error [Errno 2] No such file or directory: '/Users/xxx/.pip/downloads/simple/argparse/index.html'>
Will skip URL file:///Users/xxx/.pip/downloads/simple/argparse when looking for download links for argparse in ./lib/python2.6/site-packages
Getting page file:///Users/xxx/.pip/downloads/simple/
 file: URL is directory, getting file:///Users/xxx/.pip/downloads/simple/index.html
Could not fetch URL file:///Users/xxx/.pip/downloads/simple/: <urlopen error [Errno 2] No such file or directory: '/Users/xxx/.pip/downloads/simple/index.html'>
Will skip URL file:///Users/xxx/.pip/downloads/simple/ when looking for download links for argparse in ./lib/python2.6/site-packages
Cannot fetch index base URL file:///Users/xxx/.pip/downloads/simple/
URLs to search for versions for argparse in ./lib/python2.6/site-packages:
Found link file:///Users/xxx/.pip/downloads/simple/argparse/argparse-1.2.1.tar.gz, version: 1.2.1
Local files found: /Users/xxx/.pip/downloads/simple/argparse/argparse-1.2.1.tar.gz
Using version 1.2.1 (newest of versions: 1.2.1, 1.2.1)
Unpacking /Users/xxx/.pip/downloads/simple/argparse/argparse-1.2.1.tar.gz
  Running setup.py egg_info for package argparse
    running egg_info
    creating pip-egg-info/argparse.egg-info
    writing pip-egg-info/argparse.egg-info/PKG-INFO
    ...
  Source in ./build/argparse has version 1.2.1, which satisfies requirement argparse from file:///Users/xxx/.pip/downloads/simple/argparse/argparse-1.2.1.tar.gz
Installing collected packages: argparse
  Found existing installation: argparse 1.2.1
    Uninstalling argparse:
      Removing file or directory /Users/xxx/zzz/.tox/py26/lib/python2.6/site-packages/argparse-1.2.1-py2.6.egg-info
      Removing file or directory /Users/xxx/zzz/.tox/py26/lib/python2.6/site-packages/argparse.py
      Removing file or directory /Users/xxx/zzz/.tox/py26/lib/python2.6/site-packages/argparse.pyc
      Successfully uninstalled argparse
  Running setup.py install for argparse
    Running command /Users/xxx/zzz/.tox/py26/bin/python2.6 -c "import setuptools;__file__='/Users/xxx/zzz/.tox/py26/build/argparse/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/76/91sp8l1d1xq42s1h9vgmtzq00000gn/T/pip-GcNifK-record/install-record.txt --single-version-externally-managed --install-headers /Users/xxx/zzz/.tox/py26/bin/../include/site/python2.6
    running install
    running build
    ...
    Copying argparse.egg-info to /Users/xxx/zzz/.tox/py26/lib/python2.6/site-packages/argparse-1.2.1-py2.6.egg-info
    running install_scripts
    writing list of installed files to '/var/folders/76/91sp8l1d1xq42s1h9vgmtzq00000gn/T/pip-GcNifK-record/install-record.txt'
Successfully installed argparse
Cleaning up...
  Removing temporary dir /Users/xxx/zzz/.tox/py26/build...

[qwcode]

that section of the code did get refactored.
1.3.1 is now doing the right thing IMO when it comes to file:// index-urls, based on the "spec" for the pypi simple index http://pythonhosted.org/distribute/easy_install.html#package-index-api
i.e. it shouldn't be doing directory listings even when it's a file:// index. it's still page and link based.

@pfmoore , your opinion here?

note that you can use --find-links against a flat local directory of archives w/o having to convert anything
see here: http://www.pip-installer.org/en/latest/cookbook.html#fast-local-installs

[pfmoore]

Without having looked at the details, I agree that 1.3.1 is showing correct behaviour - --index-url should point to something with the structure of a PyPI simple index (i.e., with an index.html) and --find-links should be used for a simple directory structure.

Having said that, it is a regression, and I guess I could be persuaded that making up an index.html from the directory structure if one isn't present should be added for backward compatibility. But only for that reason - correct use would still be as I described above.

[jenisys]

I think it depends on the URL schema. With the file:// URL schema, you have the possibility to use the directory listing as "virtual index.html" and fallback solution if no "index.html" is available. Some web servers even do just that (if this feature is not disabled) as file-browser like/FTP-like overview mode. For other URL schema, you need the "index.html" file to point you the way.

Note that I am not insisting on this solution because I have a work-around (which is not nice but I can live with it).
But others might have used the same functionality and will stumble over the same problem.

[pfmoore]

I'm -1 on it being based on the URL scheme, in general. That's the point of
having the two options - one (--find-links) is for a directory and one
(--index-url) is for identifying a simple index. That's a straightforward
distinction that is easy to document and understand. Doing a directory
scan when --index-url is given a file URL pointing to a directory which
doesn't have an index.html could be added for backward compatibility with
the 1.2 behaviour. However, that would (re-)introduce confusion between the
two options.

Maybe better (if anything at all is needed) would be to add a warning - if
the user specifies --index-url pointing to a directory with no index.html
present, issue a warning saying something like "--index-url does not
contain an index.html. Either add an explicit index, or use --find-links to
search the directory for distributions."

[qwcode]

I'm closing as an open issue for now.
If we find a lot of users are bumping into this, we could consider the extra warning.

[truetug]

It is very sad. You break my workflow.

I want to have some kind of caching local pypi, so...
I try to use pip2pi. But it don't follow dependencies and i do following
(project1) env/bin/pip install -r src/requirements.txt --index-url ../packages/simple --use-mirrors --download-cache ../packages

after that i rename your ugly http...%2Fpackagename-version.tar.gz files to packagename-version.tar.gz and call dir2pi on ../packages

So if i'll want to install requrements for project2 with partially different requirements pip may use previously downloaded packages.

--find-links works only with --no-index what is unacceptable.
Give me an alternative please.

[dholth]

@truetug if you can, try the latest pip from git and the http://www.pip-installer.org/en/latest/usage.html#pip-wheel command. It is a little different than caching sdists but it is designed to do more or less what you want.

[qwcode]

@truetug can't this cookbook solution work for you?
http://www.pip-installer.org/en/latest/cookbook.html#fast-local-installs