ipython fails to install with pip-accel: pkg_resources.DistributionNotFound: gnureadline

[jquast]

preparing environment for pip-accel

$ export PS1='$ '
$ mkvirtualenv xxx
(xxx)$ python -V
Python 2.7.8
(xxx)$ pip install pip-accel
Downloading/unpacking pip-accel
  Downloading pip-accel-0.13.3.tar.gz
  Running setup.py (path:/Users/jquast/.virtualenvs/xxx/build/pip-accel/setup.py) egg_info for package pip-accel

Downloading/unpacking coloredlogs>=0.4.6 (from pip-accel)
  Downloading coloredlogs-0.8.tar.gz
  Running setup.py (path:/Users/jquast/.virtualenvs/xxx/build/coloredlogs/setup.py) egg_info for package coloredlogs

Downloading/unpacking humanfriendly>=1.6 (from pip-accel)
  Downloading humanfriendly-1.9.6.tar.gz
  Running setup.py (path:/Users/jquast/.virtualenvs/xxx/build/humanfriendly/setup.py) egg_info for package humanfriendly

Downloading/unpacking pip>=1.4,<1.5 (from pip-accel)
  Downloading pip-1.4.1.tar.gz (445kB): 445kB downloaded
  Running setup.py (path:/Users/jquast/.virtualenvs/xxx/build/pip/setup.py) egg_info for package pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.rst' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
Installing collected packages: pip-accel, coloredlogs, humanfriendly, pip
  Running setup.py install for pip-accel

    Installing pip-accel script to /Users/jquast/.virtualenvs/xxx/bin
  Running setup.py install for coloredlogs

    Installing ansi2html script to /Users/jquast/.virtualenvs/xxx/bin
  Running setup.py install for humanfriendly

  Found existing installation: pip 1.5.6
    Uninstalling pip:
      Successfully uninstalled pip
  Running setup.py install for pip

    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.rst' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
    Installing pip script to /Users/jquast/.virtualenvs/xxx/bin
    Installing pip-2.7 script to /Users/jquast/.virtualenvs/xxx/bin
Successfully installed pip-accel coloredlogs humanfriendly pip
Cleaning up...

installing ipython==2.3.0 with pip-accel (fails)

(xxx)$ pip-accel install ipython==2.3.0
2014-10-24 16:27:22 IO-PHX-L-468 pip_accel[40389] INFO Unpacking local source distributions ..
2014-10-24 16:27:22 IO-PHX-L-468 pip_accel[40389] INFO Executing command: pip install --download-cache=/Users/jquast/.pip/download-cache --find-links=file:///Users/jquast/.pip-accel/sources --build-directory=/var/folders/k3/ssq0mds55pn0lg4v0lr9cmp48g76w2/T/tmpStcttu --no-index ipython==2.3.0 --no-install
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking ipython==2.3.0
  Running setup.py egg_info for package ipython
                  readline: yes

Successfully downloaded ipython
2014-10-24 16:27:24 IO-PHX-L-468 pip_accel[40389] INFO Unpacked local source distributions in 1.08 second.
2014-10-24 16:27:24 IO-PHX-L-468 pip_accel[40389] INFO Installing from binary distributions ..
2014-10-24 16:27:25 IO-PHX-L-468 pip_accel[40389] INFO Finished installing all requirements in 1.04 second.
2014-10-24 16:27:25 IO-PHX-L-468 pip_accel[40389] INFO Done! Took 2.12 seconds to install 1 package.
(xxx)$ ipython -V
Traceback (most recent call last):
  File "/Users/jquast/.virtualenvs/xxx/bin/ipython", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/pkg_resources.py", line 2829, in <module>
    working_set = WorkingSet._build_master()
  File "/Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/pkg_resources.py", line 449, in _build_master
    ws.require(__requires__)
  File "/Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/pkg_resources.py", line 742, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/pkg_resources.py", line 639, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: gnureadline

installing ipython==2.3.0 with pip (succeeds)

(xxx)$ pip-accel uninstall ipython
Uninstalling ipython:
  /Users/jquast/.virtualenvs/xxx/bin/ipcluster
  /Users/jquast/.virtualenvs/xxx/bin/ipcluster2
  /Users/jquast/.virtualenvs/xxx/bin/ipcontroller
  /Users/jquast/.virtualenvs/xxx/bin/ipcontroller2
  /Users/jquast/.virtualenvs/xxx/bin/ipengine
  /Users/jquast/.virtualenvs/xxx/bin/ipengine2
  /Users/jquast/.virtualenvs/xxx/bin/iptest
  /Users/jquast/.virtualenvs/xxx/bin/iptest2
  /Users/jquast/.virtualenvs/xxx/bin/ipython
  /Users/jquast/.virtualenvs/xxx/bin/ipython2
  /Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/IPython
  /Users/jquast/.virtualenvs/xxx/lib/python2.7/site-packages/ipython-2.3.0-py2.7.egg-info
Proceed (y/n)? y
  Successfully uninstalled ipython
(xxx)$ pip install ipython==2.3.0
Downloading/unpacking ipython==2.3.0
  Downloading ipython-2.3.0.tar.gz (11.9MB): 11.9MB downloaded
  Running setup.py egg_info for package ipython
                  readline: yes

Installing collected packages: ipython
  Running setup.py install for ipython
                  readline: yes
    checking package data

    Installing ipengine2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing iptest script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipython2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipcluster2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipcluster script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipython script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipcontroller2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipcontroller script to /Users/jquast/.virtualenvs/xxx/bin
    Installing iptest2 script to /Users/jquast/.virtualenvs/xxx/bin
    Installing ipengine script to /Users/jquast/.virtualenvs/xxx/bin
Successfully installed ipython
Cleaning up...
(xxx)$ ipython -V
2.3.0

[jquast]

also, this is on OSX

(xxx)$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.10
BuildVersion:   14A389

[xolox]

Thanks for the bug report and sorry for not following up on this earlier. I did investigate a bit but haven't found anything conclusive. I suspect that this issue is the first time that I see that a user of pip-accel runs into "cache invalidation" issues. If that doesn't make sense to you, consider the following:

• pip-accel is basically pip + caching
• caching happens on several levels and one of those levels is the "binary cache" containing Python packages that use *.so files
• to use those *.so files your system needs to be in a state that is ABI compatible with your system at the time that the *.so files were compiled (this is an oversimplification but it explains the basic idea)

To actually verify that this is what happened in your case I need to be able to reproduce it. My wife is running Mac OS X as well so with some luck I'll be able to reproduce this issue on her system. To be continued :-)

[jquast]

Also reproduces on "Mavericks",

ProductName:    Mac OS X
ProductVersion: 10.9.5
BuildVersion:   13F34

e-mailed you an offer to provide ssh access to such machine.

[xolox]

Problem analysis

The SSH access helped to quickly narrow down the cause of this problem, so thanks for that :-). It's not at all related to cached binary distributions becoming invalid, instead it has to do with the (IMHO) strange behavior of the setup.py script of IPython:

1. When pip downloads and unpacks the IPython source distribution, the setup.py script of IPython reports a set of requirements in the install_requires keyword to the setup() function. Based on this list of requirements pip extends the requirement set and downloads missing source distributions. This process continues until all dependencies are satisfied.
2. When step 1 finishes pip-accel assumes that the requirement set is complete, i.e. all requirements have been satisfied. To me this seems like a reasonable assumption to make, because pip is also under the assumption that the dependency set is complete (this is the point where pip starts installing packages if it weren't for pip-accel).
3. What pip-accel adds in behavior compared to plain pip is that after step 1 pip uses python setup.py install to install packages while pip-accel instead runs python setup.py bdist_dumb (or python setup.py bdist) and caches the resulting binary distribution. This is what makes pip-accel faster. When the setup.py script of IPython sees bdist* in sys.argv (the command line arguments) it unconditionally adds gnureadline to install_requires. Because pip never runs any bdist* commands it never sees this dependency, so the dependency is not part of the dependency set. But it is written to the metadata included with the binary distribution, so the first time IPython is used after pip-accel installs it, setuptools notices the missing requirement and complains with:

Traceback (most recent call last):
  File "/tmp/issue-34-env/bin/ipython", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 2829, in <module>
    working_set = WorkingSet._build_master()
  File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 449, in _build_master
    ws.require(__requires__)
  File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 742, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/tmp/issue-34-env/lib/python2.7/site-packages/pkg_resources.py", line 639, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: gnureadline

Coming up with a solution

It might be possible for pip-accel to work around this issue but it would be a nasty hack, not an elegant and general solution. I think our best option is to convince the IPython developers to change their setup.py script. I just created a pull request to accomplish this, see ipython/ipython#7047.

[xolox]

@jquast: As a short term workaround you can add gnureadline to the pip command line or requirements file. It's a bit pragmatic but it will solve the immediate problem until ipython/ipython#7047 is merged (assuming the IPython developers are willing to merge it :-).

[xolox]

@jquast: My pull request ipython/ipython#7047 has been merged so the changes will eventually find their way to PyPI and until that time there is the (short term) workaround of adding gnureadline to the pip command line or requirements file. I believe this resolves the issue you reported.

[jquast]

Great job, I'm satisfied. Thank you @xolox