@pyimport numpy fails

[ufechner7]

Hello,
I use Ubuntu 12.04, 64 bit. I installed numpy 1.7.1 with pip.

Importing numpy fails:

julia> using PyCall

julia> @pyimport numpy
ERROR: PyError (PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('cannot import name scimath',)
File "/usr/local/lib/python2.7/dist-packages/numpy/init.py", line 153, in
from . import add_newdocs
File "/usr/local/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in
from numpy.lib import add_newdoc
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/init.py", line 17, in
from . import scimath as emath

in pyerr_check at /home/ufechner/.julia/PyCall/src/exception.jl:58
in pyimport at /home/ufechner/.julia/PyCall/src/PyCall.jl:85

Importing numpy from python works fine, importing other packages from julia
also works.

Any idea?

Uwe Fechner

[stevengj]

Do you have some kind of virtualenv setup, or multiple versions of Python?

[ufechner7]

No virtualenv, only one version of python on the harddrive. Any suggestions what I could try?

[stevengj]

Try

using PyCall
unshift!(PyVector(pyimport("sys")["path"]), "")
@pyimport numpy

to see if this is related to #48.

[ufechner7]

I tried this, but no change. I also tried numpy 1.8.0, same behaviour.

[ufechner7]

Hello,
the combination of numpy 1.6.1 and matplotlib 1.1 works now.
pip install -U numpy=="1.6.1"
pip install -U matplotlib=="1.1.1"

I still did not find the reason for this problem, because on other machines the newest version of numpy and matplotlib are working fine. Perhaps I should mention that this laptop uses a haswell CPU.

[ufechner7]

I tried using a clean virtual environment:

sudo pip install virtualenv
sudo apt-get install tk-dev

virtualenv /home/$USER/python-julia --no-site-packages

source /home/ufechner/python-julia/bin/activate
pip install numpy=="1.6.1"
pip install matplotlib=="1.1.1"

This is working, but any newer version of numpy does not work.

So yes, I have a workaround but no, the problem itself is not solved yet.

[arokem]

Just wanted to add that I am seeing the same with numpy 1.8. unshifting sys.path doesn't seem to help. Thanks for your work on PyCall!

[Jutho]

Same error here, on my desktop at work, which has the Canopy python distribution with latest version of all packages (numpy, scipy, matplotlib, ...). On my laptop I have installed python via homebrew and everything is working fine.

[stevengj]

PyCall doesn't work with Canopy (#42), so that's probably a separate issue.

[Jutho]

I noticed this afterwards; my apologies for polluting this issue. However, on my specific machine, the rest of Canopy's python seemed to do ok, it was only this exact same error "ImportError('cannot import name scimath',)" that was making PyPlot fail. I have removed Canopy and switched to Homebrew and everything works, as expected.

[stevengj]

If you google cannot import name scimath, you find lots of people encountering similar problems in random circumstances, but no systematic solution as far as I can find. Obviously something is missing from some path, but I'm not sure what.

[ufechner7]

A workaround for the google app engine is described here:
http://linuxonmac.wordpress.com/2014/03/22/google-app-engine-cannot-import-scimath/

I do not know how to apply this for an Ubuntu/ pip based environment, though.

[cberzan]

@stevengj The issue appears to be about circular imports. numpy.__init__ imports numpy.lib imports numpy.lib.scimath imports numpy.core.numeric (which appears to run numpy.__init__ again).

Here is how circular imports are supposed to work: http://stackoverflow.com/a/744403/744071

To demonstrate the issue: Make a virtualenv and install numpy==1.9.1. Then go to where it was installed, and add the following line near the top of numpy/__init__.py: print("numpy.__init__ running"). Add a similar line to the files numpy/lib/__init__.py and numpy/lib/scimath.py. Now, if you run this from python:

>>> import numpy
numpy.__init__ running
numpy.lib.__init__ running
numpy.lib.scimath running

That looks good. Now run the same from julia (v0.3.2) using PyCall (version 0.7.3):

julia> using PyCall
julia> @pyimport numpy
numpy.__init__ running
numpy.lib.__init__ running
numpy.lib.scimath running
numpy.__init__ running
numpy.lib.__init__ running
ERROR: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('cannot import name scimath',)
  File "/home/cberzan/.virtualenvs/captcha/local/lib/python2.7/site-packages/numpy/__init__.py", line 172, in <module>
    from . import add_newdocs
  File "/home/cberzan/.virtualenvs/captcha/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/cberzan/.virtualenvs/captcha/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 19, in <module>
    from . import scimath as emath

 in pyerr_check at /home/cberzan/.julia/v0.3/PyCall/src/exception.jl:58
 in pyimport at /home/cberzan/.julia/v0.3/PyCall/src/PyCall.jl:91

As you can see, the line numpy.__init__ running is printed twice, which shows that the circular imports aren't working properly.

I don't know enough about Python's import mechanism to debug this further, but I hope it leads you to a solution.

Update: Things get weirder. Consider this:

a.py:

print "a.py top"
import b
print "a.py bottom"

b.py:

print "b.py top"
import a
print "b.py bottom"

If I run this from the python shell, things work as expected:

>>> import a
a.py top
b.py top
b.py bottom
a.py bottom

But if I run it as a script, it does something else (I don't understand why):

$ python a.py
a.py top
b.py top
a.py top
a.py bottom
b.py bottom
a.py bottom

And Julia / PyCall behaves like the former:

$ PYTHONPATH="." julia
julia> using PyCall
julia> @pyimport a
a.py top
b.py top
b.py bottom
a.py bottom

So in this case Julia / PyCall seems to handle the circular imports properly.

[stevengj]

My first guess was that you have multiple versions of numpy etc. installed and it is getting confused about which one to import... but that doesn't explain the multiple print lines. I dunno...