pyimport("numpy") difference between 0.3 and 0.4

[kersulis]

PyPlot.jl doesn't work for me in Julia 0.4, and I think I've traced the problem to a difference between PyCall's behavior on 0.3 vs 0.4.

I created a new virtual environment for Python 2.7 and installed numpy inside it. Then I ran the following in Julia 0.3:

julia> using PyCall
julia> pyimport("numpy")
PyObject <module 'numpy' from '/home/jkersulis/.virtualenvs/notebook/local/lib/python2.7/site-packages/numpy/__init__.pyc'>

Here's what happened when I repeated the commands in Julia 0.4:

julia> using PyCall
julia> pyimport("numpy")
ERROR: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('cannot import name scimath',)
  File "/home/jkersulis/.virtualenvs/notebook/local/lib/python2.7/site-packages/numpy/__init__.py", line 170, in <module>
    from . import add_newdocs
  File "/home/jkersulis/.virtualenvs/notebook/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/jkersulis/.virtualenvs/notebook/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 17, in <module>
    from . import scimath as emath

 in pyerr_check at /home/jkersulis/.julia/v0.4/PyCall/src/PyCall.jl:58
 in pyimport at /home/jkersulis/.julia/v0.4/PyCall/src/PyCall.jl:91

I'm not much of a pythonista, so I haven't been able to figure out the cause. I did notice that PyCall imports from the .pyc in 0.3 but tries to import the .py version in 0.4.

This issue may be a duplicate of #108.

[stevengj]

Are you using the same PyCall version on both? e.g. on 0.4 are you using PyCall master, but on 0.3 you are using the tagged version?

[kersulis]

Yes, Pkg.installed()["PyCall"] returns v"0.8.2" for both 0.3 and 0.4.

[stevengj]

This is weird; it is presumably some path-related thing, but I'm not sure how the Julia version could possibly affect the Python paths...

[kersulis]

I've been playing around in a virtualenv, and found that I can pyimport numpy 1.6 and 1.7, but not 1.8 or 1.9.

[kersulis]

I just tried

using Interact, PyCall
pyinitialize("python3")
using PyPlot

And everything works fine. All I really needed was PyPlot in 0.4, so I'm good now.

There must be a problem with my Python 2.7 installation. Honestly I'm not surprised. I have had Anaconda on my machine, I've used aptitude, and sometimes I use virtualenvs. It's a miracle python2 even runs on my command line anymore.

I hope I haven't wasted too much of your time. I'll reach out if I figure out the exact cause, in case someone else has a similar problem.

[stevengj]

No problem, I'm glad it works. (It's frustrating that it's so hard to initialize libpython robustly given the desired python executable.)

[kersulis]

Since my last comment here I have re-built Julia 0.3 and 0.4 from source, and I have virtual environments for Python 2 and Python 3. Regardless of which environment I activate or which version of Julia I run, I keep getting this error when I try to import numpy:

julia> Pkg.build("PyCall")
julia> using PyCall
julia> pyimport("numpy")
ERROR: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('cannot import name scimath',)
  File "/home/jkersulis/.envs/py2/local/lib/python2.7/site-packages/numpy/__init__.py", line 170, in <module>
    from . import add_newdocs
  File "/home/jkersulis/.envs/py2/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/jkersulis/.envs/py2/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 17, in <module>
    from . import scimath as emath

 in pyerr_check at /home/jkersulis/.julia/v0.3/PyCall/src/exception.jl:58
 in pyimport at /home/jkersulis/.julia/v0.3/PyCall/src/PyCall.jl:91

I can import other Python packages like nose, so there's something special about numpy here.

When I just run Python (in the same virtualenv), I can import numpy. Seeking to reproduce the error, I looked up line 17 in numpy/lib/__init__.py and tried to run it from the Python prompt (after cd-ing into the numpy/lib directory). Here's what I got:

>>> from . import scimath as emath
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Attempted relative import in non-package

I have looked around a bit, but I'm not sure this is really the problem PyCall is running into. This is just as far as I've searched.

[kersulis]

Quick update: I installed numpy 1.7 instead of 1.9 in my Python 2 environment, and I was able to import numpy in both Julia 0.3 and 0.4.

Then I installed numpy 1.7 in my Python 3 environment, and again I was able to import it in both Julia 0.3 and 0.4.

[stevengj]

@kersulis, can you try Pkg.checkout("PyPlot")?

[kersulis]

Sorry about the delay!

I used pip in my Python 2 & 3 envs to upgrade numpy and matplotlib back to 1.9.2 and 1.4.3 respectively. Then I ran

Pkg.checkout("PyPlot")
using PyPlot

Here's what I got:

Julia Version 0.3.12-pre+2

Warning: error initializing module PyPlot:
ErrorException("Failed to pyimport("matplotlib"): PyPlot will not work until you have a functioning matplotlib module.  PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('cannot import name scimath',)
  File "/home/jkersulis/.envs/py2/local/lib/python2.7/site-packages/matplotlib/__init__.py", line 180, in <module>
    from matplotlib.cbook import is_string_like
  File "/home/jkersulis/.envs/py2/local/lib/python2.7/site-packages/matplotlib/cbook.py", line 33, in <module>
    import numpy as np
  File "/home/jkersulis/.envs/py2/local/lib/python2.7/site-packages/numpy/__init__.py", line 170, in <module>
    from . import add_newdocs
  File "/home/jkersulis/.envs/py2/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/jkersulis/.envs/py2/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 17, in <module>
    from . import scimath as emath
")

Julia Version 0.4.0-dev+6561

INFO: Precompiling module PyPlot...
WARNING: could not import Base.help into PyCall
WARNING: error initializing module PyPlot:
ErrorException("Failed to pyimport("matplotlib"): PyPlot will not work until you have a functioning matplotlib module.  PyError (:PyImport_ImportModule) <class 'ImportError'>
ImportError("cannot import name 'scimath'",)
  File "/home/jkersulis/.envs/py3/lib/python3.4/site-packages/matplotlib/__init__.py", line 180, in <module>
    from matplotlib.cbook import is_string_like
  File "/home/jkersulis/.envs/py3/lib/python3.4/site-packages/matplotlib/cbook.py", line 33, in <module>
    import numpy as np
  File "/home/jkersulis/.envs/py3/lib/python3.4/site-packages/numpy/__init__.py", line 170, in <module>
    from . import add_newdocs
  File "/home/jkersulis/.envs/py3/lib/python3.4/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/jkersulis/.envs/py3/lib/python3.4/site-packages/numpy/lib/__init__.py", line 17, in <module>
    from . import scimath as emath
")

[kersulis]

I installed Anaconda to see if that would work. Turns out, it does. I can run ipython notebook, start a Julia kernel, run using PyPlot, and plot stuff.

Also, conda list shows that numpy 1.9.2 and matplotlib 1.4.3 are installed, so the problem isn't inherent in numpy. I don't see a need to reopen this issue; I would guess that my non-Anaconda Python installation is somehow messed up.