ccall fails to import some Python modules

[jmxpearson]

I am referencing JuliaPy/PyCall.jl#65 here because I think I have eliminated almost anything in PyCall itself that could be causing the problem. I'm not sure what else could be left except an error in ccall. I'm running 0.4-rc1.

In short, the following c code compiles and runs successfully on both my OSX and Ubuntu (14.10) machines (both systems running the Anaconda Python distribution):

#include <Python.h>

int
main(int argc, char *argv[])
{
    PyObject *pModule;
    char *pName = "numpy";

    Py_InitializeEx(0);

    pModule = PyImport_ImportModule(pName);

    if (pModule != NULL) {
        printf("Success!\n");
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", pName);
        return 1;
    }
    Py_Finalize();

    return 0;
}

while the following test code, called pyimport_test.jl, succeeds with the following on both systems

$ julia pyimport_test.jl json

but fails on Ubuntu with

$ julia pyimport_test.jl numpy

const stub = homedir() * "/anaconda"
const libname = @osx? "libpython2.7.dylib" : "libpython2.7.so"
const libpython = stub * "/lib/" * libname

################# from PyCall.jl #####################################
immutable PyObject_struct
    ob_refcnt::Int
    ob_type::Ptr{Void}
end

typealias PyPtr Ptr{PyObject_struct} # type for PythonObject* in ccall

macro pysym(func)
    :(($func, libpython))
end
################# from PyCall.jl #####################################

# now, do some ccalling to initialize

ccall((@pysym :Py_InitializeEx), Void, (Cint,), 0)

modname = ARGS[1] 
println("About to try loading library $modname")

modptr = ccall((@pysym :PyImport_ImportModule), PyPtr, (Ptr{Uint8},), 
    bytestring(modname))

if modptr == C_NULL
    println("Could not load library.")
else
    println("Success!")
end

ccall((@pysym :Py_Finalize), Void, ())

[ihnorton]

Can you dlopen the library directly?
On Sep 15, 2015 9:09 AM, "John Pearson" notifications@github.com wrote:

I am referencing JuliaPy/PyCall.jl#65
JuliaPy/PyCall.jl#65 here because I think I
have eliminated almost anything in PyCall itself that could be causing the
problem. I'm not sure what else could be left except an error in ccall.

In short, the following c code compiles and runs successfully on both my
OSX and Ubuntu (14.10) machines:

#include <Python.h>
intmain(int argc, char *argv[])
{
PyObject *pModule;
char *pName = "numpy";

Py_InitializeEx(0);

pModule = PyImport_ImportModule(pName);

if (pModule != NULL) {
    printf("Success!\n");
}
else {
    PyErr_Print();
    fprintf(stderr, "Failed to load \"%s\"\n", pName);
    return 1;
}
Py_Finalize();

return 0;

}

while the following test code, called pyimport_test.jl, succeeds with the
following on both systems

$ julia pyimport_test.jl json

but fails on Ubuntu with

$ julia pyimport_test.jl numpy

const stub = homedir() * "/anaconda"const libname = @osx? "libpython2.7.dylib" : "libpython2.7.so"const libpython = stub * "/lib/" * libname
################# from PyCall.jl #####################################immutable PyObject_struct
ob_refcnt::Int
ob_type::Ptr{Void}end
typealias PyPtr Ptr{PyObject_struct} # type for PythonObject* in ccall
macro pysym(func)
:(($func, libpython))end################# from PyCall.jl #####################################

now, do some ccalling to initialize

ccall((@pysym :Py_InitializeEx), Void, (Cint,), 0)

modname = ARGS[1] println("About to try loading library $modname")

modptr = ccall((@pysym :PyImport_ImportModule), PyPtr, (Ptr{Uint8},),
bytestring(modname))
if modptr == C_NULL
println("Could not load library.")else
println("Success!")end
ccall((@pysym :Py_Finalize), Void, ())

—
Reply to this email directly or view it on GitHub
#13140.

[jmxpearson]

Seems so:

julia> Libdl.dlopen("/home/pearson/anaconda/lib/libpython2.7.so")
Ptr{Void} @0x0000000003268c40

[ihnorton]

Sorry, bot-mode response... I would first check for a Python error after import, see http://stackoverflow.com/questions/6291545/how-do-i-find-out-why-importing-failed-with-pyimportmodule ... You could also try to print out sys.path from both the C-driven process and the Julia-driven process, and compare them.

[stevengj]

@jmxpearson, thanks for digging into this. However, I really doubt that it is a ccall issue per se; ccall seems to be successfully calling the functions in libpython, and getting the return value, it is just libpython that is not succeeding.

[jmxpearson]

Right, Python reports that it can't import scimath, which is an internal import in numpy. As I think I posted in the other issue, the error seems to happen every time Python uses some relative import syntax.

Julia:
"['/home/pearson/code/thunder/python', '/home/pearson/code', '/home/pearson/Dropbox/pyimport', '/home/pearson/anaconda/lib/python27.zip', '/home/pearson/anaconda/lib/python2.7', '/home/pearson/anaconda/lib/python2.7/plat-linux2', '/home/pearson/anaconda/lib/python2.7/lib-tk', '/home/pearson/anaconda/lib/python2.7/lib-old', '/home/pearson/anaconda/lib/python2.7/lib-dynload', '/home/pearson/anaconda/lib/python2.7/site-packages', '/home/pearson/anaconda/lib/python2.7/site-packages/PIL', '/home/pearson/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg', '/home/pearson/anaconda/lib/python2.7/site-packages/cryptography-0.9.3-py2.7-linux-x86_64.egg', '/home/pearson/anaconda/lib/python2.7/site-packages/setuptools-18.1-py2.7.egg']"

C:
['/home/pearson/code/thunder/python', '/home/pearson/code', '/home/pearson/Dropbox/pyimport', '/home/pearson/anaconda/lib/python2.7', '/home/pearson/anaconda/lib/python2.7/plat-x86_64-linux-gnu', '/home/pearson/anaconda/lib/python2.7/lib-tk', '/home/pearson/anaconda/lib/python2.7/lib-old', '/home/pearson/anaconda/lib/python2.7/lib-dynload', '/home/pearson/anaconda/lib/python2.7/site-packages', '/home/pearson/anaconda/lib/python2.7/site-packages/PIL', '/home/pearson/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg', '/home/pearson/anaconda/lib/python2.7/site-packages/cryptography-0.9.3-py2.7-linux-x86_64.egg', '/home/pearson/anaconda/lib/python2.7/site-packages/setuptools-18.1-py2.7.egg']

IPython:
['',
'/home/pearson/anaconda/bin',
'/home/pearson/code/thunder/python',
'/home/pearson/code',
'/home/pearson/Dropbox/pyimport',
'/home/pearson/anaconda/lib/python27.zip',
'/home/pearson/anaconda/lib/python2.7',
'/home/pearson/anaconda/lib/python2.7/plat-linux2',
'/home/pearson/anaconda/lib/python2.7/lib-tk',
'/home/pearson/anaconda/lib/python2.7/lib-old',
'/home/pearson/anaconda/lib/python2.7/lib-dynload',
'/home/pearson/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg',
'/home/pearson/anaconda/lib/python2.7/site-packages/setuptools-18.1-py2.7.egg',
'/home/pearson/anaconda/lib/python2.7/site-packages',
'/home/pearson/anaconda/lib/python2.7/site-packages/PIL',
'/home/pearson/anaconda/lib/python2.7/site-packages/cryptography-0.9.3-py2.7-linux-x86_64.egg',
'/home/pearson/anaconda/lib/python2.7/site-packages/IPython/extensions',
'/home/pearson/.ipython']

Update: actually, just noticed the differences here. Will look into that.

[jmxpearson]

@stevengj

I agree. I was/am just confused by the fact that the C calls to the library work just fine. Shouldn't libpython fail for the same reason there?

[stevengj]

Could it be https://bugzilla.redhat.com/show_bug.cgi?id=874874 ?

[stevengj]

@jmxpearson, if you add dlopen(libpython, RTLD_LAZY | RTLD_GLOBAL) before any of the ccalls (similar to cocotb/cocotb#146), does that fix things?

[jmxpearson]

@stevengj Unfortunately, no, but see the other thread. I think we can safely close this.