mkl error - Cannot load neither libmkl_mc3.so nor libmkl_def.so

[tomaszkacprzak]

As before, I am running Scientific Linux SL release 5.4 (Boron) with python 2.7.1 and mkl 11.1 046 and no root privileges (UCL Starlink).

Scons installs galsim fine.
Then when I want to import GalSim I get:

>>> import galsim
*** libmkl_mc3.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_mc3.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
*** libmkl_def.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_def.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
MKL FATAL ERROR: Cannot load neither libmkl_mc3.so nor libmkl_def.so
bash-3.2$

It seems that mkl libraries are missing, or were wrongly linked.
Is anyone familiar with this kind of situation?

[rmjarvis]

My guess is that you are using g++ with MKL?

I tried that on one of my systems and I couldn't figure out how to get it to work correctly. Basically got the same error you are getting. However, installing with CXX=icpc (for both galsim and tmv) did work. So I think the icpc linking step must be setting up some linking information in the libraries that g++ doesn't (by default), so at run time, it can't find the right libraries.

[tomaszkacprzak]

Yes, I was using g++ - that may not be the best idea indeed. I reinstalled TMV with ICPC and use it to install GalSim.
I don't know if the problem is gone, because I am getting familiar error

Unable to build a python loadable module with Boost.Python

I opened an issue on that:

#270

[rmjarvis]

So do I understand correctly that with icpc, you are still getting linking errors when you do:

scons -c
scons
scons install
python2.7 -c "import galsim"

[tomaszkacprzak]

Yes, that's right.

now linking looks like this:

bash-3.2$ python2.7 -c "import galsim"
*** libmkl_mc3.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_mc3.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
*** libmkl_def.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_def.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
MKL FATAL ERROR: Cannot load neither libmkl_mc3.so nor libmkl_def.so
bash-3.2$ locate libmkl_def
/opt/intel/Compiler/11.1/046/mkl/lib/32/libmkl_def.so
/opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_def.so
bash-3.2$ locate libmkl_mc 
/opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_mc.so
/opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_mc3.so
bash-3.2$ ldd ~/code/GalSim_install/lib/libgalsim.so
    linux-vdso.so.1 =>  (0x00007fffe592e000)
    /home/kacprzak/local/tmv/lib/libtmv_symband.so.0 (0x00002b0118214000)
    libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00002b01199a0000)
    /home/kacprzak/local/tmv/lib/libtmv.so.0 (0x00002b0119d26000)
    libmkl_intel_lp64.so => /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_intel_lp64.so (0x00002b011af0d000)
    libmkl_core.so => /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_core.so (0x00002b011b2f0000)
    libmkl_intel_thread.so => /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_intel_thread.so (0x00002b011b51c000)
    libguide.so => /opt/intel/Compiler/11.1/046/lib/intel64/libguide.so (0x00002b011c3cc000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b011c566000)
    libimf.so => /opt/intel/Compiler/11.1/046/lib/intel64/libimf.so (0x00002b011c781000)
    libsvml.so => /opt/intel/Compiler/11.1/046/lib/intel64/libsvml.so (0x00002b011cad3000)
    libm.so.6 => /lib64/libm.so.6 (0x00002b011cceb000)
    libiomp5.so => /opt/intel/Compiler/11.1/046/lib/intel64/libiomp5.so (0x00002b011cf6e000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b011d112000)
    libintlc.so.5 => /opt/intel/Compiler/11.1/046/lib/intel64/libintlc.so.5 (0x00002b011d413000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b011d550000)
    libc.so.6 => /lib64/libc.so.6 (0x00002b011d75e000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00002b011dab6000)
    /lib64/ld-linux-x86-64.so.2 (0x000000365a800000)
bash-3.2$ ldd ~/local/tmv/lib/libtmv.so
    linux-vdso.so.1 =>  (0x00007fffe8ffc000)
    libimf.so => /opt/intel/Compiler/11.1/046/lib/intel64/libimf.so (0x00002b86bb444000)
    libsvml.so => /opt/intel/Compiler/11.1/046/lib/intel64/libsvml.so (0x00002b86bb797000)
    libm.so.6 => /lib64/libm.so.6 (0x00002b86bb9d7000)
    libiomp5.so => /opt/intel/Compiler/11.1/046/lib/intel64/libiomp5.so (0x00002b86bbc5a000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b86bbdff000)
    libintlc.so.5 => /opt/intel/Compiler/11.1/046/lib/intel64/libintlc.so.5 (0x00002b86bc0ff000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b86bc23c000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b86bc44b000)
    libc.so.6 => /lib64/libc.so.6 (0x00002b86bc666000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00002b86bc9bd000)
    /lib64/ld-linux-x86-64.so.2 (0x000000365a800000)
bash-3.2$ ldd ~/local/lib/libboost_python.so.1.51.0
        linux-vdso.so.1 =>  (0x00007fff8deba000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00002b59fe6b0000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b59fe8b3000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002b59feace000)
        librt.so.1 => /lib64/librt.so.1 (0x00002b59fecd3000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b59feedc000)
        libm.so.6 => /lib64/libm.so.6 (0x00002b59ff1dc000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b59ff460000)
        libc.so.6 => /lib64/libc.so.6 (0x00002b59ff66e000)
        /lib64/ld-linux-x86-64.so.2 (0x000000365a800000)

Can we somehow traceback the import procedure in python?

[tomaszkacprzak]

I ran all possible TMV tests and all were passed successfully.

Some more debug info about that. it looks like TMV has problems loading libraries.. but why do the TMV tests work, but not in GalSim?

bash-3.2$ gdb --args python2.7 -c "import galsim"

...

(gdb) break exit
Function "exit" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (exit) pending.
(gdb) r
Starting program: /export/zupcx32/kacprzak/local/bin/python2.7 -c import\ galsim
[Thread debugging using libthread_db enabled]
[New Thread 0x2af66a9d6080 (LWP 8157)]


*** libmkl_mc3.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_mc3.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
*** libmkl_def.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_def.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
MKL FATAL ERROR: Cannot load neither libmkl_mc3.so nor libmkl_def.so

Breakpoint 1, 0x000000365ac33430 in exit () from /lib64/libc.so.6
(gdb) bt
#0  0x000000365ac33430 in exit () from /lib64/libc.so.6
#1  0x00002af672e4a324 in mkl_serv_load_dll () from /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_core.so
#2  0x00002af672e551e6 in mkl_blas_xdcopy () from /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_core.so
#3  0x00002af6731ef49d in mkl_blas_dcopy () from /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_intel_thread.so
#4  0x00002af672b00966 in dcopy_ () from /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_intel_lp64.so
#5  0x00002af672b1081b in cblas_dcopy () from /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_intel_lp64.so
#6  0x00002af670fa9951 in tmv::DoCopySameType<double> () from /home/kacprzak/local/tmv/lib/libtmv.so.0
#7  0x00002af66e2dda0a in tmv::Copy<double, double> (v1=@0x1, v2=Cannot access memory at address 0x0
) at /home/kacprzak/local/tmv/include/tmv/TMV_Vector.h:2079
#8  0x00002af66e2dd3f6 in tmv::GenVector<double>::assignToV (this=0x1, rhs=Cannot access memory at address 0x0
) at /home/kacprzak/local/tmv/include/tmv/TMV_Vector.h:368
#9  0x00002af66e2c8e28 in tmv::VectorView<double, 0>::operator= (this=0x1, v2=@0x0) at /home/kacprzak/local/tmv/include/tmv/TMV_Vector.h:856
#10 0x00002af66ecebd18 in tmv::DoCopy<double, double> () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#11 0x00002af66ecea4a7 in tmv::DoCopy1<double, double> () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#12 0x00002af66ecd86c9 in tmv::Copy<double, double> () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#13 0x00002af66ecd7b72 in tmv::GenBandMatrix<double>::assignToB () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#14 0x00002af66eeaff60 in tmv::BandMatrixView<double, 0>::operator= () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#15 0x00002af66f6495c7 in tmv::GenSymBandMatrix<double>::assignTosB () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#16 0x00002af66f640694 in tmv::GenSymBandMatrix<double>::assignToB () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#17 0x00002af66f640510 in virtual thunk to tmv::GenSymBandMatrix<double>::assignToB(tmv::BandMatrixView<double, 0>) const () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#18 0x00002af66f86f01f in tmv::BandMatrixView<double, 0>::operator= () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#19 0x00002af66f9b5b52 in tmv::BandLUDiv<double>::BandLUDiv () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#20 0x00002af66f6655ea in non-virtual thunk to tmv::GenSymBandMatrix<double>::setDiv() const () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#21 0x00002af6710bb022 in tmv::DivHelper<double>::doLDiv<double, double> () from /home/kacprzak/local/tmv/lib/libtmv.so.0
#22 0x00002af6742887f3 in tmv::QuotVsB<double, double, double>::assignToV (this=0x1, v0=Cannot access memory at address 0x0
) at /home/kacprzak/local/tmv/include/tmv/TMV_AuxVecComposite.h:215
#23 0x00002af66fc5af74 in tmv::Vector<double, 0>::operator= () from /home/kacprzak/local/tmv/lib/libtmv_symband.so.0
#24 0x00002af67428d26e in galsim::Table<double, double>::setupSpline (this=0x1) at src/Table.cpp:300
...

[rmjarvis]

Could you please post the full output from scons after doing scons -c? I want to see what some of the compilation lines look like. In particular the line that links the galsim libraries. It looks like it's not including the omp linkage for some reason. I have some possible workarounds if we can't figure out the problem, but I'd rather solve the problem for you.

[rmjarvis]

Actually, this might be harder than I thought. I googled the error you were having and found a thread that ends with this comment:

I remember now that I had the same problem recently - it is a
fundamental incompatibility between MKL and Python way of loading shared
libraries through dlopen. AFAIK, there is no solution to this problem,
except for using the static libraries.

I'm not sure what they mean about using static libraries though, since python usually can't handle static libraries.

[rmjarvis]

Another possibly relevant post from googling suggests adding the library mkl_mc. So maybe also try:

scons EXTRA_LIBS=mkl_mc

and see if that works?

[rmjarvis]

I think I fixed this on branch #261. Please try it out. Bob found a computer here at Penn with the same behavior, so I worked on it there, and I managed to fix it on that system, so hopefully it will work for you too.

It turns out that there are extra libraries necessary for loading MKL from a module rather than loading from an executable. In my case it was mkl_base. The post I mentioned above suggested mkl_mc for a different system. But Intel actually make this easier starting with MKL 10.3 (which I have on mine, but not sure if you do too). They made mkl_rt (for "run time" I presume) which is supposed to include all the necessary libraries for your system, so you don't have to try a bunch of different combinations anymore. Not that you had to, but the TMV installation process tries about 15 different combinations looking for a combination that works. In the next version, I'll have it try mkl_rt first, since that should usually work.

Anyway, in this branch, I added a test to the SConstruct file that checks if TMV can be loaded as a python module rather than just as an executable. If it fails, then I try adding mkl_rt (and then the other two options if that doesn't work). So if it doesn't work, it will find out much earlier in the process, rather than at run-time after compiling everything. But I think it shouldn't fail for you anymore, so please try it and let me know.

[tomaszkacprzak]

That looks great! I am having some problems re-creating the error now, my installation is failing again. As soon as I manage to compile again I will try it.

[rmjarvis]

If the installation is failing on this branch, please post the output. It might be due to the changes I made, since I now have it check for the kind of run-time error you were seeing during the installation rather than leave you to find it afterwards. So if what I did didn't fix it for you, then it should manifest as a failed installation now.

[tomaszkacprzak]

I managed to recreate the situation on the master branch, then I switched to #261 and got this:

bash-3.2$ git checkout remotes/origin/#261
Note: checking out 'remotes/origin/#261'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 36b2504... Use the full python path name in the shebang if the user specifies something with PYTHON
bash-3.2$ scons
scons: Reading SConscript files ...
SCons is version 2.2.0 using python version 2.4.3
Python is from /usr/include/python2.4
Using the following (non-default) scons options:
   CXX = icpc
   FLAGS = -mkl
   PYTHON = python2.7
   PREFIX = /home/kacprzak/code/GalSim_install
   PYPREFIX = /home/kacprzak/local/lib/python2.7/
   TMV_DIR = /home/kacprzak/local/tmv
   FFTW_DIR = /home/kacprzak/local/fftw3
   BOOST_DIR = /home/kacprzak/local
   EXTRA_LIB_PATH = /opt/intel/Compiler/11.1/046/lib/intel64:/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib:/opt/intel/Compiler/11.1/046/mkl/lib/em64t:/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Compiler/11.1/046/lib/intel64:/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib:/opt/intel/Compiler/11.1/046/mkl/lib/em64t:/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib
   IMPORT_PREFIX = False
   CACHE_LIB = False
These can be edited directly in the file gs_scons.conf.
Type scons -h for a full list of available options.
Using python =  /home/kacprzak/local/bin/python2.7
Using compiler: /opt/intel/Compiler/11.1/046/bin/intel64/icpc
compiler version: 11.1
Determined that a good number of jobs = 6
Checking for C++ library fftw3... yes
Checking for C++ header file boost/shared_ptr.hpp... yes
Checking for C++ header file TMV.h... yes
Using TMV_LINK file: /home/kacprzak/local/tmv/share/tmv/tmv-link
     -L/home/kacprzak/local/tmv/lib -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread -openmp
Checking for correct TMV linkage... (this may take a little while)
Checking for correct TMV linkage... yes
Checking if we can build against Python... yes
Checking if we can build module using TMV...
Unable to build a python loadable module that uses tmv

Please fix the above error(s) and re-run scons

bash-3.2$                                                                               

config_log:

...
icpc -o .sconf_temp/conftest_12_mod/check_tmv.so -openmp -shared .sconf_temp/conftest_12.os -Llib -L/home/kacprzak/local/lib -L/home/kacprzak/local/tmv/lib -L/home/kacprzak/local/fftw3/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -lmkl_rt -ltmv_symband -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread
ld: cannot find -lmkl_rt
.sconf_temp/conftest_13.cpp <-
  |
  |#include "Python.h"
  |#include "TMV_Sym.h"
  |
  |static void useTMV() {
  |    tmv::SymMatrix<double> S(10,4.);
  |    //tmv::Matrix<double> S(10,10,4.);
  |    tmv::Matrix<double> m(10,3,2.);
  |    tmv::Matrix<double> m2 = m / S;
  |}
  |
  |static PyObject* run(PyObject* self, PyObject* args)
  |{
  |    useTMV();
  |    return Py_BuildValue("i", 23);
  |}
  |
  |static PyMethodDef Methods[] = {
  |    {"run",  run, METH_VARARGS, "return 23"},
  |    {NULL, NULL, 0, NULL}
  |};
  |
  |PyMODINIT_FUNC initcheck_tmv(void)
  |{ Py_InitModule("check_tmv", Methods); }
  |
icpc -o .sconf_temp/conftest_13.os -c -mkl -fPIC -Iinclude -I/home/kacprzak/local/include -I/home/kacprzak/local/tmv/include -I/home/kacprzak/local/fftw3/include -I/home/kacprzak/local/include/python2.7 .sconf_temp/conftest_13.cpp
icpc -o .sconf_temp/conftest_13_mod/check_tmv.so -openmp -shared .sconf_temp/conftest_13.os -Llib -L/home/kacprzak/local/lib -L/home/kacprzak/local/tmv/lib -L/home/kacprzak/local/fftw3/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -lmkl_base -ltmv_symband -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread
ld: cannot find -lmkl_base
.sconf_temp/conftest_14.cpp <-
  |
  |#include "Python.h"
  |#include "TMV_Sym.h"
  |
  |static void useTMV() {
  |    tmv::SymMatrix<double> S(10,4.);
  |    //tmv::Matrix<double> S(10,10,4.);
  |    tmv::Matrix<double> m(10,3,2.);
  |    tmv::Matrix<double> m2 = m / S;
  |}
  |
  |static PyObject* run(PyObject* self, PyObject* args)
  |{
  |    useTMV();
  |    return Py_BuildValue("i", 23);
  |}
  |
  |static PyMethodDef Methods[] = {
  |    {"run",  run, METH_VARARGS, "return 23"},
  |    {NULL, NULL, 0, NULL}
  |};
  |
  |PyMODINIT_FUNC initcheck_tmv(void)
  |{ Py_InitModule("check_tmv", Methods); }
  |
...

Somehow I don't have mkl_rt on my system. locate mkl_rt and locate mkl_base both don't give me anything. My MKL version is 11.1/046, it seems that I should have it.

[rmjarvis]

Can you tell me what ls /opt/intel/Compiler/11.1/046/mkl/lib/em64t/ gives. Maybe I can figure out which library needs to get added...

[tomaszkacprzak]

Sure,

bash-3.2$ ls /opt/intel/Compiler/11.1/046/mkl/lib/em64t/
libmkl_avx.so                   libmkl_blacs_openmpi_lp64.a  libmkl_def.so         libmkl_intel_ilp64.so    libmkl_lapack95_lp64.a    libmkl_scalapack_ilp64.so         libmkl_solver_lp64_sequential.a
libmkl_blacs_ilp64.a            libmkl_blacs_sgimpt_ilp64.a  libmkl_gf_ilp64.a     libmkl_intel_lp64.a      libmkl_lapack.so          libmkl_scalapack_lp64.a           libmkl_vml_avx.so
libmkl_blacs_intelmpi_ilp64.a   libmkl_blacs_sgimpt_lp64.a   libmkl_gf_ilp64.so    libmkl_intel_lp64.so     libmkl_mc3.so             libmkl_scalapack_lp64.so          libmkl_vml_def.so
libmkl_blacs_intelmpi_ilp64.so  libmkl_blas95_ilp64.a        libmkl_gf_lp64.a      libmkl_intel_sp2dp.a     libmkl_mc.so              libmkl_sequential.a               libmkl_vml_mc2.so
libmkl_blacs_intelmpi_lp64.a    libmkl_blas95_lp64.a         libmkl_gf_lp64.so     libmkl_intel_sp2dp.so    libmkl_p4n.so             libmkl_sequential.so              libmkl_vml_mc3.so
libmkl_blacs_intelmpi_lp64.so   libmkl_cdft_core.a           libmkl_gnu_thread.a   libmkl_intel_thread.a    libmkl_pgi_thread.a       libmkl_solver_ilp64.a             libmkl_vml_mc.so
libmkl_blacs_lp64.a             libmkl_core.a                libmkl_gnu_thread.so  libmkl_intel_thread.so   libmkl_pgi_thread.so      libmkl_solver_ilp64_sequential.a  libmkl_vml_p4n.so
libmkl_blacs_openmpi_ilp64.a    libmkl_core.so               libmkl_intel_ilp64.a  libmkl_lapack95_ilp64.a  libmkl_scalapack_ilp64.a  libmkl_solver_lp64.a              locale

[rmjarvis]

OK, it looks like you have mkl_mc. This was the next library that should have been tried after mkl_base. But you cut off the config.log for that. What happened for that attempt? (i.e. What's in config.log after what you posted above?)

[tomaszkacprzak]

It looked fine, so I didn't include it. I see what is happening, indeed it's trying now mkl_mc.

icpc -o .sconf_temp/conftest_14.os -c -mkl -fPIC -Iinclude -I/home/kacprzak/local/include -I/home/kacprzak/local/tmv/include -I/home/kacprzak/local/fftw3/include -I/home/kacprzak/local/include/python2.7 .sconf_temp/conftest_14.cpp
icpc -o .sconf_temp/conftest_14_mod/check_tmv.so -openmp -shared .sconf_temp/conftest_14.os -Llib -L/home/kacprzak/local/lib -L/home/kacprzak/local/tmv/lib -L/home/kacprzak/local/fftw3/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -lmkl_mc -ltmv_symband -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread
.sconf_temp/conftest_15 <-
  |import sys; sys.path.append('.sconf_temp/conftest_14_mod'); import check_tmv; print check_tmv.run()
/home/kacprzak/local/bin/python2.7 < .sconf_temp/conftest_15 > .sconf_temp/conftest_15.out

[rmjarvis]

That last one is the relevant test that SCons says is failing. It looks like it's linking fine, but apparently the execution is failing. Can you please run

/home/kacprzak/local/bin/python2.7 < .sconf_temp/conftest_15

and post the output.

[tomaszkacprzak]

We get the familiar error.

bash-3.2$ /home/kacprzak/local/bin/python2.7 < .sconf_temp/conftest_15


*** libmkl_mc3.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_mc3.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
*** libmkl_def.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_def.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
MKL FATAL ERROR: Cannot load neither libmkl_mc3.so nor libmkl_def.so
bash-3.2$ cat .sconf_temp/conftest_15
import sys; sys.path.append('.sconf_temp/conftest_14_mod'); import check_tmv; print check_tmv.run()bash-3.2$
                                                                                                                ```

[rmjarvis]

OK, I think it's because I put the mkl_mc at the beginning rather than the end. Before I make the change to the SConstruct file, try this directly, and let's see if it works.

icpc -o .sconf_temp/conftest_14_mod/check_tmv.so -openmp -shared .sconf_temp/conftest_14.os -ltmv_symband -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread -lmkl_mc

Then again

/home/kacprzak/local/bin/python2.7 < .sconf_temp/conftest_15

If that doesn't work, try moving mkl_mc before guide and pthread. Sometimes the libraries are finicky in where they get placed.

[tomaszkacprzak]

I am afraid that it didn't help..

bash-3.2$ icpc -o .sconf_temp/conftest_14_mod/check_tmv.so -openmp -shared .sconf_temp/conftest_14.os -Llib -L/home/kacprzak/local/lib -L/home/kacprzak/local/tmv/lib -L/home/kacprzak/local/fftw3/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -ltmv_symband -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread -lmkl_mc

bash-3.2$ /home/kacprzak/local/bin/python2.7 < .sconf_temp/conftest_15
*** libmkl_mc3.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_mc3.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
*** libmkl_def.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_def.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
MKL FATAL ERROR: Cannot load neither libmkl_mc3.so nor libmkl_def.so

bash-3.2$ icpc -o .sconf_temp/conftest_14_mod/check_tmv.so -openmp -shared .sconf_temp/conftest_14.os -Llib -L/home/kacprzak/local/lib -L/home/kacprzak/local/tmv/lib -L/home/kacprzak/local/fftw3/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -L/opt/intel/Compiler/11.1/046/lib/intel64 -L/opt/intel/Compiler/11.1/046/ipp/em64t/sharedlib -L/opt/intel/Compiler/11.1/046/mkl/lib/em64t -L/opt/intel/Compiler/11.1/046/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib -ltmv_symband -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lmkl_mc -lguide -lpthread

bash-3.2$ /home/kacprzak/local/bin/python2.7 < .sconf_temp/conftest_15
*** libmkl_mc3.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_mc3.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
*** libmkl_def.so *** failed with error : /opt/intel/Compiler/11.1/046/mkl/lib/em64t/libmkl_def.so: undefined symbol: mkl_dft_commit_descriptor_s_c2c_md_omp
MKL FATAL ERROR: Cannot load neither libmkl_mc3.so nor libmkl_def.so
bash-3.2$ 

[rmjarvis]

Can you try it without all those -L flags? They shouldn't be necessary, and it's possible they are messing up icpc's normal search path.

Also, let's try linking mkl_mc3 rather than mkl_mc. Maybe that's the one your system requires.

[tomaszkacprzak]

I think it might be it. I added mkl_mc3. It didn't matter if I put it at the beginning, before guide and pthread or at the end.
All gave the same output and no error.

bash-3.2$ icpc -o .sconf_temp/conftest_14_mod/check_tmv.so -openmp -shared .sconf_temp/conftest_14.os -Llib -L/home/kacprzak/local/lib -L/home/kacprzak/local/tmv/lib -L/home/kacprzak/local/fftw3/lib -ltmv_symband -lmkl_mc3 -lfftw3 -ltmv -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lguide -lpthread
bash-3.2$ /home/kacprzak/local/bin/python2.7 < .sconf_temp/conftest_15
23
``

[rmjarvis]

Great! I added that as a tested library, so try the current '#261' branch and see if it works for you. It should, but let's just make sure before we merge the changes to master.

[tomaszkacprzak]

'import galsim' works! Thanks very much Mike!

[rmjarvis]

Great. I merged the changes into master. Closing this issue.