Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.

[kalefranz]

From @jakirkham on February 10, 2016 2:20

Seeing this on Travis CI (Linux) and in Docker containers during install. ( https://travis-ci.org/jakirkham/nanshe/builds/108187039#L749 )

Copied from original issue: conda/conda#2048

[kalefranz]

From @bilderbuchi on February 10, 2016 12:22

I just encountered a similar/equivalent one (on Windows): Intel MKL FATAL ERROR: Cannot load mkl_avx.dll or mkl_def.dll.

[kalefranz]

From @msarahan on February 10, 2016 13:50

Pinging @ilanschnell

[kalefranz]

From @bilderbuchi on February 10, 2016 15:11

Here is another instance of this error, apparently. It provides a repro procedure (which doesn't repro on my side, though), and hints that maybe scipy 0.17 is the culprit, and downgrading to 0.15 could circumvent it?

[kalefranz]

From @jakirkham on February 10, 2016 15:34

FWIW, this problem does not occur on Mac. So, it is only Linux and Windows.

[kalefranz]

From @jakirkham on February 10, 2016 15:39

It appears like the libraries are in the packages.

@bilderbuchi, are using Windows natively or are using it in a VM? If the latter what are you using for virtualization? If the former, do you know what architecture you have?

[kalefranz]

From @bilderbuchi on February 11, 2016 6:59

natively, 64bit, Windows 8.1.

[kalefranz]

From @desilinguist on February 11, 2016 18:15

Yes, I am having the same issue on my RHEL box. Works fine on my Mac. Seems like scipy is the culprit but all the paths seem so the set correctly when I run show_config() and all the libraries seem to be there under $PREFIX/lib.

[kalefranz]

From @jakirkham on February 11, 2016 18:17

Sorry, by architecture, @bilderbuchi, I was meaning what kind of processor are you using?

[kalefranz]

From @bilderbuchi on February 12, 2016 7:45

An Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz

[kalefranz]

Does this issue still exist? Regardless, going to kick it over to anaconda-issues. Pretty sure it's packaging-related.

[jakirkham]

This was a problem on Linux. I spun up a docker container and installed numpy, scipy, and mkl. I tried importing numpy and scipy, which did not fail indicating this is probably fixed. Also, I tried using numpy.dot, which uses the BLAS if available, and imported/used a few functions from scipy.linalg.blas, which only use the BLAS. These seemed to work and give correct results. So, I believe this is fixed and can be closed.

[kalefranz]

Cool!

[KelSolaar]

I have some Travis builds failing with the same error: https://travis-ci.org/colour-science/colour/builds/118175098

[KelSolaar]

@kalefranz: Is it possible to re-open that issue please, it is not fixed or there are no clear step-by-step info to correct the problem.

[jakirkham]

Why do you need to pin the patch number for NumPy in your builds?

[KelSolaar]

@jakirkham: It is a good point, back then I think we had specific requirements (especially for Scipy). I will try a build without specifying any version. Thanks!

[KelSolaar]

Seems like it was that, for my curiosity any specific reason why this problem happened? Cheers,

[jakirkham]

There were some issues with the first MKL package released as you have seen. In this case, missing libraries on Linux. I expect (though have not checked) that NumPy 1.10.1 was pinned to a certain version of the MKL package. However, this was fixed in a later version of the MKL package and I believe the next NumPy package (think 1.10.2) changed its pinning to this new version.

[KelSolaar]

Excellent, makes sense! Thanks for the help, appreciated 👍

[maximz]

EDIT: My mistake, the wipe was not as complete as I wanted. With a truly fresh install, this problem is gone.

Leaving the comment below because it shows how to reproduce the original error.

---

I don't think this is fixed. I am experiencing this issue from a wipe and fresh download and install of Anaconda today. Again, the particular issue is: Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.

Here is a way to reproduce, thanks to BVLC/caffe#3884:

> python -c 'import sklearn.linear_model.tests.test_randomized_l1'
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.

According to that thread, the solution is to not use MKL or to do the following:

> LD_PRELOAD=~/anaconda2/lib/libmkl_core.so python -c 'import sklearn.linear_model.tests.test_randomized_l1'

However, with LD_PRELOAD set that way, I get this further issue when running my code base (which used to work fine): python: symbol lookup error: /home/username/anaconda2/lib/libmkl_core.so: undefined symbol: mkl_blas_dtrsm

What gives?

> conda list
...
mkl                       11.3.1                        0
mkl-service               1.1.2                    py27_0
...
numpy                     1.10.4                   py27_1
...
scikit-learn              0.17.1              np110py27_0
...
scipy                     0.17.0              np110py27_2

[adamdivak]

I had the same problem, despite using the latest available packages. Turns out the solution was easier than I thought: for whatever reason Anaconda installed the MKL-enabled versions of the numpy/scipy stack, but did not actually install mkl itself. I have seen this when building Docker images based on the Jupyter minimal notebook stack.

A simple conda install --yes mkl mkl-service solved it for me.

[ibmua]

Updating via
conda install mkl
solved it for me. It seems to have updated several modules including mkl, mkl-service and numpy.

[smithsp]

This is my first post to anaconda-issues, so suggested alternative actions to posting here are welcome.

On my system, I have the following packages installed:

: conda list -n omfit
# packages in environment at /p/omfit/envs/omfit:
#
anaconda-client           1.5.1                    py27_0    defaults
astropy                   1.2.1               np111py27_0    defaults
async                     0.6.1                    py27_0    https://conda.binstar.org/minadyn/linux-64/async-0.6.1-py27_0.tar.bz2
backports                 1.0                      py27_0    defaults
backports.shutil-get-terminal-size 1.0.0                     <pip>
binstar                   0.12                          2    http://repo.continuum.io/pkgs/free/linux-64/binstar-0.12-2.tar.bz2
boto                      2.38.0                    <pip>
boto3                     1.1.2                     <pip>
botocore                  1.1.11                    <pip>
cairo                     1.12.18                       6    defaults
clyent                    1.2.2                    py27_0    defaults
configobj                 5.0.6                    py27_0    http://repo.continuum.io/pkgs/free/linux-64/configobj-5.0.6-py27_0.tar.bz2
curl                      7.49.0                        1    defaults
cvxopt                    1.1.8                    py27_3    defaults
cycler                    0.10.0                   py27_0    defaults
cython                    0.24.1                   py27_0    defaults
dateutil                  2.4.1                    py27_0    http://repo.continuum.io/pkgs/free/linux-64/dateutil-2.4.1-py27_0.tar.bz2
decorator                 4.0.10                   py27_0    defaults
docutils                  0.12                      <pip>
emcee                     2.1.0                     <pip>
fontconfig                2.11.1                        6    defaults
fortranformat             0.2.5                    py27_0    https://conda.binstar.org/auto/linux-64/fortranformat-0.2.5-py27_0.tar.bz2
freetype                  2.5.5                         1    defaults
futures                   2.2.0                     <pip>
get_terminal_size         1.0.0                    py27_0    defaults
gitdb                     0.5.4                    py27_0    https://conda.binstar.org/minadyn/linux-64/gitdb-0.5.4-py27_0.tar.bz2
gitpython                 0.3.2.RC1                py27_0    https://conda.binstar.org/minadyn/linux-64/gitpython-0.3.2.RC1-py27_0.tar.bz2
glib                      2.43.0                        1    defaults
gptools                   0.2.2                     <pip>
harfbuzz                  0.9.39                        1    defaults
hdf4                      4.2.12                        0    defaults
hdf5                      1.8.17                        1    defaults
ipython                   5.0.0                    py27_0    defaults
ipython-genutils          0.1.0                     <pip>
ipython_genutils          0.1.0                    py27_0    http://repo.continuum.io/pkgs/free/linux-64/ipython_genutils-0.1.0-py27_0.tar.bz2
jbig                      2.1                           0    defaults
jmespath                  0.7.1                     <pip>
jpeg                      8d                            1    defaults
lcms                      1.19                          0    http://repo.continuum.io/pkgs/free/linux-64/lcms-1.19-0.tar.bz2
libffi                    3.2.1                         0    defaults
libgfortran               3.0.0                         1    defaults
libnetcdf                 4.4.1                         0    defaults
libpng                    1.6.22                        0    defaults
libtiff                   4.0.6                         2    defaults
libxml2                   2.9.2                         0    http://repo.continuum.io/pkgs/free/linux-64/libxml2-2.9.2-0.tar.bz2
lmfit                     0.9.2                     <pip>
matplotlib                1.5.1               np111py27_0    defaults
mdsplus-6.1               82                        <pip>
mkl                       11.3.3                        0    defaults
mkl-service               1.1.2                    py27_2    defaults
mpmath                    0.19                      <pip>
netcdf4                   1.2.4               np111py27_0    defaults
numeric                   24.2                      <pip>
numpy                     1.11.1                   py27_0    defaults
openblas                  0.2.14                        4    defaults
openssl                   1.0.2h                        1    defaults
pandas                    0.17.1                    <pip>
pango                     1.39.0                        1    defaults
path.py                   8.2.1                    py27_0    defaults
pathlib2                  2.1.0                    py27_0    defaults
pexpect                   4.0.1                    py27_0    defaults
pickleshare               0.7.3                    py27_0    defaults
pidly                     0.2.6                     <pip>
pil                       1.1.7                    py27_2    defaults
pillow                    3.3.0                    py27_0    defaults
pip                       8.1.2                    py27_0    defaults
pixman                    0.32.6                        0    defaults
prompt-toolkit            1.0.3                     <pip>
prompt_toolkit            1.0.3                    py27_0    defaults
ptyprocess                0.5.1                    py27_0    defaults
py2cairo                  1.10.0                   py27_2    http://repo.continuum.io/pkgs/free/linux-64/py2cairo-1.10.0-py27_2.tar.bz2
pycairo                   1.10.0                   py27_0    defaults
pygments                  2.1.3                    py27_0    defaults
pyodbc                    3.0.10                   py27_0    defaults
pyparsing                 2.1.4                    py27_0    defaults
pyqt                      4.11.4                   py27_4    defaults
python                    2.7.12                        1    defaults
python-dateutil           2.5.3                    py27_0    defaults
pytz                      2016.6.1                 py27_0    defaults
pyyaml                    3.11                     py27_4    defaults
qt                        4.8.7                         4    defaults
readline                  6.2                           2    http://repo.continuum.io/pkgs/free/linux-64/readline-6.2-2.tar.bz2
requests                  2.10.0                   py27_0    defaults
scipy                     0.18.0              np111py27_0    defaults
setuptools                25.1.6                   py27_0    defaults
simplegeneric             0.8.1                    py27_1    defaults
sip                       4.18                     py27_0    defaults
six                       1.10.0                   py27_0    http://repo.continuum.io/pkgs/free/linux-64/six-1.10.0-py27_0.tar.bz2
smmap                     0.8.2                    py27_0    https://conda.binstar.org/minadyn/linux-64/smmap-0.8.2-py27_0.tar.bz2
sqlite                    3.13.0                        0    defaults
system                    5.8                           2    http://repo.continuum.io/pkgs/free/linux-64/system-5.8-2.tar.bz2
tk                        8.5.18                        0    http://repo.continuum.io/pkgs/free/linux-64/tk-8.5.18-0.tar.bz2
traitlets                 4.2.2                    py27_0    defaults
triangle                  2015.12.14                <pip>
triangle-plot             0.3.0                     <pip>
uncertainties             2.4.4                    py27_0    https://conda.binstar.org/auto/linux-64/uncertainties-2.4.4-py27_0.tar.bz2
unixodbc                  2.3.4                         0    defaults
wcwidth                   0.1.7                    py27_0    defaults
wheel                     0.29.0                   py27_0    defaults
xarray                    0.7.0                     <pip>
xz                        5.2.2                         0    defaults
yaml                      0.1.6                         0    http://repo.continuum.io/pkgs/free/linux-64/yaml-0.1.6-0.tar.bz2
zlib                      1.2.8                         3    defaults

That is after I did conda update --all -n omfit and conda install -f -n omfit mkl mkl-service. However, I am still seeing the following problem:

: python
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:42:40) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from cvxopt import solvers
>>> from cvxopt import matrix
>>> 
>>> Q = 2*matrix([ [2., .5],[.5, 1.] ] )
Intel MKL FATAL ERROR: Cannot load libmkl_def.so.

Is there any workaround or suggested action?

[jskDr]

I got also the following error:
python: symbol lookup error: /home/username/anaconda2/lib/libmkl_core.so: undefined symbol: mkl_blas_dtrsm

[jskDr]

I finally solved this problem using two steps for my deep learning applications with Keras/Theano. Notice that I am using Ubuntu 14.04.

First, I removed mkl with the following two commands.
$ [sudo] conda install nomkl numpy scipy scikit-learn numexpr
$ [sudo] conda remove mkl mkl-service

Although mkl is removed from my anaconda python 3, LinearRegression fit in skearn still makes error related to scipy. During I am searching Web. Some brilliants said to remove python-scipy but install pip based way. So, I applied this solution to my case. I removed conda scipy and install pip sciy py as follows:
$ [sudo] conda remove scipy
$ [sudo] pip install scipy
while scipy was removed from conda, it also took his followers such as sklearn. Hence, I installed sklearn again using the pip tool such as.
$ [sudo] pip install sklearn

Now everything works perfectly. I am very fine without invoking mkl.

[smithsp]

Following the instructions of @jskDr worked for me.
Thanks @jskDr . 👍

[maksimt]

Also had to use @jskDr 's solution on a fresh anaconda install on Ubuntu 16.04. The other upvoted methods didn't work for me. I'm hoping to be able to use MKL in the future.

[nehaljwani]

I came across the same error while building cvxopt with mkl:11.3.3 and mkl:2017.0.0, separately. I got around it by linking libmkl_rt.so and now no LD_PRELOAD trick is required. The linking order used by me was: -lmkl_rt -lmkl_core -lmkl_intel_lp64 -lmkl_sequential

[oxydron]

@jskDr solution indeed do work and don't know why.

Thanks mate!

[rcywongaa]

Occurs when using python3.5 but not python3.6

[jdhao]

I am using Anaconda 2 on Centos 7, I also met this issue. Following @jskDr 's solution, the problem disappeared. Thanks!

[chrishyx]

I am using Anaconda3-4.3.0 on Ubuntu16.04, also met this issue and solved by following @jskDr 's solution.
Thanks!

[maartenbreddels]

For me the problem was resolved by removing numpy from ~/.local/.... If any pip install --user <package> command dragged numpy to install in ~/.local/lib/pythonX.Y/site-packages/numpy/ it will always use that numpy, and conda list will not show it, so you may thing you don't have that installed. pip uninstall numpy fixed it. I think that explains many of the issues, for instance switching to 3.6, since numpy for 3.5 won't be found anymore. Possibly similar issues exists for scipy, I didn't check.

[havakv]

For me (on Ubuntu 14.04, python 3.6.1) it started working after installing numexpr (conda install numexpr). No paths specified.

[MInner]

For me problems started appearing after installing a Python 2.7 with corresponding numpy alongside root env. Creating a new env with python 3 and numpy helped and none of suggestions above did not.

[pamdla]

This issue still exists.

My environment configs are:
VM: {gues_os: Ubutu 14.04 64, anaconda2: 4.4.0}
And I can assure both mkl and mkl-service are installed, and resides at '/opt/anaconda2/lib/'.
However, when I fire python -c "import gensim", the exact error msg is:
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so

Using LD_PRELOAD=... does not help me out either.
Using LD_DEBUG=symbol python -c 'import gensim' does give me details, and found 4 places for ****

1.  9582:	symbol=mkl_dft_avx_xs_f32_1df;  lookup in file=/opt/anaconda2/lib/python2.7/site-packages/scipy/special/../../../../libmkl_avx.so [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=python [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/opt/anaconda2/bin/../lib/libpython2.7.so.1.0 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libutil.so.1 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/opt/anaconda2/lib/python2.7/site-packages/scipy/special/../../../../libmkl_avx.so [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      **9582:	/opt/anaconda2/lib/python2.7/site-packages/scipy/special/../../../../libmkl_avx.so**: error: symbol lookup error: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8 (fatal)
      
2. 9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/opt/anaconda2/bin/../lib/libpython2.7.so.1.0 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libutil.so.1 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/opt/anaconda2/lib/python2.7/site-packages/scipy/special/../../../../libmkl_def.so [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
      9582:	symbol=mkl_sparse_optimize_bsr_trsm_i8;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     ** 9582:	/opt/anaconda2/lib/python2.7/site-packages/scipy/special/../../../../libmkl_def.so**: error: symbol lookup error: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8 (fatal)

It reveals that the installed scipy is referencing MKL libs. But the installed MKL coming along with anaconda is located at '/opt/anaconda2/lib/libmkl_{core,avx,def}.so'.

And for aforementioned solutions, I believe either to uninstall mkl or to install nomkl is not solving this issue.
If understand correctly, nomkl is NON-MKL version for originally mkl-relavant packages such as scipy, numpy. Therefore installing nomkl will not give us MKL's benefit for later use while mkl- relevant boosting was assumed in MKL-verion of numpy/scipy/etc, right?

I tried to find which file is referencing these so file with grep -r lmkl_avx and grep -r lmkl_def, but no result returned.

I also installed Intel MKL package downloaded from Intel's site. Similar error log apears, but in addition I found:

      9517:	symbol=COIProcessLoadSinkLibraryFromFile;  lookup in file=/opt/anaconda2/lib/python2.7/site-packages/scipy/special/../../../../libmkl_intel_lp64.so [0]
      9517:	symbol=COIProcessLoadSinkLibraryFromFile;  lookup in file=/opt/anaconda2/lib/python2.7/site-packages/scipy/special/../../../../libmkl_intel_thread.so [0]
      9517:	symbol=COIProcessLoadSinkLibraryFromFile;  lookup in file=/opt/anaconda2/lib/python2.7/site-packages/scipy/special/../../../../libmkl_core.so [0]
      9517:	symbol=COIProcessLoadSinkLibraryFromFile;  lookup in file=/opt/anaconda2/lib/python2.7/site-packages/scipy/special/../../../../libiomp5.so [0]
     ........

The problem, I guess, is how to correctly link the /opt/anaconda2/lib/liblmk_{core|avx|def}.so for scipy or gensim, by hard code or dynamic way -ld.

I will continuing working and the post a solution, if no one solves.

[pamdla]

In my problematic Ubuntu vbox, this issue is solved by using:

LD_PRELOAD=/opt/anaconda2/lib/libmkl_core.so:/opt/anaconda2/lib/libmkl_sequential.so python -c 'import gensim'

Also, in another copy of Ubuntu vbox as well as Centos vbox, I can do python -c "import gensim" after installing the same version (4.4.0) of anaconda and gensim in my problematic Ubuntu without any additional settings.

It seems that the issue has sth to with OS or Shell environment.

[mrdivine]

On my scientific linux box, I fixed this issue by pinning mkl to version 11.3.3

conda install mkl=11.3.3

[akshay-hazari]

Nothing seems to work for me here. Though for me its not an anaconda issiue may be. I am getting this error from a VirtualEnv set up for django. Any help is appreciated,

[akshay-hazari]

This did the trick when everything else failed -
sudo cp /home/akshay/anaconda2/pkgs/mkl-2017.0.1-0/lib/* /usr/lib
Or this might also work export PATH="/home/akshay/anaconda2/pkgs/mkl-2017.0.1-0/lib:$PATH"
While someone else also suggested this export PATH="/home/anaconda2/bin:$PATH"

[emailweixu]

Executing the following before other imports solves the problem for me:
sys.setdlopenflags(sys.getdlopenflags() | ctypes.RTLD_GLOBAL)

[rprasadn]

I was having this error inside a conda environment using py3.5 in Ubuntu 14.04. My problem was resolved by executing conda install mkl inside the environment.

[Nzumino]

I had the "Intel MKL FATAL ERROR: Error on loading function mkl_blas_avx_xdcopy". It was resolved by executing conda update mkl.

[SankBad]

This issue is still persisting for me

[2020zyc]

The above solutions are invalid for me, and I solved it by adding the two lines to the beginning of my code:
"""
import mkl
mkl.get_max_threads()
"""