MKL library load error (libmkl_avx2.so or libmkl_def.so)

[mcg1969]

Actual Behavior

Python crashes with the following error:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so

Expected Behavior

No crashy

Steps to Reproduce

On Linux 64:

conda create -n btest cvxopt --yes
source activate btest

Then start a python prompt and:

import cvxopt
x = cvxopt.matrix([[1,2],[3,4]], tc='d')
y = cvxopt.matrix([[1,2],[3,4]], tc='d')
z = cvxopt.matrix([[1,2],[3,4]], tc='d')
cvxopt.blas.gemm(x,y,z)

Anaconda or Miniconda version:

Here's the exact environment:

    ca-certificates: 2018.03.07-0        defaults
    certifi:         2018.4.16-py36_0    defaults
    cvxopt:          1.1.8-py36_4        defaults
    intel-openmp:    2018.0.0-8          defaults
    libedit:         3.1-heed3624_0      defaults
    libffi:          3.2.1-hd88cf55_4    defaults
    libgcc-ng:       7.2.0-hdf63c60_3    defaults
    libstdcxx-ng:    7.2.0-hdf63c60_3    defaults
    mkl:             2017.0.4-h4c4d0af_0 defaults
    ncurses:         6.0-h9df7e31_2      defaults
    openssl:         1.0.2o-h20670df_0   defaults
    pip:             9.0.3-py36_0        defaults
    python:          3.6.5-hc3d631a_0    defaults
    readline:        7.0-ha6073c6_4      defaults
    setuptools:      39.0.1-py36_0       defaults
    sqlite:          3.23.1-he433501_0   defaults
    tk:              8.6.7-hc745277_3    defaults
    wheel:           0.31.0-py36_0       defaults
    xz:              5.2.3-h55aa19d_2    defaults
    zlib:            1.2.11-ha838bed_2   defaults

Operating System:

Linux 64. This is on AE5.

[mcg1969]

Clearly related to #720

[mcg1969]

I can fix the issue by setting this environment variable:

LD_PRELOAD=/opt/continuum/anaconda/envs/default/lib/libmkl_def.so:/opt/continuum/anaconda/envs/default/lib/libmkl_avx.so:/opt/continuum/anaconda/envs/default/lib/libmkl_core.so:/opt/continuum/anaconda/envs/default/lib/libmkl_intel_lp64.so:/opt/continuum/anaconda/envs/default/lib/libmkl_intel_thread.so:/opt/continuum/anaconda/envs/default/lib/libiomp5.so

[msarahan]

I think it's related to AnacondaRecipes/numpy-feedstock#5

even though numpy isn't involved here, the load order of openmp and mkl matters a lot. If the system loads GNU openmp, then MKL needs to respect that (but does not default to it). Intel's patch to numpy tries to do a runtime detection for this. It will be added to future MKL versions too, but that doesn't help you much here.

What exactly is pulling in mkl here? Is it just scipy, or are other things also using it?

[mcg1969]

Interestingly, cvxopt is not listed as depending on NumPy or SciPy, only mkl itself. And I'm calling cvxopt.blas.* here, so it seems like a direct call to the underlying MKL library.

[mcg1969]

Ah, cvxopt.blas is a C extension.

[msarahan]

Confirmed that this works with the latest cvxopt 1.2.0 packages I've built. Closing this out.

There are mkl and openblas variants there - you should be able to pick between them by adding the nomkl package (or not).

[jacksonloper]

Wait so is

LD_PRELOAD=/opt/continuum/anaconda/envs/default/lib/libmkl_def.so:/opt/continuum/anaconda/envs/default/lib/libmkl_avx.so:/opt/continuum/anaconda/envs/default/lib/libmkl_core.so:/opt/continuum/anaconda/envs/default/lib/libmkl_intel_lp64.so:/opt/continuum/anaconda/envs/default/lib/libmkl_intel_thread.so:/opt/continuum/anaconda/envs/default/lib/libiomp5.so

The accepted solution? That seems... suboptimal. Why is the problem happening? I'm no ldconfig whiz... I wonder if anybody could explain to me in layman's term why some packages are having a tough time loading shared libraries from /opt/conda/lib?

[xzhai12]

Very interesting and confusing: (with Windows 10)

Installed Anaconda 5.3 today after uninstalling Anaconda 5.2, and now I keep getting this error when importing numpy. Nothing can fix this and no idea why this is happening.

[msarahan]

My guess would be that LD_PRELOAD is proposed as the answer because other MKL libraries get found otherwise. We compile all of our binaries so that they look for MKL in our stuff, but if you load MKL into your process before our stuff goes looking for it, the other one will already be loaded.

New packages are built with newer MKL. The lower bound of their compatibility is the version that they were built with. Newer MKL works with stuff built with older MKL, but not vice versa. @shldngzh you probably have MKL somewhere else on your system. It used to be a compatible version, but now it is too old. You must find any MKL DLL's that would be getting found first and remove them. See #8561 for more discussion on this issue.

[jacksonloper]

Thanks @msarahan, that is very clear. It feels deeply tragic that the dynamic linking architecture wasn't designed to deal better with loading multiple versions of the "same" library.

[msarahan]

Tell me about it! That's the vast majority of the issues that our users have.

[mingwandroid]

Run with LD_DEBUG=libs to see what's going on here.

[xzhai12]

@msarahan Thanks for your advice! I do not have any mkl*.dll in my system32 folder and whether I add Python to my system PATH, importing numpy would always throw an error. However, the errors are different:

if Python is added to PATH (chosen when installing with Anaconda), error message is "cannot load mkl_intel_thread.dll".

if Python is NOT added to PATH, error message is

>>> import numpy
Traceback (most recent call last):
  File "C:\Users\xiaodong.zhai\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
    from . import multiarray
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\sdfasdfa\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\__init__.py", line 142, in <module>
    from . import add_newdocs
  File "C:\Users\sdfasdfa\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "C:\Users\sdfasdfa\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
    from .type_check import *
  File "C:\Users\sdfasdfa\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "C:\Users\sdfasdfa\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\core\__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: DLL load failed: The specified module could not be found.

I'm still trying to locate (if any) an older version of mkl lib, probably it's my Matlab 2011b.

[mingwandroid]

You need to activate your env rather than manually editing PATH.

[xzhai12]

@msarahan @mingwandroid
Finally figured out why my numpy fails. It's Mosek 6.0. Somehow the numpy loads mkl lib from Mosek first (I guess it's because Mosek was in my PATH) and the mkl version is too old to work with current numpy version.

So the issue may be viewed as - somewhere you have an old version of mkl lib gets loaded before the mkl libs in Anaconda. This may not cause any trouble if you run something from Anaconda prompt; however, if you use IDE like PyCharm or run a script in PowerShell, it might do.

Thanks to everyone.

[mingwandroid]

We do not support running outside of activate.

You can activate from the Anaconda Prompt then launch PyCharm from that if you want things to work properly. Otherwise use VSCode, that does activate conda envs properly (a fairly recent addition I think). Maybe the PyCharm people would consider fixing this? It may be worth asking them to.

[mcg1969]

@shidnzgh to build on Ray's point, environment activation requires doing more than just setting the PATH properly. For instance, many packages require certain environment variables to be set. In fact. In theory, a conda package can require arbitrary code to execute on activation, though that's strongly discouraged. Still, the only way to ensure that a conda environment is properly ready to use is to activate.

[xzhai12]

We do not support running outside of activate.

You can activate from the Anaconda Prompt then launch PyCharm from that if you want things to work properly. Otherwise use VSCode, that does activate conda envs properly (a fairly recent addition I think). Maybe the PyCharm people would consider fixing this? It may be worth asking them to.

This is good to know. I didn't think about launching PyCharm from Anaconda Prompt. Thanks.

My problem is solved by removing an old version of mkl lib, but now I do get your point that "it needs to be activated" and it should be the most guaranteed way to make sure a correct and consistent environment is in use.