Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

[jhyuklee]

After all the installations (faiss, drqa, and the two requirements.txt from this repo), run_index_pred_eval.py gives an error like below:

$ python open/run_index_pred_eval.py
/home/jinhyuk/github/kernel-sparse/dense
/data_nfs/camist002/data/dev-3.json
--para
--no_od
sampling from:
/home/jinhyuk/github/kernel-sparse/dense/phrase.hdf5
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 36.63it/s]
WARNING clustering 788 points to 256 centroids: please provide at least 9984 training points████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 341.87it/s]
Clustering 788 points in 481D to 256 clusters, redo 1 times, 10 iterations
Preprocessing in 0.00 s
INTEL MKL ERROR: /home/jinhyuk/miniconda3/envs/kesper/lib/python3.6/site-packages/faiss/../../../libmkl_avx2.so: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Following the recommendation from here, the conda install nomkl numpy scipy scikit-learn numexpr command shows that there are some conflicts between the versions.

$ conda install nomkl numpy scipy scikit-learn numexpr
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Package libopenblas conflicts for:
scikit-learn -> numpy[version='>=1.11.3,<2.0a0'] -> libopenblas[version='>=0.3.2,<0.3.3.0a0']
Package blas conflicts for:
mkl_fft -> numpy-base[version='>=1.0.6,<2.0a0'] -> blas[version='|1.0',build=openblas]
blas
scikit-learn -> blas[version='||1.0',build='mkl|openblas|mkl|openblas']
nomkl -> blas=[build=openblas]
mkl_fft -> blas[version='|1.0',build=mkl]
numexpr -> blas[version='||1.0',build='mkl|openblas|mkl|openblas']
numpy -> blas[version='||1.0',build='mkl|openblas|mkl|openblas']
mkl_random -> blas[version='|1.0',build=mkl]
faiss-cpu=1.5.2 -> numpy[version='>=1.11'] -> numpy-base==1.16.0=py36hde5b4d6_1 -> blas[version='|1.0',build=openblas]
scipy -> blas[version='||1.0',build='mkl|openblas|mkl|openblas']
mkl_random -> numpy-base[version='>=1.0.2,<2.0a0'] -> blas[version='|1.0',build=openblas]
numpy-base -> blas[version='|*|1.0',build='mkl|openblas|mkl|openblas']
faiss-cpu=1.5.2 -> blas=[build=mkl]
faiss-cpu=1.5.2 -> numpy[version='>=1.11'] -> blas==1.0=mkl

Any idea how to resolve this?

[jhyuklee]

Seems like Anaconda related issue (see here).
This solved the issue.