Unable to build mxnet with USE_OPENMP=OFF with MKLDNN backend

[cyrusbehr]

I need to compile mxnet to run in single thread mode. From what I have found, using the environment variables MXNET_ENGINE_TYPE=NaiveEngine and OMP_NUM_THREADS=1 and using omp_set_num_threads(1); in my c++ code gets the number of threads down to 2. To get the number of threads down to 1, I need to compile the library with USE_OPENMP=OFF (discussed more in this issue). In the past, this has worked successfully for me. However, when I tried doing this again, I get an error during compilation. I am using MKLDNN for the backend.

Here are my steps:

git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet
cd mxnet
mkdir build 
cd build

cmake -DUSE_CPP_PACKAGE=1 -DUSE_CUDA=0 -DUSE_MKL_IF_AVAILABLE=1 -DUSE_OPENCV=0 -DUSE_LAPACK=0 -DUSE_OPENMP=0 \
  -DMKL_INCLUDE_DIR=/opt/intel/compilers_and_libraries/linux/mkl/include -DMKL_RT_LIBRARY=/opt/intel/compilers_and_libraries/linux/mkl/lib/intel64/libmkl_rt.so ..

make -j16

Here is the cmake output:

-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMAKE_CROSSCOMPILING FALSE
-- CMAKE_HOST_SYSTEM_PROCESSOR x86_64
-- CMAKE_SYSTEM_PROCESSOR x86_64
-- CMAKE_SYSTEM_NAME Linux
-- CMake version '3.13.3' using generator 'Unix Makefiles'
-- Performing Test SUPPORT_CXX11
-- Performing Test SUPPORT_CXX11 - Success
-- Performing Test SUPPORT_CXX0X
-- Performing Test SUPPORT_CXX0X - Success
-- Performing Test SUPPORT_MSSE3
-- Performing Test SUPPORT_MSSE3 - Success
-- Performing Test SUPPORT_MSSE2
-- Performing Test SUPPORT_MSSE2 - Success
-- Determining F16C support
-- Performing Test COMPILER_SUPPORT_MF16C
m-- Performing Test COMPILER_SUPPORT_MF16C - Success
-- F16C enabled
-- CMAKE_BUILD_TYPE is unset, defaulting to Release
-- MKL-DNN compat: set DNNL_BUILD_EXAMPLES to MKLDNN_BUILD_EXAMPLES with value `OFF`
-- MKL-DNN compat: set DNNL_BUILD_TESTS to MKLDNN_BUILD_TESTS with value `OFF`
-- MKL-DNN compat: set DNNL_ENABLE_JIT_PROFILING to MKLDNN_ENABLE_JIT_PROFILING with value `OFF`
-- MKL-DNN compat: set DNNL_LIBRARY_TYPE to MKLDNN_LIBRARY_TYPE with value `STATIC`
-- MKL-DNN compat: set DNNL_ARCH_OPT_FLAGS to MKLDNN_ARCH_OPT_FLAGS with value ``
-- Looking for pthread.h
a-- Looking for pthread.h - found
-- Looking for pthread_create
ke-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
 -- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
--- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
j-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- GPU support is disabled
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Found Git: /usr/bin/git (found version "2.17.1") 
-- Intel(R) VTune(TM) Amplifier JIT profiling disabled
-- Found MKL: /opt/intel/compilers_and_libraries/linux/mkl/include  
-- Found MKL (include: /opt/intel/compilers_and_libraries/linux/mkl/include, lib: /opt/intel/compilers_and_libraries/linux/mkl/lib/intel64/libmkl_rt.so
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) 
-- Could NOT find Jemalloc (missing: JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR) 
-- OpenCV Disabled
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) 
-- Could NOT find Jemalloc (missing: JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR) 
CMake Warning at 3rdparty/googletest/googletest/CMakeLists.txt:47 (project):
  VERSION keyword not followed by a value or was followed by a value that
  expanded to nothing.


-- Found PythonInterp: /usr/bin/python (found version "2.7.17") 
-- Found GTest: gtest  
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Looking for fopen64
-- Looking for fopen64 - not found
-- Looking for C++ include cxxabi.h
-- Looking for C++ include cxxabi.h - found
-- Looking for nanosleep
-- Looking for nanosleep - found
-- Looking for backtrace
-- Looking for backtrace - found
-- backtrace facility detected in default set of libraries
-- Found Backtrace: /usr/include  
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- /root/mxnet/3rdparty/dmlc-core/cmake/build_config.h.in -> include/dmlc/build_config.h
CMake Warning at CMakeLists.txt:830 (message):
  OpenCV_VERSION_MAJOR: , version 3 with imgcodecs is required for im2rec,
  im2rec will not be available


CMake Warning at example/image-classification/predict-cpp/CMakeLists.txt:3 (message):
  OpenCV version >= 3 should be enabled and found to build image
  classification example, skipping...


-- Configuring done
-- Generating done
-- Build files have been written to: /root/mxnet/build_openmpoff

The build reaches the 96% point then prints out a bunch of errors (regarding openmp and mkldnn). Here is a preview of the errors:

[ 96%] Building CXX object tests/CMakeFiles/mxnet_unit_tests.dir/cpp/test_main.cc.o
In file included from /root/mxnet/3rdparty/mshadow/../../src/operator/nn/activation-inl.h:40:0,
                 from /root/mxnet/tests/cpp/operator/tune/operator_tune_test.cc:21:
/root/mxnet/3rdparty/mshadow/../../src/operator/nn/../mshadow_op.h: In static member function ‘static bool mxnet::op::mshadow_op::mul::Map(bool, bool)’:
/root/mxnet/3rdparty/mshadow/../../src/operator/nn/../mshadow_op.h:350:42: warning: ‘*’ in boolean context, suggest ‘&&’ instead [-Wint-in-bool-context]
/root/mxnet/3rdparty/mshadow/../../src/operator/nn/../mshadow_op.h:111:14:
       return (expr); \
              ~~~~~~                       
/root/mxnet/3rdparty/mshadow/../../src/operator/nn/../mshadow_op.h:350:42:
 MXNET_BINARY_MATH_OP_NC_WITH_BOOL(mul, a * b);
/root/mxnet/3rdparty/mshadow/../../src/operator/nn/../mshadow_op.h:111:15: note: in definition of macro ‘MXNET_BINARY_MATH_OP_NC_WITH_BOOL’
       return (expr); \
               ^~~~
In file included from /root/mxnet/tests/cpp/include/../../../src/ndarray/ndarray_function.h:34:0,
                 from /root/mxnet/tests/cpp/include/test_util.h:37,
                 from /root/mxnet/tests/cpp/storage/storage_test.cc:29:
/root/mxnet/tests/cpp/include/../../../src/ndarray/../operator/mshadow_op.h: In static member function ‘static bool mxnet::op::mshadow_op::mul::Map(bool, bool)’:
/root/mxnet/tests/cpp/include/../../../src/ndarray/../operator/mshadow_op.h:350:42: warning: ‘*’ in boolean context, suggest ‘&&’ instead [-Wint-in-bool-context]
/root/mxnet/tests/cpp/include/../../../src/ndarray/../operator/mshadow_op.h:111:14:
       return (expr); \
              ~~~~~~                       
/root/mxnet/tests/cpp/include/../../../src/ndarray/../operator/mshadow_op.h:350:42:
 MXNET_BINARY_MATH_OP_NC_WITH_BOOL(mul, a * b);
/root/mxnet/tests/cpp/include/../../../src/ndarray/../operator/mshadow_op.h:111:15: note: in definition of macro ‘MXNET_BINARY_MATH_OP_NC_WITH_BOOL’
       return (expr); \
               ^~~~
[ 96%] Built target mxnet
Scanning dependencies of target cpp_package_op_h
Running: OpWrapperGenerator.py
Traceback (most recent call last):
  File "OpWrapperGenerator.py", line 433, in <module>
    raise(e)
OSError: /root/mxnet/build_openmpoff/libmxnet.so: undefined symbol: omp_get_max_threads
cpp-package/CMakeFiles/cpp_package_op_h.dir/build.make:58: recipe for target 'cpp-package/CMakeFiles/cpp_package_op_h' failed
make[2]: *** [cpp-package/CMakeFiles/cpp_package_op_h] Error 1
CMakeFiles/Makefile2:1730: recipe for target 'cpp-package/CMakeFiles/cpp_package_op_h.dir/all' failed
make[1]: *** [cpp-package/CMakeFiles/cpp_package_op_h.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 96%] Linking CXX executable mxnet_unit_tests
../3rdparty/mkldnn/src/libdnnl.a(cpu_engine.cpp.o): In function `void dnnl::impl::parallel_nd<int const&, int const&, dnnl::impl::cpu::ref_shuffle_t<4>::ref_shuffle_t(dnnl::impl::cpu::ref_shuffle_t<4>::pd_t const*)::{lambda(int, int)#1}>(int const&, int const&, dnnl::impl::cpu::ref_shuffle_t<4>::ref_shuffle_t(dnnl::impl::cpu::ref_shuffle_t<4>::pd_t const*)::{lambda(int, int)#1}&&) [clone ._omp_fn.2]':
cpu_engine.cpp:(.text._ZN4dnnl4impl11parallel_ndIJRKiS3_ZNS0_3cpu13ref_shuffle_tILi4EEC4EPKNS6_4pd_tEEUliiE_EEEvDpOT_._omp_fn.2[_ZNK4dnnl4impl3cpu13ref_shuffle_tILi4EE4pd_t16create_primitiveEPP14dnnl_primitive]+0x3e): undefined reference to `omp_get_num_threads'
cpu_engine.cpp:(.text._ZN4dnnl4impl11parallel_ndIJRKiS3_ZNS0_3cpu13ref_shuffle_tILi4EEC4EPKNS6_4pd_tEEUliiE_EEEvDpOT_._omp_fn.2[_ZNK4dnnl4impl3cpu13ref_shuffle_tILi4EE4pd_t16create_primitiveEPP14dnnl_primitive]+0x45): undefined reference to `omp_get_thread_num'
../3rdparty/mkldnn/src/libdnnl.a(cpu_engine.cpp.o): In function `void dnnl::impl::parallel_nd<int const&, int const&, dnnl::impl::cpu::ref_shuffle_t<1>::ref_shuffle_t(dnnl::impl::cpu::ref_shuffle_t<1>::pd_t const*)::{lambda(int, int)#1}>(int const&, int const&, dnnl::impl::cpu::ref_shuffle_t<1>::ref_shuffle_t(dnnl::impl::cpu::ref_shuffle_t<1>::pd_t const*)::{lambda(int, int)#1}&&) [clone ._omp_fn.0]':
cpu_engine.cpp:(.text._ZN4dnnl4impl11parallel_ndIJRKiS3_ZNS0_3cpu13ref_shuffle_tILi1EEC4EPKNS6_4pd_tEEUliiE_EEEvDpOT_._omp_fn.0[_ZNK4dnnl4impl3cpu13ref_shuffle_tILi1EE4pd_t16create_primitiveEPP14dnnl_primitive]+0x3e): undefined reference to `omp_get_num_threads'
cpu_engine.cpp:(.text._ZN4dnnl4impl11parallel_ndIJRKiS3_ZNS0_3cpu13ref_shuffle_tILi1EEC4EPKNS6_4pd_tEEUliiE_EEEvDpOT_._omp_fn.0[_ZNK4dnnl4impl3cpu13ref_shuffle_tILi1EE4pd_t16create_primitiveEPP14dnnl_primitive]+0x45): undefined reference to `omp_get_thread_num'
../3rdparty/mkldnn/src/libdnnl.a(cpu_engine.cpp.o): In function `void dnnl::impl::parallel_nd<int const&, int const&, dnnl::impl::cpu::ref_shuffle_t<2>::ref_shuffle_t(dnnl::impl::cpu::ref_shuffle_t<2>::pd_t const*)::{lambda(int, int)#1}>(int const&, int const&, dnnl::impl::cpu::ref_shuffle_t<2>::ref_shuffle_t(dnnl::impl::cpu::ref_shuffle_t<2>::pd_t const*)::{lambda(int, int)#1}&&) [clone ._omp_fn.1]':
cpu_engine.cpp:(.text._ZN4dnnl4impl11parallel_ndIJRKiS3_ZNS0_3cpu13ref_shuffle_tILi2EEC4EPKNS6_4pd_tEEUliiE_EEEvDpOT_._omp_fn.1[_ZNK4dnnl4impl3cpu13ref_shuffle_tILi2EE4pd_t16create_primitiveEPP14dnnl_primitive]+0x3e): undefined reference to `omp_get_num_threads'
cpu_engine.cpp:(.text._ZN4dnnl4impl11parallel_ndIJRKiS3_ZNS0_3cpu13ref_shuffle_tILi2EEC4EPKNS6_4pd_tEEUliiE_EEEvDpOT_._omp_fn.1[_ZNK4dnnl4impl3cpu13ref_shuffle_tILi2EE4pd_t16create_primitiveEPP14dnnl_primitive]+0x45): undefined reference to `omp_get_thread_num'
../3rdparty/mkldnn/src/libdnnl.a(cpu_engine.cpp.o): In function `dnnl_status_t dnnl_primitive_desc::create<dnnl::impl::cpu::jit_avx512_core_bf16_convolution_fwd_t::pd_t>(dnnl_primitive_desc**, dnnl::impl::op_desc_t const*, dnnl_primitive_attr const*, dnnl_engine*, dnnl_primitive_desc const*)':
cpu_engine.cpp:(.text._ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu38jit_avx512_core_bf16_convolution_fwd_t4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_[_ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu38jit_avx512_core_bf16_convolution_fwd_t4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_]+0x24a): undefined reference to `omp_get_max_threads'
../3rdparty/mkldnn/src/libdnnl.a(cpu_engine.cpp.o): In function `dnnl_status_t dnnl_primitive_desc::create<dnnl::impl::cpu::jit_avx512_common_convolution_fwd_t<(dnnl_data_type_t)3, (dnnl_data_type_t)3, (dnnl_data_type_t)3>::pd_t>(dnnl_primitive_desc**, dnnl::impl::op_desc_t const*, dnnl_primitive_attr const*, dnnl_engine*, dnnl_primitive_desc const*)':
cpu_engine.cpp:(.text._ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu35jit_avx512_common_convolution_fwd_tIL16dnnl_data_type_t3ELS5_3ELS5_3EE4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_[_ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu35jit_avx512_common_convolution_fwd_tIL16dnnl_data_type_t3ELS5_3ELS5_3EE4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_]+0x11d): undefined reference to `omp_get_max_threads'
../3rdparty/mkldnn/src/libdnnl.a(cpu_engine.cpp.o): In function `dnnl_status_t dnnl_primitive_desc::create<dnnl::impl::cpu::jit_avx512_core_x8s8s32x_convolution_fwd_t<(dnnl_data_type_t)6, (dnnl_data_type_t)3>::pd_t>(dnnl_primitive_desc**, dnnl::impl::op_desc_t const*, dnnl_primitive_attr const*, dnnl_engine*, dnnl_primitive_desc const*)':
cpu_engine.cpp:(.text._ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu42jit_avx512_core_x8s8s32x_convolution_fwd_tIL16dnnl_data_type_t6ELS5_3EE4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_[_ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu42jit_avx512_core_x8s8s32x_convolution_fwd_tIL16dnnl_data_type_t6ELS5_3EE4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_]+0x134): undefined reference to `omp_get_max_threads'
../3rdparty/mkldnn/src/libdnnl.a(cpu_engine.cpp.o): In function `dnnl_status_t dnnl_primitive_desc::create<dnnl::impl::cpu::jit_avx512_core_x8s8s32x_convolution_fwd_t<(dnnl_data_type_t)6, (dnnl_data_type_t)4>::pd_t>(dnnl_primitive_desc**, dnnl::impl::op_desc_t const*, dnnl_primitive_attr const*, dnnl_engine*, dnnl_primitive_desc const*)':
cpu_engine.cpp:(.text._ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu42jit_avx512_core_x8s8s32x_convolution_fwd_tIL16dnnl_data_type_t6ELS5_4EE4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_[_ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu42jit_avx512_core_x8s8s32x_convolution_fwd_tIL16dnnl_data_type_t6ELS5_4EE4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_]+0x134): undefined reference to `omp_get_max_threads'
../3rdparty/mkldnn/src/libdnnl.a(cpu_engine.cpp.o): In function `dnnl_status_t dnnl_primitive_desc::create<dnnl::impl::cpu::jit_avx512_core_x8s8s32x_convolution_fwd_t<(dnnl_data_type_t)6, (dnnl_data_type_t)6>::pd_t>(dnnl_primitive_desc**, dnnl::impl::op_desc_t const*, dnnl_primitive_attr const*, dnnl_engine*, dnnl_primitive_desc const*)':
cpu_engine.cpp:(.text._ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu42jit_avx512_core_x8s8s32x_convolution_fwd_tIL16dnnl_data_type_t6ELS5_6EE4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_[_ZN19dnnl_primitive_desc6createIN4dnnl4impl3cpu42jit_avx512_core_x8s8s32x_convolution_fwd_tIL16dnnl_data_type_t6ELS5_6EE4pd_tEEE13dnnl_status_tPPS_PKNS2_9op_desc_tEPK19dnnl_primitive_attrP11dnnl_enginePKS_]+0x134): undefined reference to `omp_get_max_threads'

[TaoLv]

@cyrusbehr Could you please try to apply below change to the CMakeList file?

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 2612854..3068925 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -288,6 +288,7 @@ if(USE_MKLDNN)
   set(MKLDNN_ARCH_OPT_FLAGS "" CACHE INTERNAL "" FORCE)
   set(MKLDNN_ENABLE_JIT_PROFILING OFF CACHE INTERNAL "" FORCE)
   set(MKLDNN_LIBRARY_TYPE STATIC CACHE INTERNAL "" FORCE)
+  set(MKLDNN_CPU_RUNTIME SEQ CACHE INTERNAL "" FORCE)

   add_subdirectory(3rdparty/mkldnn)

Let me know if it works for you. Thanks.

[leezu]

@TaoLv this fixes the issue for me (though I didn't try linking with MKL). I'll reassign this issue to you, as it seems to be due to MKLDNN integration (ie above missing variable).

Ie with your diff applied, cmake -DUSE_CPP_PACKAGE=1 -DUSE_CUDA=0 -DUSE_MKLDNN=ON -DUSE_OPENCV=0 -DUSE_LAPACK=0 -DUSE_OPENMP=0 -GNinja ..; ninja works

[TaoLv]

@leezu Thanks. We need check USE_OPENMP in cmake as what we did for makefile: https://github.com/apache/incubator-mxnet/blob/master/mkldnn.mk#L35. I will file a PR to fix the problem once the patch is verified by the reporter.

[cyrusbehr]

@leezu @TaoLv
Adding set(MKLDNN_CPU_RUNTIME SEQ CACHE INTERNAL "" FORCE) solved the problem.
Thanks!