omp_lock_t is undefined on Intel serial build

[lifflander]

Describe the bug
From an EMPIRE dev:

If I build intel-serial (no openmp), I get the compiler error,
/pscratch/nroberd/EMPIRE/TPL/vt/src/vt/utils/mutex/omp_mutex.h(57): error: identifier "omp_lock_t" is undefined
omp_lock_t omp_lock;

[lifflander]

stdout_and_stderr.txt

Here's the output file that was provided to me. Not sure if it will be super useful!

[cz4rs]

ok, this actually contains one major hint:

FAILED: src/circuit/CMakeFiles/circuit.dir/TransmissionLineSolver.cpp.o 
/opt/openmpi/4.0/intel/bin/mpicxx  -DDISABLE_TPL_CHECKPOINT=0 -DEMPIRE_CHECK_FPE -DFMT_HEADER_ONLY=1 -DFMT_USE_USER_DEFINED_LITERALS=0 -DHAS_DETECTION_COMPONENT=1 -DKOKKOS_ENABLED_CHECKPOINT=1 -DKOKKOS_KERNELS_ENABLED=1 -I/pscratch/nroberd/EMPIRE/src/circuit -Isrc/circuit/.. -I/pscratch/nroberd/EMPIRE/src/utils -I/pscratch/nroberd/EMPIRE/src/input_deck -I/pscratch/nroberd/EMPIRE/src/em_solvers -I/pscratch/nroberd/trilinos/install/INTEL-18.0.2_OPENMPI-4.0.1-RELEASE-SERIAL-STATIC/include -I/projects/sparc/tpls/cts1-bdw/superlu_dist-5.4.0/a3121eaff44f7bf7d44e625c3b3d2a9911e58876/cts1-bdw_intel-19.0.5_openmpi-4.0.1/include -I/projects/sparc/tpls/cts1-bdw/netcdf-4.7.0/24baa07a3fa1ff9dbc8e70dc591ebbdec56783b2/cts1-bdw_intel-19.0.5_openmpi-4.0.1/include -I/projects/sparc/tpls/cts1-bdw/hdf5-1.10.5/00000000/cts1-bdw_intel-19.0.5_openmpi-4.0.1/include -I/projects/sparc/tpls/cts1-bdw/pnetcdf-1.12.1/6144dc67b2041e4093063a04e89fc1e33398bd09/cts1-bdw_intel-19.0.5_openmpi-4.0.1/include -I/projects/sparc/tpls/cts1-bdw/cgns-c09a5cd/d313cc2f822078e47c7dbdee074ecb0431e573eb/cts1-bdw_intel-19.0.5_openmpi-4.0.1/include -I/projects/sparc/tpls/cts1-bdw/parmetis-4.0.3/00000000/cts1-bdw_intel-19.0.5_openmpi-4.0.1/include -I/projects/sparc/tpls/cts1-bdw/boost-1.72.0/00000000/cts1-bdw_intel-19.0.5/include -Isrc -I/pscratch/nroberd/EMPIRE -ITPL/checkpoint/src -I/pscratch/nroberd/EMPIRE/TPL/checkpoint/src -I/pscratch/nroberd/EMPIRE/TPL/detector/src -I/pscratch/nroberd/EMPIRE/src/evaluators -I/pscratch/nroberd/EMPIRE/TPL/vt/lib/fmt -I/pscratch/nroberd/EMPIRE/TPL/vt/lib/CLI -ITPL/vt/release -I/pscratch/nroberd/EMPIRE/TPL/vt/src -I/pscratch/nroberd/EMPIRE/src/linear_solvers -isystem /pscratch/nroberd/trilinos/install/INTEL-18.0.2_OPENMPI-4.0.1-RELEASE-SERIAL-STATIC/lib/cmake/Kokkos/../../../include -fPIC -xCORE-AVX2 -O3 -DNDEBUG -Wall -Wshadow -Wunused-variable -pedantic -Wno-unknown-pragmas -Wno-narrowing -Wno-format -O3 -DNDEBUG   -fPIC -qopenmp -std=c++14 -MD -MT src/circuit/CMakeFiles/circuit.dir/TransmissionLineSolver.cpp.o -MF src/circuit/CMakeFiles/circuit.dir/TransmissionLineSolver.cpp.o.d -o src/circuit/CMakeFiles/circuit.dir/TransmissionLineSolver.cpp.o -c /pscratch/nroberd/EMPIRE/src/circuit/TransmissionLineSolver.cpp
In file included from /pscratch/nroberd/EMPIRE/TPL/vt/src/vt/utils/mutex/mutex.h(55),
(...)
                 from /pscratch/nroberd/EMPIRE/src/circuit/TransmissionLineSolver.cpp(38):
/pscratch/nroberd/EMPIRE/TPL/vt/src/vt/utils/mutex/omp_mutex.h(57): error: identifier "omp_lock_t" is undefined
    omp_lock_t omp_lock;
    ^

looking at the flags in the long compilation line, the interesting part is:

(...) -DNDEBUG   -fPIC -qopenmp -std=c++14 (...)

Intel documentation says that -qopenmp (which is off by default)

enables the parallelizer to generate multi-threaded code based on OpenMP* directives

so as suspected #if vt_check_enabled(openmp) works fine.

I am looking at cmake/threading_config.cmake and cmake/load_threading_package.cmake to see if VT could be pushing this flag overzealously.

[cz4rs]

removing the beta.10.3 label since this bug didn't make it into the release - I will get back to this after Christmas

[cz4rs]

@lifflander
I have tried to reproduce this using using docker images, but CMake would always rightfully complain about OpenMP missing (and either pick std::thread or exit with error). I couldn't get a configuration that would allow for a compilation error like the one described in the issue.
I have looked at the code extensively, but the threading configuration in CMake looks bulletproof, and feature check vt_check_enabled(openmp) also looks fine.

Long story short: to progress with this, I definitely need CMake part of the build log (configuration step).
Additional information:

• CMake version
• compiler version (should be included in CMake log anyways)

[PhilMiller]

I can try to reproduce this tomorrow, and get you more detail

[cz4rs]

I can try to reproduce this tomorrow, and get you more detail

Great, any additional info will be nice!

[cz4rs]

I can try to reproduce this tomorrow, and get you more detail

@PhilMiller any success with the reproduction?

[PhilMiller]

I tried to reproduce this with Trilinos and EMPIRE built with Intel 18.0.5, and it built successfully. I noticed in the full output that Nick may have been building in a mixed environment between Intel 18.0.2 and 19.0.5. I could vaguely imagine that leading to this sort of trouble, if the wrong headers were available.

[lifflander]

Let's disable threading by default.