/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:224:18: error: unknown type name 'CBLAS_LAYOUT'

[jeffhammond]

Summary

Everything fails like this:

In file included from /opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/gemm_bias.cpp:32:
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/onemkl_blas_helper.hpp:64:8: error: unknown type name 'CBLAS_LAYOUT'
inline CBLAS_LAYOUT convert_to_cblas_layout(oneapi::mkl::layout is_column) {
       ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/onemkl_blas_helper.hpp:65:61: error: use of undeclared identifier 'CBLAS_LAYOUT'
    return is_column == oneapi::mkl::layout::column_major ? CBLAS_LAYOUT::CblasColMajor
                                                            ^

Version

jehammond@dgx-a100-math:/opt/intel/onemkl-cublas/build$ git log -n1
commit 3cb60dd57e5606b71c9fd7e53d42a6999ceb9082 (HEAD -> develop, origin/develop)
Author: Andrew T. Barker <andrew1.barker@intel.com>
Date:   Tue Dec 14 00:22:55 2021 +0000

    Fix empty kernel name issue for old and new compilers. (#150)

Environment

oneMKL works with multiple HW and backend libraries and also depends on the
compiler and build environment. Include
the following information to help reproduce the issue:

• HW you use
  DGX A100 station: AMD 7742 and NVIDIA A100
• Backend library version
  ?
• OS name and version

jehammond@dgx-a100-math:/opt/intel/onemkl-cublas$ uname -a 
Linux dgx-a100-math 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

• Compiler version

DPC++

jehammond@dgx-a100-math:/opt/intel/onemkl-cublas/build$ /opt/intel/dpcpp-cuda/build/install/bin/clang++  --version
clang version 14.0.0 (https://github.com/intel/llvm.git 7b7e044cc73977f5a6a3d434487252bcd45ae3da)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/intel/dpcpp-cuda/build/install/bin

GCC

jehammond@dgx-a100-math:/opt/intel/onemkl-cublas$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-17ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-HskZEa/gcc-9-9.3.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04) 

• CMake output log

jehammond@dgx-a100-math:/opt/intel/onemkl-cublas/build$ cmake .. -DCMAKE_CXX_COMPILER=/opt/intel/dpcpp-cuda/build/install/bin/clang++ \
>          -DCMAKE_C_COMPILER=/opt/intel/dpcpp-cuda/build/install/bin/clang \
>          -DENABLE_CUBLAS_BACKEND=True  \
>          -DENABLE_MKLCPU_BACKEND=False \
>          -DENABLE_MKLGPU_BACKEND=False \
>          -DREF_BLAS_ROOT=/opt/intel/onemkl-cublas/lapack/build/lib
-- CMAKE_BUILD_TYPE: None, set to Release by default
-- The CXX compiler identification is Clang 14.0.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/intel/dpcpp-cuda/build/install/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- TARGET_DOMAINS: blas
-- Looking for dpc++
-- Performing Test is_dpcpp
-- Performing Test is_dpcpp - Success
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /usr/local/cuda-11.4 (found suitable version "11.4", minimum required is "10.0") 
-- Found cuBLAS: /usr/local/cuda-11.4/include  
-- The C compiler identification is Clang 14.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/intel/dpcpp-cuda/build/install/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Deprecation Warning at deps/googletest/CMakeLists.txt:53 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Found PythonInterp: /usr/bin/python (found version "3.8.10") 
-- Found CBLAS: /opt/intel/onemkl-cublas/lapack/build/lib/libcblas.so  
-- Found CBLAS: /opt/intel/onemkl-cublas/lapack/build/lib/libblas.so  
-- Found CBLAS: /usr/include/x86_64-linux-gnu  
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/intel/onemkl-cublas/build
jehammond@dgx-a100-math:/opt/intel/onemkl-cublas/build$ cmake --build .
[  0%] Building CXX object bin/blas/CMakeFiles/onemkl_blas.dir/blas_loader.cpp.o

[  0%] Built target onemkl_blas
[  1%] Linking CXX shared library ../lib/libonemkl.so
[  1%] Built target onemkl
[  2%] Building CXX object bin/blas/backends/cublas/CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_level1.cpp.o
[  2%] Building CXX object bin/blas/backends/cublas/CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_level2.cpp.o

[  3%] Building CXX object bin/blas/backends/cublas/CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_level3.cpp.o
[  3%] Building CXX object bin/blas/backends/cublas/CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_batch.cpp.o
[  3%] Building CXX object bin/blas/backends/cublas/CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_extensions.cpp.o
[  4%] Building CXX object bin/blas/backends/cublas/CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_scope_handle.cpp.o
[  4%] Building CXX object bin/blas/backends/cublas/CMakeFiles/onemkl_blas_cublas_obj.dir/cublas_wrappers.cpp.o
[  4%] Built target onemkl_blas_cublas_obj
[  4%] Linking CXX shared library ../../../../lib/libonemkl_blas_cublas.so
[  4%] Built target onemkl_blas_cublas
[  4%] Building CXX object deps/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[  5%] Linking CXX shared library ../../lib/libgtest.so
[  5%] Built target gtest
[  5%] Building CXX object deps/googletest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[  5%] Linking CXX shared library ../../lib/libgtest_main.so
[  5%] Built target gtest_main
[  5%] Building CXX object tests/unit_tests/blas/extensions/CMakeFiles/blas_extensions_ct.dir/gemm_bias.cpp.o
In file included from /opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/gemm_bias.cpp:32:
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/onemkl_blas_helper.hpp:64:8: error: unknown type name 'CBLAS_LAYOUT'
inline CBLAS_LAYOUT convert_to_cblas_layout(oneapi::mkl::layout is_column) {
       ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/onemkl_blas_helper.hpp:65:61: error: use of undeclared identifier 'CBLAS_LAYOUT'
    return is_column == oneapi::mkl::layout::column_major ? CBLAS_LAYOUT::CblasColMajor
                                                            ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/onemkl_blas_helper.hpp:66:61: error: use of undeclared identifier 'CBLAS_LAYOUT'
                                                          : CBLAS_LAYOUT::CblasRowMajor;
                                                            ^
In file included from /opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/gemm_bias.cpp:33:
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:118:41: error: unknown type name 'CBLAS_LAYOUT'
static inline void copy_mat(T_src &src, CBLAS_LAYOUT layout, CBLAS_TRANSPOSE trans, int row,
                                        ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:138:41: error: unknown type name 'CBLAS_LAYOUT'
static inline void copy_mat(T_src &src, CBLAS_LAYOUT layout, CBLAS_TRANSPOSE trans, int row,
                                        ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:158:41: error: unknown type name 'CBLAS_LAYOUT'
static inline void copy_mat(T_src &src, CBLAS_LAYOUT layout, int row, int col, int ld,
                                        ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:195:41: error: unknown type name 'CBLAS_LAYOUT'
static inline void update_c(T_src &src, CBLAS_LAYOUT layout, CBLAS_UPLO upper_lower, int row,
                                        ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:224:18: error: unknown type name 'CBLAS_LAYOUT'
static void gemm(CBLAS_LAYOUT layout, CBLAS_TRANSPOSE transa, CBLAS_TRANSPOSE transb, const int *m,
                 ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:229:11: error: unknown type name 'CBLAS_LAYOUT'
void gemm(CBLAS_LAYOUT layout, CBLAS_TRANSPOSE transa, CBLAS_TRANSPOSE transb, const int *m,
          ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:261:11: error: unknown type name 'CBLAS_LAYOUT'
void gemm(CBLAS_LAYOUT layout, CBLAS_TRANSPOSE transa, CBLAS_TRANSPOSE transb, const int *m,
          ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:268:11: error: unknown type name 'CBLAS_LAYOUT'
void gemm(CBLAS_LAYOUT layout, CBLAS_TRANSPOSE transa, CBLAS_TRANSPOSE transb, const int *m,
          ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:275:11: error: unknown type name 'CBLAS_LAYOUT'
void gemm(CBLAS_LAYOUT layout, CBLAS_TRANSPOSE transa, CBLAS_TRANSPOSE transb, const int *m,
          ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:283:11: error: unknown type name 'CBLAS_LAYOUT'
void gemm(CBLAS_LAYOUT layout, CBLAS_TRANSPOSE transa, CBLAS_TRANSPOSE transb, const int *m,
          ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:292:18: error: unknown type name 'CBLAS_LAYOUT'
static void gemm(CBLAS_LAYOUT layout, CBLAS_TRANSPOSE transa, CBLAS_TRANSPOSE transb, const int *m,
                 ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:297:11: error: unknown type name 'CBLAS_LAYOUT'
void gemm(CBLAS_LAYOUT layout, CBLAS_TRANSPOSE transa, CBLAS_TRANSPOSE transb, const int *m,
          ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:320:11: error: unknown type name 'CBLAS_LAYOUT'
void gemm(CBLAS_LAYOUT layout, CBLAS_TRANSPOSE transa, CBLAS_TRANSPOSE transb, const int *m,
          ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:344:18: error: unknown type name 'CBLAS_LAYOUT'
static void symm(CBLAS_LAYOUT layout, CBLAS_SIDE left_right, CBLAS_UPLO uplo, const int *m,
                 ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:349:11: error: unknown type name 'CBLAS_LAYOUT'
void symm(CBLAS_LAYOUT layout, CBLAS_SIDE left_right, CBLAS_UPLO uplo, const int *m, const int *n,
          ^
/opt/intel/onemkl-cublas/tests/unit_tests/blas/extensions/../include/reference_blas_templates.hpp:356:11: error: unknown type name 'CBLAS_LAYOUT'
void symm(CBLAS_LAYOUT layout, CBLAS_SIDE left_right, CBLAS_UPLO uplo, const int *m, const int *n,
          ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[2]: *** [tests/unit_tests/blas/extensions/CMakeFiles/blas_extensions_ct.dir/build.make:76: tests/unit_tests/blas/extensions/CMakeFiles/blas_extensions_ct.dir/gemm_bias.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:794: tests/unit_tests/blas/extensions/CMakeFiles/blas_extensions_ct.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

Steps to reproduce

Please check that the issue is reproducible with the latest revision on
master. Include all the steps to reproduce the issue.

cd /opt/intel/onemkl-cublas
git clone https://github.com/Reference-LAPACK/lapack
mkdir -p build && cd build && git clean -dfx
cmake .. -G Ninja -DBUILD_SHARED_LIBS=ON -DCBLAS=ON -DCMAKE_Fortran_COMPILER=gfortran -DCMAKE_C_COMPILER=gcc 
cmake --build .

# oneMKL CUBLAS

cd /opt/intel/onemkl-cublas
mkdir -p build && cd build && git clean -dfx
cmake .. -DCMAKE_CXX_COMPILER=/opt/intel/dpcpp-cuda/build/install/bin/clang++ \
         -DCMAKE_C_COMPILER=/opt/intel/dpcpp-cuda/build/install/bin/clang \
         -DENABLE_CUBLAS_BACKEND=True  \
         -DENABLE_MKLCPU_BACKEND=False \
         -DENABLE_MKLGPU_BACKEND=False \
         -DREF_BLAS_ROOT=/opt/intel/onemkl-cublas/lapack/build/lib
cmake --build .
#ctest
#cmake --install . --prefix /opt/intel/onemkl-cublas/install

[jeffhammond]

The fix is to explicitly add the path to the correct cblas.h file, which should have been taken care of by -DREF_BLAS_ROOT.

>          -DCMAKE_C_FLAGS=-I/opt/intel/onemkl-cublas/lapack/build/include \
>          -DCMAKE_CXX_FLAGS=-I/opt/intel/onemkl-cublas/lapack/build/include \

[andrewtbarker]

I can't reproduce this. As you point out, the key issue seems to be the include path in the configure output:

-- Found CBLAS: /opt/intel/onemkl-cublas/lapack/build/lib/libcblas.so  
-- Found CBLAS: /opt/intel/onemkl-cublas/lapack/build/lib/libblas.so  
-- Found CBLAS: /usr/include/x86_64-linux-gnu  

When I run your reproducing steps I get:

-- Found CBLAS: <snip>/oneMKL/lapack/build/lib/libcblas.so  
-- Found CBLAS: <snip>/oneMKL/lapack/build/lib/libblas.so  
-- Found CBLAS: <snip>/oneMKL/lapack/build/include  

It looks like you've found a workaround, but if you're interested in pushing further can you copy the following into a file try.cmake:

find_path(CBLAS_INCLUDE cblas.h HINTS ${REF_BLAS_ROOT} PATH_SUFFIXES include)
message(CBLAS_INCLUDE="${CBLAS_INCLUDE}")

find_path(CBLAS_INCLUDE_2 cblas.h HINTS ${REF_BLAS_ROOT} PATH_SUFFIXES include NO_DEFAULT_PATH)
message(CBLAS_INCLUDE_2="${CBLAS_INCLUDE_2}")

find_path(CBLAS_INCLUDE_3 cblas.h HINTS ${REF_BLAS_ROOT} PATH_SUFFIXES include NO_CMAKE_PATH)
message(CBLAS_INCLUDE_3="${CBLAS_INCLUDE_3}")

find_path(CBLAS_INCLUDE_4 cblas.h HINTS ${REF_BLAS_ROOT} PATH_SUFFIXES include NO_CMAKE_ENVIRONMENT_PATH)
message(CBLAS_INCLUDE_4="${CBLAS_INCLUDE_4}")

find_path(CBLAS_INCLUDE_5 cblas.h HINTS ${REF_BLAS_ROOT} PATH_SUFFIXES include NO_SYSTEM_ENVIRONMENT_PATH)
message(CBLAS_INCLUDE_5="${CBLAS_INCLUDE_5}")

find_path(CBLAS_INCLUDE_6 cblas.h HINTS ${REF_BLAS_ROOT} PATH_SUFFIXES include NO_CMAKE_SYSTEM_PATH)
message(CBLAS_INCLUDE_6="${CBLAS_INCLUDE_6}")

and then post the output of the following command line:

printenv; cmake -DREF_BLAS_ROOT=../lapack/build -P try.cmake

I'm suspicious that cmake is picking up the wrong include directory somehow from an environment variable.

[zjin-lcf]

I can reproduce the issue posted. I disable the gtests in the CMakefile.txt

[andrewtbarker]

Hi @zjin-lcf , can you post the output you get from the configure step?

[zjin-lcf]

I pasted the relevant info:

-- Found CBLAS: /usr/lib/x86_64-linux-gnu/libcblas.so
-- Found CBLAS: /usr/lib/x86_64-linux-gnu/libblas.so
-- Found CBLAS: /usr/include/x86_64-linux-gnu