How to setup CMakeLists to use OneMKL on a GPU (Intel Arc/Nvidia sm_89)

[salehjg]

Hi there,

I am having trouble using oneMKL with sycl on GPUs.
So far, I have tried to compile and run this repository on Intel Arc A770, Nvidia RTX A2000 (sm_89) and Nvidia gtx1650 with icpx from HPC toolkit 2024.1 with the codeplay plugin for nvidia gpus.
For Intel Arc, I get an exception from oneMKL stating that the given device is not supported.
For Nvidia GPUs (A2000 and gtx1650) I get a SIGSEGV crash on:

sycl::_V1::detail::is_kernel_info_desc<sycl::_V1::info::kernel::num_args>::return_type sycl::_V1::kernel::get_info<sycl::_V1::info::kernel::num_args>() const:

It works on CPUs (on intel 13950HX, intel 6700, and i7 4770K to be exact).

Could anyone please advise me on how to approach this problem? Should I build oneMKL from the source to enable GPU support?

Thank you,
Saleh

[hjabird]

Hi Saleh,
Does 'sycl-ls' run successfully? And what version of CUDA are you using?

[salehjg]

Thank you for the quick reply @hjabird

nvidia-smi
Mon May 27 19:21:23 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.171.04             Driver Version: 535.171.04   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX 2000 Ada Gene...    Off | 00000000:01:00.0 Off |                  N/A |
| N/A   44C    P3              N/A /  60W |      8MiB /  8188MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2558      G   /usr/lib/xorg/Xorg                            4MiB |
+---------------------------------------------------------------------------------------+

And for sycl-ls:

$ sycl-ls
[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2  [2024.17.3.0.08_160000]
[opencl:cpu:1] Intel(R) OpenCL, 13th Gen Intel(R) Core(TM) i9-13950HX OpenCL 3.0 (Build 0) [2024.17.3.0.08_160000]
[ext_oneapi_cuda:gpu:0] NVIDIA CUDA BACKEND, NVIDIA RTX 2000 Ada Generation Laptop GPU 8.9 [CUDA 12.2]

I can write my own kernels with sycl and run them on the gpu without any issues. I just cannot use oneMKL on a GPU queue.

[hjabird]

I'll have more time to look at the linked code tomorrow, but can you check that you're linking against this oneMKL interface library, and not the Intel oneMKL library? I think (but need to confirm tomorrow) for this (oneMKL interface library), you want find_package(oneMKL REQUIRED), but you're using find_package(MKL REQUIRED) in the linked sample. I don't know how this would cause the error you're seeing however.

[salehjg]

I tried find_package(oneMKL) and got this:

Could not find a package configuration file provided by "oneMKL" with any
  of the following names:

    oneMKLConfig.cmake
    onemkl-config.cmake

  Add the installation prefix of "oneMKL" to CMAKE_PREFIX_PATH or set
  "oneMKL_DIR" to a directory containing one of the above files.  If "oneMKL"
  provides a separate development package or SDK, be sure it has been
  installed.

Here is my oneAPI lib/cmake tree:

$ tree /opt/intel/oneapi/2024.1/lib/cmake
/opt/intel/oneapi/2024.1/lib/cmake
├── dnnl
│   ├── dnnl-config.cmake -> ../../../../dnnl/2024.1/lib/cmake/dnnl/dnnl-config.cmake
│   ├── dnnl-config-version.cmake -> ../../../../dnnl/2024.1/lib/cmake/dnnl/dnnl-config-version.cmake
│   ├── dnnl-targets.cmake -> ../../../../dnnl/2024.1/lib/cmake/dnnl/dnnl-targets.cmake
│   ├── dnnl-targets-release.cmake -> ../../../../dnnl/2024.1/lib/cmake/dnnl/dnnl-targets-release.cmake
│   └── FindTBB.cmake -> ../../../../dnnl/2024.1/lib/cmake/dnnl/FindTBB.cmake
├── IntelDPCPP
│   ├── IntelDPCPPConfig.cmake -> ../../../../compiler/2024.1/lib/cmake/IntelDPCPP/IntelDPCPPConfig.cmake
│   ├── IntelDPCPPConfigVersion.cmake -> ../../../../compiler/2024.1/lib/cmake/IntelDPCPP/IntelDPCPPConfigVersion.cmake
│   └── ReadMeDPCPP.txt -> ../../../../compiler/2024.1/lib/cmake/IntelDPCPP/ReadMeDPCPP.txt
├── IntelSYCL
│   ├── IntelSYCLConfig.cmake -> ../../../../compiler/2024.1/lib/cmake/IntelSYCL/IntelSYCLConfig.cmake
│   ├── IntelSYCLConfigVersion.cmake -> ../../../../compiler/2024.1/lib/cmake/IntelSYCL/IntelSYCLConfigVersion.cmake
│   └── ReadMeSYCL.txt -> ../../../../compiler/2024.1/lib/cmake/IntelSYCL/ReadMeSYCL.txt
├── ipp
│   ├── IPPConfig.cmake -> ../../../../ipp/2021.11/lib/cmake/ipp/IPPConfig.cmake
│   ├── IPPConfigVersion.cmake -> ../../../../ipp/2021.11/lib/cmake/ipp/IPPConfigVersion.cmake
│   ├── IPPPathLayout.cmake -> ../../../../ipp/2021.11/lib/cmake/ipp/IPPPathLayout.cmake
│   └── IPPUtils.cmake -> ../../../../ipp/2021.11/lib/cmake/ipp/IPPUtils.cmake
├── ippcp
│   ├── ippcp-config.cmake -> ../../../../ippcp/2021.11/lib/cmake/ippcp/ippcp-config.cmake
│   └── ippcp-config-version.cmake -> ../../../../ippcp/2021.11/lib/cmake/ippcp/ippcp-config-version.cmake
├── mkl
│   ├── MKLConfig.cmake -> ../../../../mkl/2024.1/lib/cmake/mkl/MKLConfig.cmake
│   └── MKLConfigVersion.cmake -> ../../../../mkl/2024.1/lib/cmake/mkl/MKLConfigVersion.cmake
├── oneCCL
│   ├── oneCCLConfig.cmake -> ../../../../ccl/2021.12/lib/cmake/oneCCL/oneCCLConfig.cmake
│   └── oneCCLConfigVersion.cmake -> ../../../../ccl/2021.12/lib/cmake/oneCCL/oneCCLConfigVersion.cmake
├── oneDAL
│   ├── oneDALConfig.cmake -> ../../../../dal/2024.2/lib/cmake/oneDAL/oneDALConfig.cmake
│   └── oneDALConfigVersion.cmake -> ../../../../dal/2024.2/lib/cmake/oneDAL/oneDALConfigVersion.cmake
├── oneDPL
│   ├── oneDPLConfig.cmake -> ../../../../dpl/2022.5/lib/cmake/oneDPL/oneDPLConfig.cmake
│   ├── oneDPLConfigVersion.cmake -> ../../../../dpl/2022.5/lib/cmake/oneDPL/oneDPLConfigVersion.cmake
│   ├── oneDPLWindowsIntelLLVMApply.cmake -> ../../../../dpl/2022.5/lib/cmake/oneDPL/oneDPLWindowsIntelLLVMApply.cmake
│   └── oneDPLWindowsIntelLLVMConfig.cmake -> ../../../../dpl/2022.5/lib/cmake/oneDPL/oneDPLWindowsIntelLLVMConfig.cmake
└── tbb
    ├── TBBConfig.cmake -> ../../../../tbb/2021.12/lib/cmake/tbb/TBBConfig.cmake
    └── TBBConfigVersion.cmake -> ../../../../tbb/2021.12/lib/cmake/tbb/TBBConfigVersion.cmake

10 directories, 29 files

So oneMKL interface is not part of oneAPI HPC toolkit. Am I correct? Then, I have to install this repo and provide the path to it, right?

[hjabird]

Yes, this repo - oneMKL interface library - is not included in the base toolkit.

There are three oneMKLs:

• The oneMKL specification, owned by the UXL foundation. Describes the oneMKL interface.
• This, the oneMKL interface library, owned by the UXL foundation. Lets you use different maths libraries from multiple vendors with a single interface.
• The Intel(R) oneMKL library. Math library for Intel CPU and Intel GPU. Distributed as part of Intel's oneAPI toolkits.

If you want to use this library, the oneMKL interface library, you need to clone the repo, build it, and install it.

[salehjg]

Thank you! @hjabird

I finally got it working with FetchContent: https://github.com/salehjg/my-oneapi-workspace/tree/with-cmake-fetch/e00-onemkl-gemm .

Although, when building and installing the interface onemkl repo, the include dir has a lower priority compared to /opt/intel/oneapi.... I tried set_target_properties(test PROPERTIES NO_SYSTEM_FROM_IMPORTED true) as mentioned in #306 (comment) but it did not make a difference.

Also, it was a bit confusing to find the target names of [open-]onemkl to link against, maybe it is documented somewhere and I have missed it, but for the future reference:

onemkl
onemkl_blas
onemkl_dft
onemkl_lapack
onemkl_sparse_blas
onemkl_rng

[Rbiessy]

Hi @salehjg,

Regarding the include issue did you make sure to adapt the target of set_target_properties so that it uses the target that you are linking against oneMKL? You probably have something like target_link_libraries(<your_target> ... onemkl) somewhere in your CMake, the same target should be used with this call of set_target_properties. Alternatively this should work fine with CMake >=3.25. Let us know if either of these solutions work for you.

I agree that it would be useful to document which targets should be used to integrate with oneMKL CMake. As @hjabird mentioned we will try to improve this as part of the existing issue linked above.
Note that the examples only include the onemkl target for the runtime dispatching examples and the onemkl_<domain>_<backend> targets for compile-time dispatching.

[Rbiessy]

Hi @salehjg do you need more help with this issue?

[salehjg]

@Rbiessy Thank you so much for the help.
I close the issue.