Building NekRS with OpenMP support

[kvoronin-intel]

Hello all!

I am new to NekRS and while I successfully built NekRS following the instructions in readme, when I tried to run the FDM benchmark on a CPU, I get only serial performance.

Also, I noticed that the OCCA backend is in my case serial, per
https://github.com/argonne-lcf/nekRS/blob/oneapi-build/CMakeLists.txt#L330

Is there something I can/should do to enable OpenMP support in NekRS properly and get scalability? Or is this benchmark not multi-threaded (but then what's the point of reporting OMPthreads?)?

I am also not sure whether "active occa mode: Serial" is right for my purpose but I don't know how to change it.

Any advice would be appreciated!

Runline:

export OMP_NUM_THREADS=28
mpirun -n 1 -genv OMP_NUM_THREADS=${OMP_NUM_THREADS} nekrs-bench-fdm --p-order 7 --elements 6180 --backend CPU --iterations 10000

Output (same GFLOP/s for 1, 14 28 OpenMP threads):

Initializing device
active occa mode: Serial
Error in kernel compared to reference implementation 0: 0
fdm:MPItasks=1 OMPthreads=28 NRepetitions=226 N=7 Nelements=6180 elapsed time=0.0439418 wordSize=64 GDOF/s=0.0482397 GB/s=1.94421 GFLOPS/s=6.98477 kernelVer=0

Initializing device
active occa mode: Serial
Error in kernel compared to reference implementation 0: 0
fdm:MPItasks=1 OMPthreads=14 NRepetitions=230 N=7 Nelements=6180 elapsed time=0.0431859 wordSize=64 GDOF/s=0.049084 GB/s=1.97824 GFLOPS/s=7.10703 kernelVer=0

Initializing device
active occa mode: Serial
Error in kernel compared to reference implementation 0: 0
fdm:MPItasks=1 OMPthreads=1 NRepetitions=227 N=7 Nelements=6180 elapsed time=0.0438715 wordSize=64 GDOF/s=0.048317 GB/s=1.94733 GFLOPS/s=6.99596 kernelVer=0