memadvise.cpp test fails on CUDA BE after #3365

[vladimirlaz]

Describe the bug
intel/llvm-test-suite/blob/intel/SYCL/USM/memadvise.cpp test start to fail with the following output:

[2021-04-13T23:31:43.662Z] FAIL: SYCL :: USM/memadvise.cpp (481 of 501)

[2021-04-13T23:31:43.662Z] ******************** TEST 'SYCL :: USM/memadvise.cpp' FAILED ********************

[2021-04-13T23:31:43.662Z] Script:

[2021-04-13T23:31:43.662Z] --

[2021-04-13T23:31:43.662Z] : 'RUN: at line 1';    /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm.obj/bin/clang++     -fsycl -fsycl-targets=nvptx64-nvidia-cuda-sycldevice /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm-test-suite/SYCL/USM/memadvise.cpp -o /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out

[2021-04-13T23:31:43.662Z] : 'RUN: at line 2';   true /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out

[2021-04-13T23:31:43.662Z] : 'RUN: at line 3';   true /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out

[2021-04-13T23:31:43.662Z] : 'RUN: at line 4';    env SYCL_DEVICE_FILTER=cuda:gpu,host  /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out

[2021-04-13T23:31:43.662Z] --

[2021-04-13T23:31:43.662Z] Exit Code: -6

[2021-04-13T23:31:43.662Z] 

[2021-04-13T23:31:43.662Z] Command Output (stdout):

[2021-04-13T23:31:43.662Z] --

[2021-04-13T23:31:43.662Z] $ ":" "RUN: at line 1"

[2021-04-13T23:31:43.662Z] note: command had no output on stdout or stderr

[2021-04-13T23:31:43.662Z] $ "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm.obj/bin/clang++" "-fsycl" "-fsycl-targets=nvptx64-nvidia-cuda-sycldevice" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm-test-suite/SYCL/USM/memadvise.cpp" "-o" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out"

[2021-04-13T23:31:43.662Z] note: command had no output on stdout or stderr

[2021-04-13T23:31:43.662Z] $ ":" "RUN: at line 2"

[2021-04-13T23:31:43.662Z] note: command had no output on stdout or stderr

[2021-04-13T23:31:43.662Z] $ "true" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out"

[2021-04-13T23:31:43.662Z] note: command had no output on stdout or stderr

[2021-04-13T23:31:43.662Z] $ ":" "RUN: at line 3"

[2021-04-13T23:31:43.662Z] note: command had no output on stdout or stderr

[2021-04-13T23:31:43.662Z] $ "true" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out"

[2021-04-13T23:31:43.662Z] note: command had no output on stdout or stderr

[2021-04-13T23:31:43.662Z] $ ":" "RUN: at line 4"

[2021-04-13T23:31:43.662Z] note: command had no output on stdout or stderr

[2021-04-13T23:31:43.662Z] $ "env" "SYCL_DEVICE_FILTER=cuda:gpu,host" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out"

[2021-04-13T23:31:43.662Z] # command stderr:

[2021-04-13T23:31:43.662Z] memadvise.cpp.tmp1.out: /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm.src/sycl/plugins/cuda/pi_cuda.cpp:329: _pi_event::_pi_event(pi_command_type, pi_context, pi_queue): Assertion `type != PI_COMMAND_TYPE_USER' failed.

[2021-04-13T23:31:43.662Z] 

[2021-04-13T23:31:43.662Z] error: command failed with exit status: -6

[2021-04-13T23:31:43.662Z] 

[2021-04-13T23:31:43.662Z] --

[2021-04-13T23:31:43.662Z] 

To Reproduce
The problem can be reproduced following instructions under https://github.com/intel/llvm-test-suite/blob/intel/SYCL/README.md

git clone https://github.com/intel/llvm-test-suite
cd llvm-test-suite
mkdir build
cd build
cmake -G "Ninja" -DCMAKE_CXX_COMPILER=<path_to_intel_sycl_compiler_with cuda_support>/clang++  -DTEST_SUITE_SUBDIRS=SYCL -DTEST_SUITE_LIT=<path_to_lit_fw>/lit.py -DCHECK_SYCL_ALL="cuda:gpu" -DTEST_SUITE_COLLECT_CODE_SIZE=OFF ..
# Build/run all tests
ninja check-sycl-all
# Run single test
llvm-lit --param target_devices=gpu --param sycl_be=cuda SYCL/USM/memadvise.cpp

Environment (please complete the following information):

• OS: ubuntu 18.04
• Target device and vendor: NVidia GPU under CUDA BE
• DPC++ version: All verasion after [SYCL][CUDA] Implemented cuda_piextUSMEnqueueMemAdvise #3365
• Dependencies version: NVIDIA-SMI 435.21 Driver Version: 435.21 CUDA Version: 10.1

[bader]

@vladimirlaz, I expect 3269c47 to fix this bug. Could you check, please?

[vladimirlaz]

according to postcommit testing for the commit. The test is still failing but fail symptoms changed to:

[2021-04-14T20:43:00.833Z] ******************** TEST 'SYCL :: USM/memadvise.cpp' FAILED ********************

[2021-04-14T20:43:00.833Z] Script:

[2021-04-14T20:43:00.833Z] --

[2021-04-14T20:43:00.833Z] : 'RUN: at line 1';    /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm.obj/bin/clang++     -fsycl -fsycl-targets=nvptx64-nvidia-cuda-sycldevice /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm-test-suite/SYCL/USM/memadvise.cpp -o /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out

[2021-04-14T20:43:00.833Z] : 'RUN: at line 2';   true /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out

[2021-04-14T20:43:00.834Z] : 'RUN: at line 3';   true /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out

[2021-04-14T20:43:00.834Z] : 'RUN: at line 4';    env SYCL_DEVICE_FILTER=cuda:gpu,host  /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out

[2021-04-14T20:43:00.834Z] --

[2021-04-14T20:43:00.834Z] Exit Code: -6

[2021-04-14T20:43:00.834Z] 

[2021-04-14T20:43:00.834Z] Command Output (stdout):

[2021-04-14T20:43:00.834Z] --

[2021-04-14T20:43:00.834Z] $ ":" "RUN: at line 1"

[2021-04-14T20:43:00.834Z] note: command had no output on stdout or stderr

[2021-04-14T20:43:00.834Z] $ "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm.obj/bin/clang++" "-fsycl" "-fsycl-targets=nvptx64-nvidia-cuda-sycldevice" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm-test-suite/SYCL/USM/memadvise.cpp" "-o" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out"

[2021-04-14T20:43:00.834Z] note: command had no output on stdout or stderr

[2021-04-14T20:43:00.834Z] $ ":" "RUN: at line 2"

[2021-04-14T20:43:00.834Z] note: command had no output on stdout or stderr

[2021-04-14T20:43:00.834Z] $ "true" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out"

[2021-04-14T20:43:00.834Z] note: command had no output on stdout or stderr

[2021-04-14T20:43:00.834Z] $ ":" "RUN: at line 3"

[2021-04-14T20:43:00.834Z] note: command had no output on stdout or stderr

[2021-04-14T20:43:00.834Z] $ "true" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out"

[2021-04-14T20:43:00.834Z] note: command had no output on stdout or stderr

[2021-04-14T20:43:00.834Z] $ ":" "RUN: at line 4"

[2021-04-14T20:43:00.834Z] note: command had no output on stdout or stderr

[2021-04-14T20:43:00.834Z] $ "env" "SYCL_DEVICE_FILTER=cuda:gpu,host" "/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/USM/Output/memadvise.cpp.tmp1.out"

[2021-04-14T20:43:00.834Z] # command stderr:

[2021-04-14T20:43:00.834Z] 

[2021-04-14T20:43:00.834Z] PI CUDA ERROR:

[2021-04-14T20:43:00.834Z] 	Value:           1

[2021-04-14T20:43:00.834Z] 	Name:            CUDA_ERROR_INVALID_VALUE

[2021-04-14T20:43:00.834Z] 	Description:     invalid argument

[2021-04-14T20:43:00.834Z] 	Function:        cuda_piextUSMEnqueueMemAdvise

[2021-04-14T20:43:00.834Z] 	Source Location: /workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm.src/sycl/plugins/cuda/pi_cuda.cpp:4465

[2021-04-14T20:43:00.834Z] 

[2021-04-14T20:43:00.834Z] terminate called after throwing an instance of 'cl::sycl::runtime_error'

[2021-04-14T20:43:00.834Z]   what():  Native API failed. Native API returns: -30 (CL_INVALID_VALUE) -30 (CL_INVALID_VALUE)

[2021-04-14T20:43:00.834Z] 

[2021-04-14T20:43:00.834Z] error: command failed with exit status: -6

[2021-04-14T20:43:00.834Z] 

[2021-04-14T20:43:00.834Z] --

[2021-04-14T20:43:00.834Z] 

[bader]

@nyalloc, does this test work for you?

[nyalloc]

Memadvise test is not correct. The value passed into memadvise is backend-specific, this test assumes 0 is a valid value. For CUDA this is not the case. The valid values for CUDA are 1 through 6.

I think it would be a good idea to modify these tests so that the enumerations passed to memadvise are driven through the preprocessor, so if the tests are targetting CUDA, pass one of the valid enumerations for CUDA, if if the tests are targetting OpenCL, pass one of the valid enumerations for OpenCL, etc.

[bader]

@nyalloc, thanks for sharing this insight.

@vladimirlaz, could you update the test, please?

[nyalloc]

No problem

[vladimirlaz]

Fixed in intel/llvm-test-suite#240