CUDA Kernel Launch Failed when cudaDeviceSynchronize()

[MurrayC7]

Hello, thanks for the brilliant implementation on CUDA.
I have successfully built this library with these dependencies:

• Ubuntu 16.04
• CMake 3.14.5
• OpenCV 3.4.6
• python 3.6
• Eigen 3.2.10
• CUDA 10.0

When I use your code to generate surface normal with KITTI dataset, I have met the error below.
It seems the error is from cudaDeviceSynchronize().

I'm not good at CUDA :(
So, could u please tell me how to fix it.
Thank you very much!

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/xxx/anaconda3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "generate_normals.py", line 20, in _process
    normals_from_depth(*args)
RuntimeError: Kernel Launch Failed
File: /home/xxx/pycharm_remote/stereo_depth/surface-normal/src/surface_normal.cu
Line Number: 66
Reason: unspecified launch failure
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "generate_normals.py", line 61, in <module>
    generate_normals(sys.argv[1])
  File "generate_normals.py", line 54, in generate_normals
    for _ in tqdm(pool.imap_unordered(_process, process_args), total=len(process_args)):
  File "/home/xxx/anaconda3/lib/python3.6/site-packages/tqdm/std.py", line 1081, in __iter__
    for obj in iterable:
  File "/home/xxx/anaconda3/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
RuntimeError: Kernel Launch Failed
File: /home/xxx/surface-normal/src/surface_normal.cu
Line Number: 66
Reason: unspecified launch failure

[valgur]

That error is rather vague, unfortunately, so I can't immediately say what went wrong there.

Pull the latest version and try the following:

mkdir build && cd build
cmake .. -DBUILD_TEST=y
make
cd ..
cuda-memcheck build/test

This SO answer has some more suggestions on how to potentially debug this.

[MurrayC7]

OK, thanks. I will try first.

[valgur]

@MurrayC7 Did you try cuda-memcheck and was it of any help?