Digits 2.0 with GTX 1080 not working

[hanyunfan]

Error message is:

ERROR: Check failed: status == CURAND_STATUS_SUCCES(201 VS 0)
CURAND_STATUS_LAUNCH_FAILURE

Picture attached.

[IsaacYangSLA]

@hanyunfan , can you upgrade your cuda to 8.0? The curand error is likely due to GTX 1080 and cuda 7.5 or earlier.

[hanyunfan]

We are using GTX1080 driver and cuda8.0 toolkit. the default driver in CUDA8 wasn't working before. Do you suggest to use that one?

[IsaacYangSLA]

@hanyunfan , thanks for that information. You can use newer driver than the one bundled with CUDA8. May I ask if your rebuilt your caffe after installing cuda8? You can also check if your caffe is using cuda8 libraries by typing ldd which caffe in your terminal. You should be able to see lines like libcurand.so.7.5 => /usr/local/cuda/lib64/libcurand.so.7.5 (0x00007fcc5b0a8000)

In my case, I am still using cuda 7.5, so my caffe was built with libcurand.so.7.5. Yours should have 8.0.

[hanyunfan]

I see, it is still 7.5. Is there a fast way that I can rebuilt caffe with little changes.

[IsaacYangSLA]

If your caffe was originally built from source codes, just make sure this line CUDA_DIR := /usr/local/cuda in Makefile.config points to cuda8, and make clean && make all && make test && make runtest && make pycaffe. There are experts in Caffe's github repository who can give better answers. Mine is not the optimal approach. It just works for me.

[lukeyeager]

Do you need to use 2.0 for some reason? The latest release is 4.0.
https://github.com/NVIDIA/DIGITS/releases

[hanyunfan]

update. After recompile with CUDA 8, both 2.0 and 4.0 works fine.