Installation issue with libdevice

[kayhan-batmanghelich]

Hi Everyone,

I have a new machine that I tried to install JAX on. I still have an issue with libdevice. I followed the instruction in the closed issue about this but I still get the error :

[...]
RuntimeError: Internal: libdevice not found at ./libdevice.10.bc

cuda is installed and I made a soft link. Here is some more info:

(jax) kayhan@lambda-dual:~$ whereis -b cuda
cuda: /usr/include/cuda.h /usr/include/cuda

(jax) kayhan@lambda-dual:~$ ls -lt /usr/local/cuda-11.1
lrwxrwxrwx 1 root root 39 Mar 11 15:16 /usr/local/cuda-11.1 -> /usr/lib/nvidia-cuda-toolkit/libdevice/

(jax) kayhan@lambda-dual:~$ ls -lt /usr/lib/nvidia-cuda-toolkit/libdevice/
total 464
lrwxrwxrwx 1 root root     13 Apr 15 15:21 cuda -> /usr/lib/cuda
-rw-r--r-- 1 root root 471124 Oct 16 13:42 libdevice.10.bc

I tried setting XLA flag but it was not successful:

(jax) kayhan@lambda-dual:~$ export XLA_FLAGS="--xla_gpu_cuda_data_dir=/usr/lib/nvidia-cuda-toolkit/libdevice/"
(jax) kayhan@lambda-dual:~$ ipython
[...]
RuntimeError: Internal: libdevice not found at ./libdevice.10.bc

Any idea how to fix it?

[davisyoshida]

I get the following output from a find command:

> find /usr/local/cuda-11.2/ -name libdevice*
/usr/local/cuda-11.2/nvvm-prev/libdevice
/usr/local/cuda-11.2/nvvm-prev/libdevice/libdevice.10.bc
/usr/local/cuda-11.2/nvvm/libdevice
/usr/local/cuda-11.2/nvvm/libdevice/libdevice.10.bc

So maybe try making a nvvm subdirectory in the cuda-11.1 dir then put the libdevice symlink in there?

[davisyoshida]

@pharringtonp19 well it seemed pretty specific what kayhan was asking, but the point was that the symlinks they were making didn't actually put libdevice where JAX was looking for it.

[pharringtonp19]

Thanks @davisyoshida for the response. I am not as well versed in this stuff as I would like to be, so I was wondering if you could elaborate on what you meant by "making a nvvm subdirectory in the cuda-11.1 dir then put the libdevice symlink". I think I am facing the same issues/problems as kayhan, so I was wondering if you could expand on how one would make a nvvm subdirectory and then put the libdevice symlink in there. -- Thanks for your time

[davisyoshida]

@pharringtonp19 Well basically what I was getting at was that kayhan's question was about why the symlinks they made aren't working, so if you aren't already manually making symlinks you're unlikely to be having that problem. If you're just getting libdevice not found errors, it's likely that you don't have the right CUDA version or it's installed in an unexpected location. What's your CUDA setup currently?

[pharringtonp19]

sorry for the delay -- @davisyoshida

I think I have a similar machine as kayhan-batmanghelich which is why I am curious about the exact steps he took to solve this issue.

I followed his initial steps and seem to get a similar error--

pharrington19@deep-lambda:~$ whereis -b cuda
cuda: /usr/include/cuda.h /usr/include/cuda

pharrington19@deep-lambda:~$ ls -lt /usr/local/cuda-11.1
lrwxrwxrwx 1 root root 39 Apr 27 16:55 /usr/local/cuda-11.1 -> /usr/lib/nvidia-cuda-toolkit/libdevice/

pharrington19@deep-lambda`:~$ ls -lt /usr/lib/nvidia-cuda-toolkit/libdevice/'
total 464
-rw-r--r-- 1 root root 471124 Oct 16  2020 libdevice.10.bc

export XLA_FLAGS="--xla_gpu_cuda_data_dir=/usr/lib/nvidia-cuda-toolkit/libdevice/"
...
RuntimeError: Internal: libdevice not found at ./libdevice.10.bc

[davisyoshida]

Okay my best guess is to try this:

export XLA_FLAGS="--xla_gpu_cuda_data_dir=/usr/lib/cuda"
ln -sfn /usr/local/cuda /usr/lib/cuda-11.1

If this breaks anything, you can restore the old symlink you had at /usr/local/cuda-11.1 by doing

ln -sfn /usr/lib/nvidia-cuda-toolkit/libdevice/ /usr/local/cuda-11.1

I don't understand what was meant to be accomplished by making the symlinks as they originally were, so there's a good chance my advice breaks something.