[BUG]: Error loading nvrtc-builtins64_131.dll

[pcgeek86]

Version

1.0.1

Version

13.1

Which installation method(s) does this occur on?

Pip

Describe the bug.

I copied the example code from the cuTile quickstart and tried running it.

https://docs.nvidia.com/cuda/cutile-python/quickstart.html

Project Setup:

uv init .
uv add cuda-tile
uv add numpy
uv add pytest
uv add cup-cuda13x

# Copy the quickstart sample code into main.py

uv run main.py

This results in the following error:

Traceback (most recent call last):
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 731, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends\\cuda\\libs\\nvrtc.pyx", line 125, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\\cuda\\libs\\nvrtc.pyx", line 138, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\\cuda\\libs\\nvrtc.pyx", line 53, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_BUILTIN_OPERATION_FAILURE (7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\git\cutile-test\main.py", line 60, in <module>
    test()
  File "C:\git\cutile-test\main.py", line 37, in test
    a = cp.random.uniform(-1, 1, vector_size)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\random\_distributions.py", line 843, in uniform
    return rs.uniform(low, high, size=size, dtype=dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\random\_generator.py", line 984, in uniform
    rand = self.random_sample(size=size, dtype=dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\random\_generator.py", line 624, in random_sample
    RandomState._mod1_kernel(out)
  File "cupy\\_core\\_kernel.pyx", line 920, in cupy._core._kernel.ElementwiseKernel.__call__
  File "cupy\\_core\\_kernel.pyx", line 945, in cupy._core._kernel.ElementwiseKernel._get_elementwise_kernel
  File "cupy\\_util.pyx", line 67, in cupy._util.memoize.decorator.ret
  File "cupy\\_core\\_kernel.pyx", line 728, in cupy._core._kernel._get_elementwise_kernel
  File "cupy\\_core\\_kernel.pyx", line 82, in cupy._core._kernel._get_simple_elementwise_kernel_from_code
  File "cupy\\_core\\core.pyx", line 2377, in cupy._core.core.compile_with_cache
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 536, in _compile_module_with_cache
    return _compile_with_cache_cuda(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 580, in _compile_with_cache_cuda
    base = _preprocess('', options, arch, backend)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 473, in _preprocess
    result, _ = prog.compile(options)
                ^^^^^^^^^^^^^^^^^^^^^
  File "C:\git\cutile-test\.venv\Lib\site-packages\cupy\cuda\compiler.py", line 750, in compile
    raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: nvrtc: error: failed to open nvrtc-builtins64_131.dll.
  Make sure that nvrtc-builtins64_131.dll is installed correctly.

Minimum reproducible example

Provided above

Relevant log output

Provided above

Full env printout

Doesn't work on Windows 11. I don't have Bash. Bash refers to Windows Subsystem for Linux (WSL), not Bash itself running on Windows.

• Windows 11 Pro
• PNY NVIDIA GeForce RTX 5070 Ti
• NVIDIA Driver 591.74
• NVIDIA CUDA Toolkit 13.1 installed

Output of nvidia-smi:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 591.74                 Driver Version: 591.74         CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5070 Ti   WDDM  |   00000000:01:00.0  On |                  N/A |
|  0%   50C    P0             36W /  300W |    2989MiB /  16303MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Other/Misc.

No response

Contributing Guidelines

• I agree to follow cuTile Python's contributing guidelines
• I have searched the open bugs and have found no duplicates for this bug report

[haijieg]

@pcgeek86 this is a cupy setup issue. Please refer to cupy installation guide. https://docs.cupy.dev/en/stable/install.html

[ZhangZhiPku]

I encountered a similar issue on a B200 GPU, and I resolved it by reinstalling CUDA Toolkit 13.1.
https://developer.nvidia.com/cuda-downloads

[leofang]

NVRTC is the bare minimum for CuPy to run. In CuPy v13.x and earlier, CuPy assumes that a system CUDA Toolkit (CTK) is installed. However, NVRTC and other CTK components can be installed via pip/conda to the user space as well, and to use them today we have a few options:

1. pip install "cuda-toolkit[nvrtc]==13.1.*" cuda-pathfinder after installing cuTile Python and CuPy, and then call from cuda import pathfinder; pathfinder.load_nvidia_dynamic_lib("nvrtc") after importing CuPy. This will force the OS to load NVRTC from a non-system installation location (aka pip and conda) for CuPy to make use of it.
2. Wait for CuPy v14.0.0 which we start supporting non-system CTK installations out of box (currently targeting end of this month).

[leofang]

Sorry, was juggling between meetings and did not give the full picture here. CuPy has been fully working with conda CTK for many years, so here is the Option 3 (if you use conda/pixi to manage your Python env):

3. Install CuPy from conda-forge via conda install -c conda-forge cupy cuda-version=13.1 and then pip install cuTile Python to the same env.