BUG: module 'cuda.ccudart' has no attribute '__pyx_capi__'

[leofang]

tl;dr: This error happens with cuda-python 12.6.1 & 11.8.4, the first patch releases with the new layout (#75). To tentatively work around this error, please downgrade cuda-python to either 12.6.0 or below (if you’re using 12.x) or 11.8.3 or below (if you’re using 11.x).

This only impacts projects satisfying all of the following conditions:

1. access CUDA Python through the Cython level (=cimport); through the Python level (=import) it should work fine
2. were built against CUDA Python with the old layout (12.6.0 and below, or 11.8.3 and below)
3. are running against the latest CUDA Python patch releases with the new layout (12.6.1 or 11.8.4)

Reproducer:

# test_cuda_python_abi.pyx
# distutils: include_dirs = /usr/local/cuda-12.6/include

from cuda.ccudart cimport cudaGetDevice


def test_function():
    cdef int dev
    print(cudaGetDevice(&dev))
    print(dev)

Steps:

1. pip install "cuda-python==12.6.0" (build against the "old layout")
2. cythonize -i -3 -f test_cuda_python_abi.pyx
3. pip install "cuda-python==12.6.1" (run against the "new layout")
4. python -c "import test_cuda_python_abi; test_cuda_python_abi.test_function()"

Error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "test_cuda_python_abi.pyx", line 1, in init test_cuda_python_abi
    # distutils: include_dirs = /usr/local/cuda-12.6/include
AttributeError: module 'cuda.ccudart' has no attribute '__pyx_capi__'

[leofang]

It seems our Cython trampoline modules cuda.{ccuda,ccudart,cnvrtc} do not call __Pyx_ExportFunction to re-export the Cython symbols. This function is responsible for populating the module attribute __pyx_capi__.

[jakirkham]

From the Cython FAQ:

How do I compile Cython with subpackages?

Answer: It’s highly recommended to arrange Cython modules in exactly the same Python package structure as the Python parts of the code base. As long as you don’t keep your Cython code in unusual places, everything should just work.

This is in part due to the fact that fully qualified names are resolved at compile time, and moving .so files around or adding __init__ files between the Cython compile and the Python runtime invocation means that cimports and imports may resolve differently. Failure to do this may result in errors like .pxd files not found or 'module' object has no attribute '__pyx_capi__'.

IOW perhaps this is related to the layout?

[leofang]

Update: We've looked at this internally, and we think monkey-patching __pyx_capi__ like this

diff --git a/cuda_bindings/cuda/ccudart.pyx b/cuda_bindings/cuda/ccudart.pyx
index b32eece..3cef081 100644
--- a/cuda_bindings/cuda/ccudart.pyx
+++ b/cuda_bindings/cuda/ccudart.pyx
@@ -5,3 +5,8 @@ cdef extern from *:
     #pragma message ( "The cuda.ccudart module is deprecated and will be removed in a future release, " \
                       "please switch to use the cuda.bindings.cyruntime module instead." )
     """
+
+
+from cuda.bindings import cyruntime
+__pyx_capi__ = cyruntime.__pyx_capi__
+del cyruntime

for all Cython-layer trampoline modules cuda.{ccuda,ccudart,cnvrtc} (which will be removed in a future release #161) seems to be way to go.

Despite the purpose of __pyx_capi__ is undocumented and considered a Cython internal...

• Questions about __pyx_capi__ use/stability. cython/cython#3077 (by Numba lead)

it's been quite stable over the years. It’s taught in the wild:

• https://numba.readthedocs.io/en/stable/extending/high-level.html#importing-cython-functions
• https://pythran.readthedocs.io/en/latest/examples/Third%20Party%20Libraries.html#With-Cython

... and also used in the wild

• numba/_helperlib.c
• jaxlib/cpu/lapack.cc
• scipy/_lib/deprecation.py
• src/binding/petsc4py/src/cython.h

Therefore we believe there's no harm in applying a hacky solution like the above to the short-lived trampolines.

[leofang]

Vlad is working on a hot fix release 11.8.5 & 12.6.2.

[vzhurba01]

https://pypi.org/project/cuda-python/12.6.2/
https://pypi.org/project/cuda-python/11.8.5/
https://anaconda.org/nvidia/cuda-python/files?version=12.6.2
https://anaconda.org/nvidia/cuda-python/files?version=11.8.5

Hot fix for linux-x64 and linux-aarch64 is now posted.

I ran into a build error with Windows where the package_data wildcards are not respected as they do with the Linux counter part. In particular I get an error: can't copy 'cuda\bindings': doesn't exist or not a regular file which is odd given the fact that all package_data wildcards explicitly state their extension. While this issue is being triaged, I'm holding off on creating the new tags.

[vzhurba01]

The hot fix for Windows is now posted.

https://github.com/NVIDIA/cuda-python/releases/tag/v12.6.2
https://github.com/NVIDIA/cuda-python/releases/tag/v11.8.5

The two tags for this patch are created and this issue can now be closed.

[leofang]

Thanks @vzhurba01 for verifying the fix and preparing new packages, @vyasr for reporting the issue offline.

[jakirkham]

Working with Leo and Bradley to get these on conda-forge too:

• cuda-python 12.6.2 conda-forge/cuda-python-feedstock#94
• cuda-python 11.8.5 conda-forge/cuda-python-feedstock#92