Error moving adata.X to GPU

[joseph-siefert]

Describe the bug
When running either command adata.X = cpx.scipy.sparse.csr_matrix(adata.X) or rsc.get.anndata_to_GPU(adata), I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File <timed exec>:1

File [/mambaforge/envs/rapids_singlecell_24.06/lib/python3.11/site-packages/cupyx/scipy/sparse/_compressed.py:228](http://nruslj-cph510003:8888/mambaforge/envs/rapids_singlecell_24.06/lib/python3.11/site-packages/cupyx/scipy/sparse/_compressed.py#line=227), in _compressed_sparse_matrix.__init__(self, arg1, shape, dtype, copy)
    225 elif scipy_available and scipy.sparse.issparse(arg1):
    226     # Convert scipy.sparse to cupyx.scipy.sparse
    227     x = arg1.asformat(self.format)
--> 228     data = cupy.array(x.data)
    229     indices = cupy.array(x.indices, dtype='i')
    230     indptr = cupy.array(x.indptr, dtype='i')

File [/mambaforge/envs/rapids_singlecell_24.06/lib/python3.11/site-packages/cupy/_creation/from_data.py:53](http://nruslj-cph510003:8888/mambaforge/envs/rapids_singlecell_24.06/lib/python3.11/site-packages/cupy/_creation/from_data.py#line=52), in array(obj, dtype, copy, order, subok, ndmin, blocking)
      7 def array(obj, dtype=None, copy=True, order='K', subok=False, ndmin=0, *,
      8           blocking=False):
      9     """Creates an array on the current device.
     10 
     11     This function currently does not support the ``subok`` option.
   (...)
     51 
     52     """
---> 53     return _core.array(obj, dtype, copy, order, subok, ndmin, blocking)

File cupy[/_core/core.pyx:2408] in cupy._core.core.array()

File cupy[/_core/core.pyx:2435] in cupy._core.core.array()

File cupy[/_core/core.pyx:2578] in cupy._core.core._array_default()

File cupy[/_core/core.pyx:137] in cupy._core.core.ndarray.__new__()

File cupy[/_core/core.pyx:225] in cupy._core.core._ndarray_base._init()

File cupy[/cuda/memory.pyx:738] in cupy.cuda.memory.alloc()

TypeError: 'module' object is not callable

Steps/Code to reproduce bug

import os, sys, time
import cudf
import cupy as cp
import cupyx as cpx
import anndata
import scanpy as sc

from cuml.decomposition import PCA
from cuml.manifold import TSNE
from cuml.cluster import KMeans
from cuml.preprocessing import StandardScaler

import rapids_singlecell as rsc

import warnings
warnings.filterwarnings('ignore', 'Expected ')

import rmm
rmm.reinitialize(
    managed_memory=True, 
    pool_allocator=False,
    devices=0,
)

cp.cuda.set_allocator(rmm.allocators.cupy)

adata = sc.read('adata_file.h5ad')

The above works fine. Running either of the following gives the error
adata.X = cpx.scipy.sparse.csr_matrix(adata.X)
or
rsc.get.anndata_to_GPU(adata)

Expected behavior
Expected the command to be executed and adata.X be moved to GPU without error

Environment details (please complete the following information):

• Environment location: Linux server
• Linux Distro/Architecture: [Ubuntu 22.04 amd64]
• GPU Model/Driver: [A100 and driver 535.129.03]
• CUDA: [11.8 or 12.5]
• Method of Rapids install: [conda, pip or from source]
  • Tested multiple methods of install, including:
    mamba env create -f conda/rsc_rapids_24.06.yml
    and
    mamba env create -f conda/rsc_rapids_24.08.yml
    with the yaml files from here: https://github.com/scverse/rapids_singlecell/tree/main/conda
• Also tested installing rapids with:
  mamba create -n rapids-24.08 -c rapidsai -c conda-forge -c nvidia rapids=24.08 python=3.11 cuda-version=12.5
  followed by:
  pip install 'rapids-singlecell[rapids12]' --extra-index-url=https://pypi.nvidia.com
  or:
  mamba create -n rapids-24.08 -c rapidsai -c conda-forge -c nvidia rapids=24.08 python=3.11 cuda-version=11.8
  followed by:
  pip install 'rapids-singlecell[rapids11]' --extra-index-url=https://pypi.nvidia.com
• All installations give same error

Additional context
Add any other context about the problem here.

[Intron7]

I think this is an issue that you have with cupy and your cuda installation.

[joseph-siefert]

I have done the install exactly as described in the documentation. For 2 different cuda versions. I have no other issues with cuda in these environments. If you think this is an issue with cupy and the cuda installation, how do you suggest I fix it given that I have followed the documentation for installation?

[Intron7]

Can try the yaml file to install rsc. You might have errored you environment with installing rapids 2times

[joseph-siefert]

I have used the yaml files. The 2 installations are in different environments, so that is not the issue

[Intron7]

mamba create -n rapids-24.08 -c rapidsai -c conda-forge -c nvidia rapids=24.08 python=3.11 cuda-version=11.8
followed by:
pip install 'rapids-singlecell[rapids11]' --extra-index-url=https://pypi.nvidia.com

This installs rapids 2times into one environment

[joseph-siefert]

The first thing I tried was this:
mamba env create -f conda/rsc_rapids_24.06.yml

This gave the error. Then, in a separate environment, I tried this:
mamba env create -f conda/rsc_rapids_24.08.yml

That gave the same error. I then went on and tried the other things, which all gave the exact same error. So what is your suggestion?

[Intron7]

Please check out the rapids.ai docs and try to install rapids. This looks to me like a cuda issue

[joseph-siefert]

That is what I did. I installed exactly as described in the documentation

[Intron7]

https://docs.rapids.ai/install/?_gl=1*1ujej7e*_ga*MTE2NTY0MDg5My4xNzI3ODcxMTI0*_ga_RKXFW6CM42*MTcyNzg3MTEyNC4xLjAuMTcyNzg3MTEyNC42MC4wLjA.

Did you checkout rapids or rapids-singlecell?

[joseph-siefert]

Yes, after the installation from yaml gave the errors, I then tried installing with conda per the documentation from that same link and got the same exact error.

[Intron7]

Please make an issue and connect with people on the rapids team. They might be able to help you with your environment setup and cuda installation. Rapids-singlecell builds on top of rapids and cupy

[joseph-siefert]

Will do. One last question. Assuming I install rapids, what is the procedure to install rapids-singlecell if you are suggesting that:

pip install 'rapids-singlecell[rapids11]' --extra-index-url=https://pypi.nvidia.com

after:

mamba create -n rapids-24.08 -c rapidsai -c conda-forge -c nvidia rapids=24.08 python=3.11 cuda-version=11.8

is problematic

[Intron7]

No just

pip install rapids-singlecell

You already installed rapids with conda.