ImportError: libcudart.so.12

[tranhoangnguyen03]

Was working yesterday.
Today it gives the following error:

from vllm import cuda_utils
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory

I've tried turning it off and on again multiple times.

[WoosukKwon]

Hi @tranhoangnguyen03 , we just released a new version (v0.2.2) which depends on PyTorch 2.1 and CUDA 12.1. This might cause error if you environment does not have CUDA 12.1.

1. Please upgrade PyTorch using pip install --upgrade torch and see if the error persists.
2. If it does not work, please install vLLM + CUDA 11.8 following the instructions here.

[tranhoangnguyen03]

In case anyone else runs into the same trouble as I did. Here's how to make it work on Google Colab:

!pip -q install --upgrade fschat accelerate autoawq vllm
!pip install torch==2.1.0+cu121 torchvision==0.16.0+cu121 torchaudio==2.1.0 torchtext==0.16.0+cpu torchdata==0.7.0 --index-url https://download.pytorch.org/whl/cu121

[Tanio253]

In case anyone else runs into the same trouble as I did. Here's how to make it work on Google Colab:

!pip -q install --upgrade fschat accelerate autoawq vllm
!pip install torch==2.1.0+cu121 torchvision==0.16.0+cu121 torchaudio==2.1.0 torchtext==0.16.0+cpu torchdata==0.7.0 --index-url https://download.pytorch.org/whl/cu121

Thank you! Super useful.

[Hukongtao]

Hi @tranhoangnguyen03 , we just released a new version (v0.2.2) which depends on PyTorch 2.1 and CUDA 12.1. This might cause error if you environment does not have CUDA 12.1.

1. Please upgrade PyTorch using pip install --upgrade torch and see if the error persists.
2. If it does not work, please install vLLM + CUDA 11.8 following the instructions here.

Is there a cuda11.7 version?

[PeterXiaTian]

Hi @tranhoangnguyen03 , we just released a new version (v0.2.2) which depends on PyTorch 2.1 and CUDA 12.1. This might cause error if you environment does not have CUDA 12.1.

1. Please upgrade PyTorch using pip install --upgrade torch and see if the error persists.
2. If it does not work, please install vLLM + CUDA 11.8 following the instructions here.

Is there a cuda11.7 version?

Is there a cuda11.1 version?

[datalee]

CUDA 12.1.

Is there a cuda11.7 version?

[AWangji]

Hi @tranhoangnguyen03 , we just released a new version (v0.2.2) which depends on PyTorch 2.1 and CUDA 12.1. This might cause error if you environment does not have CUDA 12.1.

1. Please upgrade PyTorch using pip install --upgrade torch and see if the error persists.
2. If it does not work, please install vLLM + CUDA 11.8 following the instructions here.

Is there a cuda11.7 version?

same question, is there anyone solve it?

[DarkLight1337]

For other CUDA versions, please build vLLM from source.

[qinhuangdaoStation]

In case anyone else runs into the same trouble as I did. Here's how to make it work on Google Colab:

!pip -q install --upgrade fschat accelerate autoawq vllm
!pip install torch==2.1.0+cu121 torchvision==0.16.0+cu121 torchaudio==2.1.0 torchtext==0.16.0+cpu torchdata==0.7.0 --index-url https://download.pytorch.org/whl/cu121

Excuse me, what is the version of vllm?

[DarkLight1337]

In case anyone else runs into the same trouble as I did. Here's how to make it work on Google Colab:

!pip -q install --upgrade fschat accelerate autoawq vllm
!pip install torch==2.1.0+cu121 torchvision==0.16.0+cu121 torchaudio==2.1.0 torchtext==0.16.0+cpu torchdata==0.7.0 --index-url https://download.pytorch.org/whl/cu121

Excuse me, what is the version of vllm?

That is a very old version. I recommend you install the latest vLLM on a clean environment.