[Bug]: ailed to import from vllm._C with ImportError('/usr/local/lib/python3.8/dist-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch7LibraryC1ENS0_4KindESsSt8optionalIN3c1011DispatchKeyEEPKcj')

[MonolithFoundation]

Your current environment

The output of `python collect_env.py`

import vllm
WARNING 06-13 11:42:20 _custom_ops.py:11] Failed to import from vllm._C with ImportError('/usr/local/lib/python3.8/dist-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch7LibraryC1ENS0_4KindESsSt8optionalIN3c1011DispatchKeyEEPKcj')

🐛 Describe the bug

import vllm
WARNING 06-13 11:42:20 _custom_ops.py:11] Failed to import from vllm._C with ImportError('/usr/local/lib/python3.8/dist-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch7LibraryC1ENS0_4KindESsSt8optionalIN3c1011DispatchKeyEEPKcj')

Does the pip install vllm supports torch2.3.0? If not, why not mentioned it in README?

[youkaichao]

cc @bnellnm , this should be relevant with the recent change of binding system.

[bnellnm]

Older versions of pytorch use c10::optional instead of std::optional, so I'm guessing torch 2.3.0 is not installed in this case. Maybe there's some way to improve the warning message from _custom_ops.py?

@MonolithFoundation can you verify your pytorch version and maybe how you installed vllm?

[MonolithFoundation]

It's not just unenable to found symbols, it just breaks import, and vllm can not be use. Why not just fix it rather than throw another nice warning?

Am using torch2.3 as addressed in my issue template.

[MonolithFoundation]

@youkaichao @bnellnm Hello, would consider fix this for torch2.3 as soon as possible, am current install both with pip from pypi and build source all same error. Note I have cleaned after pypi.

[bnellnm]

@youkaichao @bnellnm Hello, would consider fix this for torch2.3 as soon as possible, am current install both with pip from pypi and build source all same error. Note I have cleaned after pypi.

I'm able to pip install vllm and run from a fresh virtualenv without any trouble. I'm guessing there is something that is not quite right in your environment that is causing problems. Can you provide the results of the collect_env.sh script?

Is it possible you have multiple versions of pytorch installed? Or a custom built version of pytorch 2.3.0?

[MonolithFoundation]

@bnellnm you just mentioned torch2.3.1 have c10:optional difference from older version, does this is ok? Does vllm master branch can build with torch2.3.1? I am pretty sure I didn't have multiple versions either torch or vllm. This issue I noticed serveral toher users claims as well just bere yesrerday

[bnellnm]

@bnellnm you just mentioned torch2.3.1 have c10:optional difference from older version, does this is ok? Does vllm master branch can build with torch2.3.1? I am pretty sure I didn't have multiple versions either torch or vllm. This issue I noticed serveral toher users claims as well just bere yesrerday

vllm is built with torch 2.3.0 and it has that torch::Library symbol defined

From my local install:

> python3 -c "import torch; print(torch.__version__)"
2.3.0+cu121

The symbol is defined in libtorch_cpu.so

> nm -AC --defined-only .local/lib/python3.10/site-packages/torch/lib/*.so | grep "torch::Library::Library"
.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so:0000000001b2a4e0 T torch::Library::Library(torch::Library::Kind, std::string, std::optional<c10::DispatchKey>, char const*, unsigned int)
.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so:0000000001b2a4e0 T torch::Library::Library(torch::Library::Kind, std::string, std::optional<c10::DispatchKey>, char const*, unsigned int)
.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so:00000000010f0518 t torch::Library::Library(torch::Library::Kind, std::string, std::optional<c10::DispatchKey>, char const*, unsigned int) [clone .cold]

And vllm's _C.abi3.so is linked against libtorch_cpu.so

> ldd .local/lib/python3.10/site-packages/vllm/_C.abi3.so  | grep libtorch_cpu
        libtorch_cpu.so => /home/bnellnm/.local/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so (0x00007f56a8e86000)

[MonolithFoundation]

@bnellnm Hi, does this sombol defined?

undefined symbol: _ZN5torch7LibraryC1ENS0_4KindESsSt8optionalIN3c1011DispatchKeyEEPKcj'

[bnellnm]

@bnellnm Hi, does this sombol defined?

undefined symbol: _ZN5torch7LibraryC1ENS0_4KindESsSt8optionalIN3c1011DispatchKeyEEPKcj'

Yes, this symbol is defined in libtorch_cpu.so in pytorch 2.3.0

[titu1992]

Hi @bnellnm
My pytorch version is : 2.3.0+cu121
and my cuda version is : CUDA Version: 12.0

I am still getting the issue while:
from vllm import LLM

Error: RuntimeError: Tried to instantiate class '_core_C.ScalarType', but it does not exist! Ensure that it is registered via torch::class_

[bnellnm]

Hi @bnellnm My pytorch version is : 2.3.0+cu121 and my cuda version is : CUDA Version: 12.0

I am still getting the issue while: from vllm import LLM

Error: RuntimeError: Tried to instantiate class '_core_C.ScalarType', but it does not exist! Ensure that it is registered via torch::class_

@titu1992 It sounds like you've got a problem with your install of vLLM. Check and see if there's a _core_C.abi3.so file that has been added recently. That's where ScalarType is defined.

[gmlwns2000]

I have the same issue here. After I perform pip install torch==2.3.0, I could be able to import vllm and torch.

[bnellnm]

I have the same issue here. After I perform pip install torch==2.3.0, I could be able to import vllm and torch.

If you are on the latest version of vLLM you'll need Pytorch 2.4.