[Installation]: Kimi-VL-A3B failed to be deployed using vllm mirroring

[nigthDust]

Your current environment

model：Kimi-VL-A3B-Thinking
image：vllm-openai：latest
vllm version:0.8.4

1.docker pull vllm/vllm-openai
里面的vllm version:0.8.4
2.docker run --gpus all -v /mnt/data1/LargeLanguageModels/qwen:/model --ipc=host --network=host --name kimi-vl -it --entrypoint vllm/vllm-openai ：latest bash
3. 在容器中如下命令启动大模型

CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \

--port 3000 \
--served-model-name kimi-vl \
--trust-remote-code \
--model /models/Kimi-VL-A3B-Thinking/Kimi-VL-A3B-Thinking \
--tensor-parallel-size 1 \
--max-num-batched-tokens 131072 \
--max-model-len 131072 \
--max-num-seqs 512 \
--limit-mm-per-prompt image=256 \
--disable-mm-preprocessor-cache

出现报错

Traceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 1121, in
uvloop.run(run_server(args))
File "/gwm-tmp/kimi_vl/venv/lib/python3.10/site-packages/uvloop/init.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/gwm-tmp/kimi_vl/venv/lib/python3.10/site-packages/uvloop/init.py", line 61, in wrapper
return await main
File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 1069, in run_server
async with build_async_engine_client(args) as engine_client:
File "/usr/local/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/local/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/gwm-tmp/kimi_vl/vllm/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client_from_engine_args
vllm_config = engine_args.create_engine_config(usage_context=usage_context)
File "/gwm-tmp/kimi_vl/vllm/vllm/engine/arg_utils.py", line 1169, in create_engine_config
model_config = self.create_model_config()
File "/gwm-tmp/kimi_vl/vllm/vllm/engine/arg_utils.py", line 1057, in create_model_config
return ModelConfig(
File "/gwm-tmp/kimi_vl/vllm/vllm/config.py", line 413, in init
self.multimodal_config = self._init_multimodal_config(
File "/gwm-tmp/kimi_vl/vllm/vllm/config.py", line 486, in _init_multimodal_config
raise ValueError("limit_mm_per_prompt is only supported for "
ValueError: limit_mm_per_prompt is only supported for multimodal models.

4.我去掉了limit_mm_per_prompt参数后用如下命令多出尝试全是如下报错
首先是vllm镜像缺少blobfile，手动导入安装解决

CUDA_VISIBLE_DEVICES=3 python3 -m vllm.entrypoints.openai.api_server --port 8888 --served-model-name kimi-vl --trust-remote-code --model moonshotai/Kimi-VL-A3B-Instruct --tensor-parallel-size 1 --max-num-batched-tokens 131072 --max-model-len 131072 --max-num-seqs 512

VLLM_USE_V1_0 CUDA_VISIBLE_DEVICES=3 python3 -m vllm.entrypoints.openai.api_server --port 8888 --served-model-name kimi-vl --trust-remote-code --model moonshotai/Kimi-VL-A3B-Instruct --tensor-parallel-size 1 --max-num-batched-tokens 131072 --max-model-len 131072 --max-num-seqs 512

VLLM_USE_V1_0 CUDA_VISIBLE_DEVICES=3 python3 -m vllm.entrypoints.openai.api_server --port 8888 --served-model-name kimi-vl --trust-remote-code --model moonshotai/Kimi-VL-A3B-Instruct --tensor-parallel-size 1

CUDA_VISIBLE_DEVICES=3 python3 -m vllm.entrypoints.openai.api_server --port 8888 --served-model-name kimi-vl --trust-remote-code --model moonshotai/Kimi-VL-A3B-Instruct --tensor-parallel-size 1

报错如下

ERROR 04-16 02:31:40[engine.py:448] Traceback (most recent call last):
ERROR 04-16 02:31:40[engine.py:448]File "/usr/local/lib/python3.12/dist-packages/vllm/engine/mult iprocess ing/engine .py", line 436, in run_mp_engine
ERROR 04-16 02:31:40[engine.py:448]engine = MQLLMEngine,from_yllm_config(
ERROR 04-16 02:31:40[engine.py:448]^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-16 02:31:40[engine.py:448]File "/usr/local/lib/python3.12/dist-packages/vllm/engine/mult iprocess ing/engine .py", line 128, in from_vllm_config
ERROR 04-16 02:31:40[engine.py:448] return cls(^^^ ERROR 04-16 02:31:40[engine.py:448]
ERROR 04-16 02:31:40[engine.py:448]3File"/usr/local/lib/python3.12/dist-packages/vllm/engine/mult iprocess ing/engine.py",line 82, in _init ERROR 04-16 02:31:40 ERROR 04-16 02:31:40[engine.py:448[engine.py:448] self,engine = LLMEngine(*args, **kwargs)
2025-04-16 17:33215000-0813815712-0014101E8791
ERROR 04-16 02:31:40[engine.py:448]File "/usr/local/lib/python3.12/dist-packages/vllm/engine/lln_engine .py", line 282, in init
ERROR 04-16 02:31:40 ERROR 04-16 02:31:40[engine.py:448][engine.py:448] self.model_executor = executor_class(vllm config=viim canfia.)
ERROR 04-16 02:31:40 ERROR 04-16 02:31:40[engine.py:448][engine.py:448] self._init_executor( ) File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in init
ERROR 04-16 02:31:40[engine.py:448]File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
ERROR 04-16 02:31:40[engine.py:448]answerself.collective_rpc("load_model" )File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor .py", line 56, in collective_rpc
ERROR 04-16 02:31:40[engine.py:448]
ERROR 04-16 02:31:40[engine.py:448]run method(self.driver_worker, method, args,kwargs )
ERROR 04-16 02:31:40[engine.py:448]
ERROR 04-16 02:31:40[engine.py:448]File "/usr/local/lib/python3.12/dist-packages/vllm/utiis.py . line 2378, in run_method
ERROR 04-16 02:31:40[engine.py:448]return func(*args,**kwargs)
ERROR 04-16 02:31:40[engine.py:448
ERROR 04-16 02:31:40[engine.py:448]File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 183, in load_model
ERROR 04-16 02:31:40[engine.py:448self.model_runner.load model()
ERROR 04-16 02:31:40[engine.py:448]File "/usr/local/lib/python3.12/dist-packages/vllm/worker/mddel_runner.py", line 1113, in load_model
ERROR 04-16 02:31:40[engine.py:448]self.model = get_model(vllm_config=solf.vllm_config)
ERROR 04-16 02:31:40ERROR 04-16 02:31:40ERROR 04-16 02:31:40[engine,py:448][engine.py:448[engine.py:448return loader.load model(vilm config=vllm config)File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/init.py". line 14, in get_model
843315712-001410188791
-04-16 17:33
ERROR 04-16 02:31:40ERROR 04-16 02:31:40[engine.py:448[engine.py:448「engine.py:448File "/usr/local/lib/python3. 12/dist-packages/vllm/model_executor/model_loader/loader.py". line 452. in load_model
ERROR 04-16 02:31:40model=initiallize model(yllm config=vlim config)
ERROR 04-16 02:31:40[engine,py:448
ERROR 04-16 02:31:40[engine.py:448File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 123, in initialize_model
ERROR 04-16 02:31:40[engine.py:448model_class, = get_model_architecture(modol_config)
ERROR 04-16 02:31:40ERROR 04-16 02:31:40ERROR 04-16 02:31:40ERROR 04-16 02:31:40[eng ine.py:448[engine.py:448[engine.py:448[engine.py:448n^^^^^^~^^^^^^^^^^^^^^^^^^^^^^^^^architectures = resolve_transformers_arch(model_config, architectures)File */usr/local/lib/python3.12/dist-packages/vllm/model_executor/mode?_loader/utils.py", line 104, in get_model_architecture
ERROR 04-16 02:31:40[engine .py:448File */usr/local/lib/python3. 12/dist-packages/vllm/model_executor/model_loader/utils.py", line 72, in resolve_transformers_arch
ERROR 04-16 02:31:40ERROR 04-16 02:31:40[engine,py:448][engine.py:448]raise ValueError(valueError: KimiyLForConditionalGeneration has no vLLM implementation and the Transformers implementation is not compatible with vLLM. Try settin
9 VLLM USE_V1=0.

How you are installing vllm

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

This model is only on main branch, so you should install the latest code rather than latest release.

[nigthDust]

此模型仅在 main 分支上，因此您应该安装最新的代码，而不是最新版本。
The image is identical to the mian version of git

[DarkLight1337]

vllm-openai:latest is the latest release of vLLM, not the latest code (i.e., main branch). Please install latest code according to https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-wheels

[nigthDust]

vllm-openai:latest是 vLLM 的最新版本，而不是最新的代码（即 main 分支）。请根据 https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-wheels 安装最新的代码

The following commands were used in the container

pip install vllm

pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

Looking in indexes: https://pypi.org/simple, https://wheels.vllm.ai/nightly Requirement already satisfied: vllm in /usr/local/lib/python3.12/dist-packages (0.8.4) Requirement already satisfied: cachetools in /usr/local/lib/python3.12/dist-packages (from vllm) (5.5.2) Requirement already satisfied: psutil in /usr/local/lib/python3.12/dist-packages (from vllm) (7.0.0) Requirement already satisfied: sentencepiece in /usr/local/lib/python3.12/dist-packages (from vllm) (0.2.0) Requirement already satisfied: numpy in /usr/local/lib/python3.12/dist-packages (from vllm) (2.2.4) Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (2.32.3) Requirement already satisfied: tqdm in /usr/local/lib/python3.12/dist-packages (from vllm) (4.67.1) Requirement already satisfied: blake3 in /usr/local/lib/python3.12/dist-packages (from vllm) (1.0.4) Requirement already satisfied: py-cpuinfo in /usr/local/lib/python3.12/dist-packages (from vllm) (9.0.0) Requirement already satisfied: transformers>=4.51.1 in /usr/local/lib/python3.12/dist-packages (from vllm) (4.51.3) Requirement already satisfied: huggingface-hub>=0.30.0 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub[hf_xet]>=0.30.0->vllm) (0.30.2) Requirement already satisfied: tokenizers>=0.19.1 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.21.1) Requirement already satisfied: protobuf in /usr/local/lib/python3.12/dist-packages (from vllm) (4.25.6) Requirement already satisfied: fastapi>=0.115.0 in /usr/local/lib/python3.12/dist-packages (from fastapi[standard]>=0.115.0->vllm) (0.115.12) Requirement already satisfied: aiohttp in /usr/local/lib/python3.12/dist-packages (from vllm) (3.11.16) Requirement already satisfied: openai>=1.52.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (1.74.0) Requirement already satisfied: pydantic>=2.9 in /usr/local/lib/python3.12/dist-packages (from vllm) (2.11.3) Requirement already satisfied: prometheus_client>=0.18.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.21.1) Requirement already satisfied: pillow in /usr/local/lib/python3.12/dist-packages (from vllm) (11.2.1) Requirement already satisfied: prometheus-fastapi-instrumentator>=7.0.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (7.1.0) Requirement already satisfied: tiktoken>=0.6.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.9.0) Requirement already satisfied: lm-format-enforcer<0.11,>=0.10.11 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.10.11) Requirement already satisfied: llguidance<0.8.0,>=0.7.9 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.7.14) Requirement already satisfied: outlines==0.1.11 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.1.11) Requirement already satisfied: lark==1.2.2 in /usr/local/lib/python3.12/dist-packages (from vllm) (1.2.2) Requirement already satisfied: xgrammar==0.1.18 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.1.18) Requirement already satisfied: typing_extensions>=4.10 in /usr/local/lib/python3.12/dist-packages (from vllm) (4.13.2) Requirement already satisfied: filelock>=3.16.1 in /usr/local/lib/python3.12/dist-packages (from vllm) (3.18.0) Requirement already satisfied: partial-json-parser in /usr/local/lib/python3.12/dist-packages (from vllm) (0.2.1.1.post5) Requirement already satisfied: pyzmq in /usr/local/lib/python3.12/dist-packages (from vllm) (26.4.0) Requirement already satisfied: msgspec in /usr/local/lib/python3.12/dist-packages (from vllm) (0.19.0) Requirement already satisfied: gguf>=0.13.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.14.0) Requirement already satisfied: importlib_metadata in /usr/local/lib/python3.12/dist-packages (from vllm) (8.0.0) Requirement already satisfied: mistral_common>=1.5.4 in /usr/local/lib/python3.12/dist-packages (from mistral_common[opencv]>=1.5.4->vllm) (1.5.4) Requirement already satisfied: opencv-python-headless>=4.11.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (4.11.0.86) Requirement already satisfied: pyyaml in /usr/local/lib/python3.12/dist-packages (from vllm) (6.0.2) Requirement already satisfied: six>=1.16.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (1.17.0) Requirement already satisfied: setuptools>=74.1.1 in /usr/local/lib/python3.12/dist-packages (from vllm) (78.1.0) Requirement already satisfied: einops in /usr/local/lib/python3.12/dist-packages (from vllm) (0.8.1) Requirement already satisfied: compressed-tensors==0.9.3 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.9.3) Requirement already satisfied: depyf==0.18.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.18.0) Requirement already satisfied: cloudpickle in /usr/local/lib/python3.12/dist-packages (from vllm) (3.1.1) Requirement already satisfied: watchfiles in /usr/local/lib/python3.12/dist-packages (from vllm) (1.0.5) Requirement already satisfied: python-json-logger in /usr/local/lib/python3.12/dist-packages (from vllm) (3.3.0) Requirement already satisfied: scipy in /usr/local/lib/python3.12/dist-packages (from vllm) (1.15.2) Requirement already satisfied: ninja in /usr/local/lib/python3.12/dist-packages (from vllm) (1.11.1.4) Requirement already satisfied: opentelemetry-sdk<1.27.0,>=1.26.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (1.26.0) Requirement already satisfied: opentelemetry-api<1.27.0,>=1.26.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (1.26.0) Requirement already satisfied: opentelemetry-exporter-otlp<1.27.0,>=1.26.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (1.26.0) Requirement already satisfied: opentelemetry-semantic-conventions-ai<0.5.0,>=0.4.1 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.4.3) Requirement already satisfied: numba==0.61.2 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.61.2) Requirement already satisfied: ray!=2.44.*,>=2.43.0 in /usr/local/lib/python3.12/dist-packages (from ray[cgraph]!=2.44.*,>=2.43.0->vllm) (2.43.0) Requirement already satisfied: torch==2.6.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (2.6.0) Requirement already satisfied: torchaudio==2.6.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (2.6.0) Requirement already satisfied: torchvision==0.21.0 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.21.0) Requirement already satisfied: xformers==0.0.29.post2 in /usr/local/lib/python3.12/dist-packages (from vllm) (0.0.29.post2) Requirement already satisfied: astor in /usr/local/lib/python3.12/dist-packages (from depyf==0.18.0->vllm) (0.8.1) Requirement already satisfied: dill in /usr/local/lib/python3.12/dist-packages (from depyf==0.18.0->vllm) (0.3.9) Requirement already satisfied: llvmlite<0.45,>=0.44.0dev0 in /usr/local/lib/python3.12/dist-packages (from numba==0.61.2->vllm) (0.44.0) Requirement already satisfied: interegular in /usr/local/lib/python3.12/dist-packages (from outlines==0.1.11->vllm) (0.3.3) Requirement already satisfied: jinja2 in /usr/local/lib/python3.12/dist-packages (from outlines==0.1.11->vllm) (3.1.6) Requirement already satisfied: nest_asyncio in /usr/local/lib/python3.12/dist-packages (from outlines==0.1.11->vllm) (1.6.0) Requirement already satisfied: diskcache in /usr/local/lib/python3.12/dist-packages (from outlines==0.1.11->vllm) (5.6.3) Requirement already satisfied: referencing in /usr/local/lib/python3.12/dist-packages (from outlines==0.1.11->vllm) (0.36.2) Requirement already satisfied: jsonschema in /usr/local/lib/python3.12/dist-packages (from outlines==0.1.11->vllm) (4.23.0) Requirement already satisfied: pycountry in /usr/local/lib/python3.12/dist-packages (from outlines==0.1.11->vllm) (24.6.1) Requirement already satisfied: airportsdata in /usr/local/lib/python3.12/dist-packages (from outlines==0.1.11->vllm) (20250224) Requirement already satisfied: outlines_core==0.1.26 in /usr/local/lib/python3.12/dist-packages (from outlines==0.1.11->vllm) (0.1.26) Requirement already satisfied: networkx in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (3.4.2) Requirement already satisfied: fsspec in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (2025.3.2) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (12.4.127) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (12.4.127) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (12.4.127) Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (9.1.0.70) Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (12.4.5.8) Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (11.2.1.3) Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (10.3.5.147) Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (11.6.1.9) Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (12.3.1.170) Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (0.6.2) Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (2.21.5) Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (12.4.127) Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (12.4.127) Requirement already satisfied: triton==3.2.0 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (3.2.0) Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.12/dist-packages (from torch==2.6.0->vllm) (1.13.1) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from sympy==1.13.1->torch==2.6.0->vllm) (1.3.0) Requirement already satisfied: starlette<0.47.0,>=0.40.0 in /usr/local/lib/python3.12/dist-packages (from fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm) (0.46.2) Requirement already satisfied: fastapi-cli>=0.0.5 in /usr/local/lib/python3.12/dist-packages (from fastapi-cli[standard]>=0.0.5; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (0.0.7) Requirement already satisfied: httpx>=0.23.0 in /usr/local/lib/python3.12/dist-packages (from fastapi[standard]>=0.115.0->vllm) (0.28.1) Requirement already satisfied: python-multipart>=0.0.18 in /usr/local/lib/python3.12/dist-packages (from fastapi[standard]>=0.115.0->vllm) (0.0.20) Requirement already satisfied: email-validator>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from fastapi[standard]>=0.115.0->vllm) (2.2.0) Requirement already satisfied: uvicorn>=0.12.0 in /usr/local/lib/python3.12/dist-packages (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (0.34.1) Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub>=0.30.0->huggingface-hub[hf_xet]>=0.30.0->vllm) (24.2) Requirement already satisfied: hf-xet>=0.1.4 in /usr/local/lib/python3.12/dist-packages (from huggingface-hub[hf_xet]>=0.30.0->vllm) (1.0.3) Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.12/dist-packages (from openai>=1.52.0->vllm) (4.9.0) Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.12/dist-packages (from openai>=1.52.0->vllm) (1.9.0) Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.12/dist-packages (from openai>=1.52.0->vllm) (0.9.0) Requirement already satisfied: sniffio in /usr/local/lib/python3.12/dist-packages (from openai>=1.52.0->vllm) (1.3.1) Requirement already satisfied: deprecated>=1.2.6 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-api<1.27.0,>=1.26.0->vllm) (1.2.18) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.12/dist-packages (from importlib_metadata->vllm) (3.21.0) Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc==1.26.0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-exporter-otlp<1.27.0,>=1.26.0->vllm) (1.26.0) Requirement already satisfied: opentelemetry-exporter-otlp-proto-http==1.26.0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-exporter-otlp<1.27.0,>=1.26.0->vllm) (1.26.0) Requirement already satisfied: googleapis-common-protos~=1.52 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-exporter-otlp-proto-grpc==1.26.0->opentelemetry-exporter-otlp<1.27.0,>=1.26.0->vllm) (1.70.0) Requirement already satisfied: grpcio<2.0.0,>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-exporter-otlp-proto-grpc==1.26.0->opentelemetry-exporter-otlp<1.27.0,>=1.26.0->vllm) (1.71.0) Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.26.0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-exporter-otlp-proto-grpc==1.26.0->opentelemetry-exporter-otlp<1.27.0,>=1.26.0->vllm) (1.26.0) Requirement already satisfied: opentelemetry-proto==1.26.0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-exporter-otlp-proto-grpc==1.26.0->opentelemetry-exporter-otlp<1.27.0,>=1.26.0->vllm) (1.26.0) Requirement already satisfied: opentelemetry-semantic-conventions==0.47b0 in /usr/local/lib/python3.12/dist-packages (from opentelemetry-sdk<1.27.0,>=1.26.0->vllm) (0.47b0) Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.12/dist-packages (from pydantic>=2.9->vllm) (0.7.0) Requirement already satisfied: pydantic-core==2.33.1 in /usr/local/lib/python3.12/dist-packages (from pydantic>=2.9->vllm) (2.33.1) Requirement already satisfied: typing-inspection>=0.4.0 in /usr/local/lib/python3.12/dist-packages (from pydantic>=2.9->vllm) (0.4.0) Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.12/dist-packages (from ray!=2.44.*,>=2.43.0->ray[cgraph]!=2.44.*,>=2.43.0->vllm) (8.1.8) Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /usr/local/lib/python3.12/dist-packages (from ray!=2.44.*,>=2.43.0->ray[cgraph]!=2.44.*,>=2.43.0->vllm) (1.1.0) Requirement already satisfied: aiosignal in /usr/local/lib/python3.12/dist-packages (from ray!=2.44.*,>=2.43.0->ray[cgraph]!=2.44.*,>=2.43.0->vllm) (1.3.2) Requirement already satisfied: frozenlist in /usr/local/lib/python3.12/dist-packages (from ray!=2.44.*,>=2.43.0->ray[cgraph]!=2.44.*,>=2.43.0->vllm) (1.5.0) Requirement already satisfied: cupy-cuda12x in /usr/local/lib/python3.12/dist-packages (from ray[cgraph]!=2.44.*,>=2.43.0->vllm) (13.4.1) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests>=2.26.0->vllm) (3.4.1) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.12/dist-packages (from requests>=2.26.0->vllm) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests>=2.26.0->vllm) (2.4.0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.12/dist-packages (from requests>=2.26.0->vllm) (2025.1.31) Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.12/dist-packages (from tiktoken>=0.6.0->vllm) (2024.11.6) Requirement already satisfied: safetensors>=0.4.3 in /usr/local/lib/python3.12/dist-packages (from transformers>=4.51.1->vllm) (0.5.3) Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp->vllm) (2.6.1) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp->vllm) (25.3.0) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.12/dist-packages (from aiohttp->vllm) (6.4.3) Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp->vllm) (0.3.1) Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp->vllm) (1.19.0) Requirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.12/dist-packages (from deprecated>=1.2.6->opentelemetry-api<1.27.0,>=1.26.0->vllm) (1.17.2) Requirement already satisfied: dnspython>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from email-validator>=2.0.0->fastapi[standard]>=0.115.0->vllm) (2.7.0) Requirement already satisfied: typer>=0.12.3 in /usr/local/lib/python3.12/dist-packages (from fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (0.15.2) Requirement already satisfied: rich-toolkit>=0.11.1 in /usr/local/lib/python3.12/dist-packages (from fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (0.14.1) Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx>=0.23.0->fastapi[standard]>=0.115.0->vllm) (1.0.8) Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx>=0.23.0->fastapi[standard]>=0.115.0->vllm) (0.14.0) Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.12/dist-packages (from jinja2->outlines==0.1.11->vllm) (3.0.2) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.12/dist-packages (from jsonschema->outlines==0.1.11->vllm) (2024.10.1) Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.12/dist-packages (from jsonschema->outlines==0.1.11->vllm) (0.24.0) Requirement already satisfied: httptools>=0.6.3 in /usr/local/lib/python3.12/dist-packages (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (0.6.4) Requirement already satisfied: python-dotenv>=0.13 in /usr/local/lib/python3.12/dist-packages (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (1.1.0) Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /usr/local/lib/python3.12/dist-packages (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (0.21.0) Requirement already satisfied: websockets>=10.4 in /usr/local/lib/python3.12/dist-packages (from uvicorn[standard]>=0.12.0; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (15.0.1) Requirement already satisfied: fastrlock>=0.5 in /usr/local/lib/python3.12/dist-packages (from cupy-cuda12x->ray[cgraph]!=2.44.*,>=2.43.0->vllm) (0.8.3) Requirement already satisfied: rich>=13.7.1 in /usr/local/lib/python3.12/dist-packages (from rich-toolkit>=0.11.1->fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (14.0.0) Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from typer>=0.12.3->fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (1.5.4) Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.7.1->rich-toolkit>=0.11.1->fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (3.0.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.7.1->rich-toolkit>=0.11.1->fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (2.19.1) Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich>=13.7.1->rich-toolkit>=0.11.1->fastapi-cli>=0.0.5->fastapi-cli[standard]>=0.0.5; extra == "standard"->fastapi[standard]>=0.115.0->vllm) (0.1.2) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

It indicates that version 0.8.4 of vllm has been installed and is the latest one

[DarkLight1337]

Can you show the full output of collect_env.py?

[nigthDust]

您能显示 的完整输出吗？collect_env.py

`Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 4.0.0
Libc version: glibc-2.35

Python version: 3.12.10 (main, Apr 9 2025, 08:55:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.119.1.el7.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB

Nvidia driver version: 550.90.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz
CPU family: 6
Model: 106
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
Stepping: 6
Frequency boost: enabled
CPU max MHz: 2601.0000
CPU min MHz: 800.0000
BogoMIPS: 5200.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 invpcid_single ssbd mba rsb_ctxsw ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq md_clear pconfig spec_ctrl intel_stibp flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 3 MiB (64 instances)
L1i cache: 2 MiB (64 instances)
L2 cache: 80 MiB (64 instances)
L3 cache: 96 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-31,64-95
NUMA node1 CPU(s): 32-63,96-127
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; Load fences, usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] flashinfer-python==0.2.1.post2+cu124torch2.6
[pip3] numpy==2.2.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] triton==3.2.0
[conda] Could not collect
`

[DarkLight1337]

Seems that there is more to the log

[DarkLight1337]

Oh, also please make sure you use -U flag when installing latest code of vLLM. Otherwise, your current vLLM will not be updated

[nigthDust]

哦，另外，请确保在安装 vLLM 的最新代码时使用 flag。否则，您当前的 vLLM 将不会更新-U
Sorry, I have tried everything but still haven't succeeded in the deployment up to now.

[DarkLight1337]

Can you show the full collect_env.py output? Your previous comment missed the end of the log

[nigthDust]

您能显示_完整的_输出吗？您之前的评论错过了日志的结尾collect_env.py
[pip3] triton==3.2.0
This is followed by no more information.

[DarkLight1337]

Can you check the vLLM version by running pip list | grep vllm?