[Bug]: 300I DUO OUT OF MEMORY

[bouyeijiang]

Your current environment

Collecting environment information...
PyTorch version: 2.5.1
Is debug build: False

OS: Kylin Linux Advanced Server V10 (Sword) (aarch64)
GCC version: (GCC) 7.3.0
Clang version: Could not collect
CMake version: version 4.0.3
Libc version: glibc-2.28

Python version: 3.11.13 (main, Jul 17 2025, 10:24:39) [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-4.19.90-25.48.v2101.ky10.aarch64-aarch64-with-glibc2.28

CPU:
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 1
Core(s) per socket: 64
Socket(s): 2
NUMA node(s): 4
Vendor ID: HiSilicon
Model: 0
Model name: Kunpeng-920
Stepping: 0x1
BogoMIPS: 200.00
L1d cache: 8 MiB
L1i cache: 8 MiB
L2 cache: 64 MiB
L3 cache: 128 MiB
NUMA node0 CPU(s): 0-31
NUMA node1 CPU(s): 32-63
NUMA node2 CPU(s): 64-95
NUMA node3 CPU(s): 96-127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm

Versions of relevant libraries:
[pip3] mindietorch==2.0rc2+torch2.1.0.abi0
[pip3] numpy==1.26.4
[pip3] pyzmq==27.0.0
[pip3] torch==2.5.1
[pip3] torch-npu==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.53.2
[conda] Could not collect
vLLM Version: 0.7.3
vLLM Ascend Version: 0.7.3

ENV Variables:
VLLM_USE_MODELSCOPE=true
TORCH_DEVICE_BACKEND_AUTOLOAD=0
PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:40960
ASCEND_TOOLKIT_HOME=/usr/local/Ascend/ascend-toolkit/latest
ASCEND_OPP_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp
LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/aarch64:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/aarch64:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/gcc-9.4/lib64:
ASCEND_AICPU_PATH=/usr/local/Ascend/ascend-toolkit/latest
ASCEND_HOME_PATH=/usr/local/Ascend/ascend-toolkit/latest
TORCHINDUCTOR_COMPILE_THREADS=1

NPU:
+--------------------------------------------------------------------------------------------------------+
| npu-smi 25.0.rc1.1 Version: 25.0.rc1.1 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 6 310P3 | OK | NA 55 0 / 0 |
| 0 0 | 0000:83:00.0 | 0 1175 / 44280 |
+-------------------------------+-----------------+------------------------------------------------------+
| 6 310P3 | OK | NA 52 3262 / 3262 |
| 1 1 | 0000:83:00.0 | 0 8322 / 43693 |
+===============================+=================+======================================================+
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===============================+=================+======================================================+
| 6 1 | 293029 | python3.11 | 6612 |
+===============================+=================+======================================================+

CANN:
package_name=Ascend-cann-toolkit
version=8.1.RC1
innerversion=V100R001C21SPC001B238
compatible_version=[V100R001C15],[V100R001C18],[V100R001C19],[V100R001C20],[V100R001C21]
arch=aarch64
os=linux
path=/usr/local/Ascend/ascend-toolkit/8.1.RC1/aarch64-linux

🐛 Describe the bug

import os
from vllm import LLM, SamplingParams
#import mindie_turbo

#os.environ["CUDA_VISIBLE_DEVICES"]="0,1"
#os.environ["PYTORCH_CUDA_ALLOC_CONF"]="max_split_size_mb:64"

prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]

Create a sampling params object.

sampling_params = SamplingParams(temperature=0.8,top_p=0.95)

Create an LLM.

llm = LLM(model="./models/qwen2.5-vl-3")

Generate texts from the prompts.

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text

[bouyeijiang]

谢谢您的来信！我会及时查阅的  

[bouyeijiang]

I tried using the qwen3-8b model and got the following error:

[rank0]:[E729 15:16:31.745537205 compiler_depend.ts:422] setup failed!
Exception raised from OperationSetup at build/third_party/op-plugin/op_plugin/CMakeFiles/op_plugin_atb.dir/compiler_depend.ts:131 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0xb8 (0xffff177dc908 in /opt/python3.11_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x70 (0xffff1778b4e0 in /opt/python3.11_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: atb::OperationSetup(atb::VariantPack, atb::Operation*, atb::Context*) + 0x8c (0xfffdea0846bc in /opt/python3.11_env/lib/python3.11/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #3: + 0x34d84 (0xfffdea084d84 in /opt/python3.11_env/lib/python3.11/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #4: + 0x1643184 (0xfffd6f4c3184 in /opt/python3.11_env/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: + 0x78d554 (0xfffd6e60d554 in /opt/python3.11_env/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: + 0x78dd68 (0xfffd6e60dd68 in /opt/python3.11_env/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: + 0x78a4dc (0xfffd6e60a4dc in /opt/python3.11_env/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #8: + 0x4c9e4c (0xffff17819e4c in /opt/python3.11_env/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #9: + 0x87c0 (0xffff2f1b87c0 in /usr/lib64/libpthread.so.0)
frame #10: + 0xdc75c (0xffff2efbc75c in /usr/lib64/libc.so.6)

[rank0]: Traceback (most recent call last):
[rank0]: File "/opt/example.py", line 17, in
[rank0]: llm = LLM(model="./models/qwen3-8b")
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/vllm/utils.py", line 1022, in inner
[rank0]: return fn(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 242, in init
[rank0]: self.llm_engine = self.engine_class.from_engine_args(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 489, in from_engine_args
[rank0]: engine = cls(
[rank0]: ^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 276, in init
[rank0]: self._initialize_kv_caches()
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 421, in _initialize_kv_caches
[rank0]: self.model_executor.determine_num_available_blocks())
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 102, in determine_num_available_blocks
[rank0]: results = self.collective_rpc("determine_num_available_blocks")
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
[rank0]: answer = run_method(self.driver_worker, method, args, kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/vllm/utils.py", line 2196, in run_method
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/vllm/vllm-ascend/vllm_ascend/worker/worker.py", line 255, in determine_num_available_blocks
[rank0]: self.model_runner.profile_run()
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/vllm/vllm-ascend/vllm_ascend/worker/model_runner.py", line 1490, in profile_run
[rank0]: self.execute_model(model_input, kv_caches, intermediate_tensors)
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/vllm/vllm-ascend/vllm_ascend/worker/model_runner.py", line 1328, in execute_model
[rank0]: output: SamplerOutput = self.model.sample(
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/vllm/model_executor/models/transformers.py", line 246, in sample
[rank0]: next_tokens = self.sampler(logits, sampling_metadata)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/python3.11_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/vllm/vllm-ascend/vllm_ascend/sample/sampler.py", line 61, in forward
[rank0]: logits = _apply_top_k_top_p_npu(logits, sampling_tensors.top_ps,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/vllm/vllm-ascend/vllm_ascend/sample/sampler.py", line 123, in _apply_top_k_top_p_npu
[rank0]: probs_sort = logits_sort.softmax(dim=-1)[:, cutoff:]
[rank0]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
[rank0]: RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is SelfAttentionOperation.
[rank0]: Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[rank0]: Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[rank0]: [ERROR] 2025-07-29-15:16:31 (PID:1680587, Device:0, RankID:-1) ERR00100 PTA call acl api failed.

[leo-pony]

https://vllm-ascend.readthedocs.io/en/latest/tutorials/single_node_300i.html
@bouyeijiang as above guide shows, does Qwen/Qwen3-0.6B can successfully run in your environment?

[bouyeijiang]

the same error message,ready to install again from the source code

[Inexzero0]

(also 300I DUO) I have similar problem when I run this code:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# Create an LLM.
llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")

# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

but when I run single_node_300i this code, another error appear and no result:
Segmentation fault [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! /usr/local/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

[leo-pony]

@Inexzero0 please following the offical guide, you code has problems,
enforce_eager=True 和 dtype="float16 is the required，and compilation_config={"custom_ops":["none", "+rms_norm", "+rotary_embedding"]}, if for High performance for 300I series

[leo-pony]

the same error message,ready to install again from the source code

plz first running one example(such as Qwen/Qwen3-0.6B example) in guide doc: https://vllm-ascend.readthedocs.io/en/latest/tutorials/single_node_300i.html, to check your env is correctlly installed.

[leo-pony]

@bouyeijiang I test with vllm v0.10.0 + vllm-ascend(main), vllm V1 engine, Qwen2.5-VL-3B-Instruct, single card: 310P3 44GB, running okay, code as following:

export VLLM_USE_V1=1
export VLLM_USE_MODELSCOPE=True

from vllm import LLM, SamplingParams
import gc
import torch
from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import (destroy_distributed_environment,
                                             destroy_model_parallel)

def clean_up():
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    torch.npu.empty_cache()
prompts = [
    "Hello, my name is",
    "The future of AI is",
]
# Create a sampling params object.
sampling_params = SamplingParams(max_tokens=100, temperature=0.0)
# Create an LLM.
llm = LLM(
    model="Qwen/Qwen2.5-VL-3B-Instruct",
    tensor_parallel_size=1,
    enforce_eager=True, # For 300I series, only eager mode is supported.
    dtype="float16", # IMPORTANT cause some ATB ops cannot support bf16 on 300I series
    compilation_config={"custom_ops":["none", "+rms_norm", "+rotary_embedding"]}, # High performance for 300I series
)
# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
del llm
clean_up()

running log:

[bouyeijiang]

[rank0]:[W917 04:55:12.607632627 compiler_depend.ts:62] Warning: Cannot create tensor with NZ format while dim < 2, tensor will be created with ND format. (function operator())
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] EngineCore failed to start.
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] Traceback (most recent call last):
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 709, in run_engine_core
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 505, in init
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] super().init(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 91, in init
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] self._initialize_kv_caches(vllm_config)
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 215, in _initialize_kv_caches
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 72, in initialize_from_config
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] self.collective_rpc("initialize_from_config",
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm/vllm/utils/init.py", line 3060, in run_method
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] return func(*args, **kwargs)
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm/vllm/worker/worker_base.py", line 606, in initialize_from_config
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] self.worker.initialize_from_config(kv_cache_config) # type: ignore
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 293, in initialize_from_config
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] self.model_runner.initialize_kv_cache(kv_cache_config)
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2396, in initialize_kv_cache
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] kv_caches = self.initialize_kv_cache_tensors(kv_cache_config)
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2579, in initialize_kv_cache_tensors
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] kv_cache = self._convert_torch_format(kv_cache)
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2377, in _convert_torch_format
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] tensor = torch_npu.npu_format_cast(tensor, ACL_FORMAT)
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] return self._op(*args, **(kwargs or {}))
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:508 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507013
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] [ERROR] 2025-09-17-04:55:13 (PID:553, Device:0, RankID:-1) ERR00100 PTA call acl api failed
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] [Error]: System Direct Memory Access (DMA) hardware execution error.
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] Rectify the fault based on the error information in the ascend log.
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] EL0004: [PID: 553] 2025-09-17-04:55:13.065.712 Failed to allocate memory.
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] Possible Cause: Available memory is insufficient.
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] Solution: Close applications not in use.
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] TraceBack (most recent call last):
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] alloc device memory failed, runtime result = 207001[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] The error from device(0), serial number is 1. there is a sdma error, sdma channel is 0, the channel exist the following problems: The SMMU returns a Terminate error during page table translation.. the value of CQE status is 2. the description of CQE status: When the SQE translates a page table, the SMMU returns a Terminate error.it's config include: setting1=0xc000000880e0000, setting2=0xff009000ff004c, setting3=0, sq base addr=0x800d00001004c000[FUNC:ProcessSdmaErrorInfo][FILE:device_error_proc.cc][LINE:779]
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] Memory async copy failed, device_id=0, stream_id=3, task_id=3020, flip_num=0, copy_type=2, memcpy_type=0, copy_data_type=0, length=2654994432[FUNC:GetError][FILE:stream.cc][LINE:1183]
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718]
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718]
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] DEVICE[0] PID[553]:
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] EXCEPTION STREAM:
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] Exception info:TGID=978701, model id=65535, stream id=3, stream phase=3
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] Message info[0]:RTS_HWTS: hwts sdma error, slot_id=3, stream_id=3
(EngineCore_DP0 pid=553) ERROR 09-17 04:55:13 [core.py:718] Other info[0]:time=2025-09-17-04:55:12.532.606, function=int_process_hwts_sdma_error, line=1381, error code=0x20b
(EngineCore_DP0 pid=553) Process EngineCore_DP0:
(EngineCore_DP0 pid=553) Traceback (most recent call last):
(EngineCore_DP0 pid=553) File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=553) self.run()
(EngineCore_DP0 pid=553) File "/usr/local/python3.11.13/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=553) self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 722, in run_engine_core
(EngineCore_DP0 pid=553) raise e
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 709, in run_engine_core
(EngineCore_DP0 pid=553) engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=553) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 505, in init
(EngineCore_DP0 pid=553) super().init(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 91, in init
(EngineCore_DP0 pid=553) self._initialize_kv_caches(vllm_config)
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm/vllm/v1/engine/core.py", line 215, in _initialize_kv_caches
(EngineCore_DP0 pid=553) self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm/vllm/v1/executor/abstract.py", line 72, in initialize_from_config
(EngineCore_DP0 pid=553) self.collective_rpc("initialize_from_config",
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_DP0 pid=553) answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=553) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm/vllm/utils/init.py", line 3060, in run_method
(EngineCore_DP0 pid=553) return func(*args, **kwargs)
(EngineCore_DP0 pid=553) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm/vllm/worker/worker_base.py", line 606, in initialize_from_config
(EngineCore_DP0 pid=553) self.worker.initialize_from_config(kv_cache_config) # type: ignore
(EngineCore_DP0 pid=553) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 293, in initialize_from_config
(EngineCore_DP0 pid=553) self.model_runner.initialize_kv_cache(kv_cache_config)
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2396, in initialize_kv_cache
(EngineCore_DP0 pid=553) kv_caches = self.initialize_kv_cache_tensors(kv_cache_config)
(EngineCore_DP0 pid=553) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2579, in initialize_kv_cache_tensors
(EngineCore_DP0 pid=553) kv_cache = self._convert_torch_format(kv_cache)
(EngineCore_DP0 pid=553) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) File "/vllm-workspace/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2377, in _convert_torch_format
(EngineCore_DP0 pid=553) tensor = torch_npu.npu_format_cast(tensor, ACL_FORMAT)
(EngineCore_DP0 pid=553) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) File "/usr/local/python3.11.13/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
(EngineCore_DP0 pid=553) return self._op(*args, **(kwargs or {}))
(EngineCore_DP0 pid=553) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=553) RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:508 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507013
(EngineCore_DP0 pid=553) [ERROR] 2025-09-17-04:55:13 (PID:553, Device:0, RankID:-1) ERR00100 PTA call acl api failed
(EngineCore_DP0 pid=553) [Error]: System Direct Memory Access (DMA) hardware execution error.
(EngineCore_DP0 pid=553) Rectify the fault based on the error information in the ascend log.
(EngineCore_DP0 pid=553) EL0004: [PID: 553] 2025-09-17-04:55:13.065.712 Failed to allocate memory.
(EngineCore_DP0 pid=553) Possible Cause: Available memory is insufficient.
(EngineCore_DP0 pid=553) Solution: Close applications not in use.
(EngineCore_DP0 pid=553) TraceBack (most recent call last):
(EngineCore_DP0 pid=553) alloc device memory failed, runtime result = 207001[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(EngineCore_DP0 pid=553) The error from device(0), serial number is 1. there is a sdma error, sdma channel is 0, the channel exist the following problems: The SMMU returns a Terminate error during page table translation.. the value of CQE status is 2. the description of CQE status: When the SQE translates a page table, the SMMU returns a Terminate error.it's config include: setting1=0xc000000880e0000, setting2=0xff009000ff004c, setting3=0, sq base addr=0x800d00001004c000[FUNC:ProcessSdmaErrorInfo][FILE:device_error_proc.cc][LINE:779]
(EngineCore_DP0 pid=553) Memory async copy failed, device_id=0, stream_id=3, task_id=3020, flip_num=0, copy_type=2, memcpy_type=0, copy_data_type=0, length=2654994432[FUNC:GetError][FILE:stream.cc][LINE:1183]
(EngineCore_DP0 pid=553) rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_DP0 pid=553) wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(EngineCore_DP0 pid=553)
(EngineCore_DP0 pid=553)
(EngineCore_DP0 pid=553) DEVICE[0] PID[553]:
(EngineCore_DP0 pid=553) EXCEPTION STREAM:
(EngineCore_DP0 pid=553) Exception info:TGID=978701, model id=65535, stream id=3, stream phase=3
(EngineCore_DP0 pid=553) Message info[0]:RTS_HWTS: hwts sdma error, slot_id=3, stream_id=3
(EngineCore_DP0 pid=553) Other info[0]:time=2025-09-17-04:55:12.532.606, function=int_process_hwts_sdma_error, line=1381, error code=0x20b
[rank0]:[W917 04:55:15.079208046 compiler_depend.ts:528] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 553] 2025-09-17-04:55:15.384.734 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeUsedDevices)
[W917 04:55:15.081489505 compiler_depend.ts:510] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 553] 2025-09-17-04:55:15.387.278 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
[W917 04:55:15.083329282 compiler_depend.ts:227] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 553] 2025-09-17-04:55:15.389.172 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function empty_cache)
[W917 04:55:17.263678529 compiler_depend.ts:510] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 553] 2025-09-17-04:55:17.569.364 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
[W917 04:55:17.266996512 compiler_depend.ts:227] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 553] 2025-09-17-04:55:17.572.893 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function empty_cache)
[W917 04:55:17.268722899 compiler_depend.ts:510] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 553] 2025-09-17-04:55:17.574.633 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
[W917 04:55:17.270466736 compiler_depend.ts:227] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 553] 2025-09-17-04:55:17.576.399 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function empty_cache)
[W917 04:55:17.272310533 compiler_depend.ts:510] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 553] 2025-09-17-04:55:17.578.107 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
[W917 04:55:17.274056390 compiler_depend.ts:227] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 553] 2025-09-17-04:55:17.579.982 wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function empty_cache)
(APIServer pid=285) Traceback (most recent call last):
(APIServer pid=285) File "/usr/local/python3.11.13/bin/vllm", line 8, in
(APIServer pid=285) sys.exit(main())
(APIServer pid=285) ^^^^^^
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/entrypoints/cli/main.py", line 54, in main
(APIServer pid=285) args.dispatch_function(args)
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/entrypoints/cli/serve.py", line 50, in cmd
(APIServer pid=285) uvloop.run(run_server(args))
(APIServer pid=285) File "/usr/local/python3.11.13/lib/python3.11/site-packages/uvloop/init.py", line 105, in run
(APIServer pid=285) return runner.run(wrapper())
(APIServer pid=285) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=285) File "/usr/local/python3.11.13/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=285) return self._loop.run_until_complete(task)
(APIServer pid=285) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=285) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=285) File "/usr/local/python3.11.13/lib/python3.11/site-packages/uvloop/init.py", line 61, in wrapper
(APIServer pid=285) return await main
(APIServer pid=285) ^^^^^^^^^^
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1941, in run_server
(APIServer pid=285) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 1961, in run_server_worker
(APIServer pid=285) async with build_async_engine_client(
(APIServer pid=285) File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 210, in aenter
(APIServer pid=285) return await anext(self.gen)
(APIServer pid=285) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 179, in build_async_engine_client
(APIServer pid=285) async with build_async_engine_client_from_engine_args(
(APIServer pid=285) File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 210, in aenter
(APIServer pid=285) return await anext(self.gen)
(APIServer pid=285) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/entrypoints/openai/api_server.py", line 221, in build_async_engine_client_from_engine_args
(APIServer pid=285) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=285) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/utils/init.py", line 1589, in inner
(APIServer pid=285) return fn(*args, **kwargs)
(APIServer pid=285) ^^^^^^^^^^^^^^^^^^^
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 212, in from_vllm_config
(APIServer pid=285) return cls(
(APIServer pid=285) ^^^^
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/v1/engine/async_llm.py", line 136, in init
(APIServer pid=285) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=285) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client
(APIServer pid=285) return AsyncMPClient(*client_args)
(APIServer pid=285) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 769, in init
(APIServer pid=285) super().init(
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/v1/engine/core_client.py", line 448, in init
(APIServer pid=285) with launch_core_engines(vllm_config, executor_class,
(APIServer pid=285) File "/usr/local/python3.11.13/lib/python3.11/contextlib.py", line 144, in exit
(APIServer pid=285) next(self.gen)
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 729, in launch_core_engines
(APIServer pid=285) wait_for_engine_startup(
(APIServer pid=285) File "/vllm-workspace/vllm/vllm/v1/engine/utils.py", line 782, in wait_for_engine_startup
(APIServer pid=285) raise RuntimeError("Engine core initialization failed. "
(APIServer pid=285) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(APIServer pid=285) [ERROR] 2025-09-17-04:55:25 (PID:285, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

[Inexzero0]

same issue, vllm 0.10.2, vllm-ascend 0.10.2rc1 @leo-pony

[bouyeijiang]

谢谢您的来信！我会及时查阅的  

[exthirteen]

same error, from docker images quay.io/ascend/vllm-ascend:v0.10.2rc1-310p

docker run cmd:
docker run --rm
--name vllm-ascend
--device /dev/davinci0
--device /dev/davinci1
--device /dev/davinci_manager
--device /dev/devmm_svm
--device /dev/hisi_hdc
-v /usr/local/dcmi:/usr/local/dcmi
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info
-v /etc/ascend_install.info:/etc/ascend_install.info
-v /root/.cache:/root/.cache
-p 8000:8000
-e VLLM_USE_MODELSCOPE=True
-e PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:1024
-it quay.io/ascend/vllm-ascend:v0.10.2rc1-310p bash

vllm run cmd:
vllm serve Qwen/Qwen3-0.6B
--tensor-parallel-size 1
--enforce-eager
--dtype float16
--compilation-config '{"custom_ops":["none", "+rms_norm", "+rotary_embedding"]}'