[Release]: Release checklist for v0.10.0rc1

[MengqingCao]

Release Checklist

Release Version: v0.10.0rc1
Release Branch: main
Release Date:
Release Manager: MengqingCao

Prepare Release Note

• Create a new issue for release feedback [v0.10.0rc1] FAQ / Feedback | 问题/反馈 #2217

• Write the release note PR.

  • Update the feedback issue link in docs/source/faqs.md

  • Add release note to docs/source/user_guide/release_notes.md

  • Update version info in docs/source/community/versioning_policy.md

  • Update contributor info in docs/source/community/contributors.md

  • Update package version in docs/conf.py -- do me after release

PR need Merge

• [BugFix] Fix the bug that qwen3 moe doesn't work with aclgraph #2183
• [V1] MTP supports torchair #2145
• [Doc] Support kimi-k2-w8a8 #2162
• [Bugfix] Fix disaggregated pd error #2242
• 【main】SP For Qwen3 MoE #2209

Functional Test

• ALL tests with VLLM_ASCEND_ENABLE_MATMUL_ALLREDUCE enabled

• Disaggregate prefill @Potabk

  • deepseek
    • Disaggregate prefill on multi-node + deepseek + torchair graph mode + V1 scheduler + dp + ep + tp
    • Disaggregate prefill on single-node + deepseek + torchair graph mode + V1 scheduler + dp + ep + tp
    • Disaggregate prefill on multi-node + deepseek + torchair graph mode + AscendScheduler (disabling chunked prefill) + dp + ep + tp
    • Disaggregate prefill on single-node + deepseek + torchair graph mode + AscendScheduler (disabling chunked prefill) + dp + ep + tp
  • qwen3 moe
    • Disaggregate prefill on multi-node + qwen3 235B + aclgraph + V1 scheduler + dp + ep + tp
    • Disaggregate prefill on single-node + qwen3 30B + aclgraph + V1 scheduler + dp + ep + tp

• qwen3 moe + eager mode + all2allv: enabling VLLM_ASCEND_ENABLE_MOE_ALL2ALL_SEQ @wangxiyuan

• deepseek v3 + torchair + all2allv: enabling VLLM_ASCEND_ENABLE_MOE_ALL2ALL_SEQ @Potabk

• Aclgraph + qwen3 moe + dp +tp @MengqingCao [Release]: Release checklist for v0.10.0rc1 #2210 (comment)

• Eager mode + qwen3 moe + dp +tp @MengqingCao [Release]: Release checklist for v0.10.0rc1 #2210 (comment)

• spec decode @shen-shanshan

  • deepseek-w8a8 + torchair graph mode+ mtp -- rely on [V1] MTP supports torchair #2145
  • eagle3
  • ngram

• w8a8 + enabling nz -- performance @MengqingCao

• deepseek w8a8 dynamic + multi-stream @zhangxinyuehfad [Release]: Release checklist for v0.10.0rc1 #2210 (comment)

• w4a8 + qwen3-8b @22dimensions [Release]: Release checklist for v0.10.0rc1 #2210 (comment)

• numpy > 2.0 with CANN 8.2.1 @MengqingCao -- not work with numpy>2.0 [Release]: Release checklist for v0.10.0rc1 #2210 (comment)

• lora perf improve @taoxudonghaha

Doc Test

• Tutorial is updated.
• User Guide is updated.
• Developer Guide is updated.

Prepare Artifacts

• Docker image is ready.
  • A3 image check @Potabk
• Wheel package is ready.

Release Step

• Release note PR is merged.
• Post the release on GitHub release page.
• Generate official doc page on https://app.readthedocs.org/dashboard/
• Wait for the wheel package to be available on https://pypi.org/project/vllm-ascend
• Wait for the docker image to be available on https://quay.io/ascend/vllm-ascend
• Upload 310p wheel to Github release page
• Broadcast the release news (By message, blog , etc)
• Close this issue
• Update 900-release-checklist.yml
• Pin feedback issue

[MengqingCao]

we still need to pin numpy < 2.0 with CANN 8.2.1

Successfully installed numpy-2.0.2 torch-npu-2.7.1.dev20250724
(atb) xxx@xxx-docker:~/code/vllm-ascend$ python 
Python 3.10.16 (main, Dec 11 2024, 16:18:56) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch_npu
/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/__init__.py:305: UserWarning: On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default.                      Do not set it to 1 to prevent some unknown errors
  warnings.warn("On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. \
>>> import torch
>>> torch.npu.set_device(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 64, in set_device
    torch_npu._C._npu_setDevice(device_id)
RuntimeError: SetPrecisionMode:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:155 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001
[ERROR] 2025-08-05-08:05:26 (PID:504068, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The internal ACL of the system is incorrect.
        Rectify the fault based on the error information in the ascend log.
EC0010: [PID: 504068] 2025-08-05-08:05:26.204.668 Failed to import Python module [AttributeError: `np.float_` was removed in the NumPy 2.0 release. Use `np.float64` instead..].
        Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.)
        TraceBack (most recent call last):
        AOE Failed to call InitCannKB[FUNC:Initialize][FILE:python_adapter_manager.cc][LINE:47]
        Failed to initialize TeConfigInfo.
        [GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeTeFusion][FILE:tbe_op_store_adapter.cc][LINE:1921]
        [GraphOpt][InitializeInner][InitTeFusion]: Failed to initialize TeFusion.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1888]
        [SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
        [SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
        [FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:115]
        PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:83]
        OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:239]
        GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:164]
        GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:382]
        [Initialize][Ge]GEInitialize failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        [Init][Compiler]Init compiler failed[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        [Set][Options]OpCompileProcessor init failed![FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]

[zhangxinyuehfad]

deepseek w8a8 dynamic + multi-stream test get wrong output:

command:

VLLM_USE_MODELSCOPE=True vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V2-Lite-W8A8 \
--tensor-parallel-size 1 \
--quantization ascend \
--trust_remote_code \
--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true}, "enable_multistream_moe": true}' &

output:

temperature: 0.5

root@67b51044c308:~/.cache/modelscope/hub/models/vllm-ascend# curl -X POST "http://localhost:8000/v1/completions"      -H "Content-Type: application/json"      -d '{
           "model": "/root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V2-Lite-W8A8",
           "prompt": "西安有什么好玩的地方",
           "max_tokens": 100,
           "temperature": 0.5
         }'
{"id":"cmpl-4a36b0672f5b4e5c9d022a75cdc8e356","object":"text_completion","created":1754384963,"model":"/root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V2-Lite-W8A8","choices":
[{"index":0,"text":"\n!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":5,"total_tokens":105,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

temperature: 0

root@67b51044c308:~/.cache/modelscope/hub/models/deepseek-ai# curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V2-Lite-W8A8",
        "prompt": "西安有什么好玩的地方",
        "max_tokens": 100,
        "temperature": 0
    }'
{"id":"cmpl-e01c7b920ccb4deeafd03ff59f64549d","object":"text_completion","created":1754396323,"model":"/root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V2-Lite-W8A8","choices":[{"index":0,"text":"？\n\n西安到西安安，西安安西安安西安安西安安西安安西安安西安西安安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安西安","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":5,"total_tokens":105,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

deepseek Non-quantitative + multi-stream test ok:

command:

VLLM_USE_MODELSCOPE=True vllm serve /root/.cache/modelscope/hub/models/deepseek-ai/DeepSeek-V2-Lite \
--tensor-parallel-size 2 \
--trust-remote-code \
--max_model_len 4096 \
--additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true}, "enable_multistream_moe": true}' &

output:

root@67b51044c308:~/.cache/modelscope/hub/models/deepseek-ai# curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/root/.cache/modelscope/hub/models/deepseek-ai/DeepSeek-V2-Lite",
        "prompt": "西安有什么好玩的地方",
        "max_tokens": 100,
        "temperature": 0.5
    }'
{"id":"cmpl-cb67dbaa63b94385b39fe9c022902056","object":"text_completion","created":1754393499,"model":"/root/.cache/modelscope/hub/models/deepseek-ai/DeepSeek-V2-Lite","choices":[{"index":0,"text":"\n\n西安好玩的地方有：大雁塔、大唐芙蓉园、陕西历史博物馆、西安城墙、钟楼、鼓楼、华清池、兵马俑、大明宫、小雁塔等。\n\n大雁塔又名大慈恩寺塔，位于中国陕西省西安市南郊大慈恩寺内。因坐落在慈恩寺西院内，大雁塔原称慈恩寺西院浮屠，是中国唐朝佛教建筑艺术","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":5,"total_tokens":105,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

[MengqingCao]

eager mode + Qwen3-30B-A3B + dp + ep works fine:

(atb) xxx@xxx-docker:~/code/vllm-ascend$ bash scripts/dp.sh 
INFO 08-05 11:16:02 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:16:02 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:16:02 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:16:02 [__init__.py:226] Platform plugin ascend is activated
WARNING 08-05 11:16:06 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
DP rank 0 needs to process 200 prompts
DP rank 1 needs to process 200 prompts
INFO 08-05 11:16:06 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
INFO 08-05 11:16:06 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3:CustomDeepseekV3ForCausalLM.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3:CustomDeepseekV3ForCausalLM.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
WARNING 08-05 11:16:08 [registry.py:430] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
INFO 08-05 11:16:26 [config.py:1604] Using max model len 40960
INFO 08-05 11:16:26 [config.py:1604] Using max model len 40960
INFO 08-05 11:16:26 [config.py:2434] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 08-05 11:16:26 [platform.py:144] Compilation disabled, using eager mode by default
INFO 08-05 11:16:26 [config.py:2434] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 08-05 11:16:26 [platform.py:144] Compilation disabled, using eager mode by default
INFO 08-05 11:16:41 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:16:41 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:16:41 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:16:41 [__init__.py:226] Platform plugin ascend is activated
INFO 08-05 11:16:41 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:16:41 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:16:41 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:16:41 [__init__.py:226] Platform plugin ascend is activated
WARNING 08-05 11:16:44 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
WARNING 08-05 11:16:44 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
(EngineCore_1 pid=515701) INFO 08-05 11:16:44 [core.py:572] Waiting for init message from front-end.
(EngineCore_0 pid=515697) INFO 08-05 11:16:44 [core.py:572] Waiting for init message from front-end.
(EngineCore_1 pid=515701) INFO 08-05 11:16:45 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
(EngineCore_0 pid=515697) INFO 08-05 11:16:45 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
(EngineCore_0 pid=515697) WARNING 08-05 11:16:46 [registry.py:430] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
(EngineCore_0 pid=515697) WARNING 08-05 11:16:46 [registry.py:430] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
(EngineCore_0 pid=515697) WARNING 08-05 11:16:46 [registry.py:430] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
(EngineCore_0 pid=515697) WARNING 08-05 11:16:46 [registry.py:430] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
(EngineCore_0 pid=515697) WARNING 08-05 11:16:46 [registry.py:430] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3:CustomDeepseekV3ForCausalLM.
(EngineCore_0 pid=515697) WARNING 08-05 11:16:46 [registry.py:430] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
(EngineCore_0 pid=515697) WARNING 08-05 11:16:46 [registry.py:430] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
(EngineCore_0 pid=515697) INFO 08-05 11:16:46 [core.py:71] Initializing a V1 LLM engine (v0.9.2.dev301+g3c545c0c3) with config: model='/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
(EngineCore_1 pid=515701) WARNING 08-05 11:16:46 [registry.py:430] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
(EngineCore_1 pid=515701) WARNING 08-05 11:16:46 [registry.py:430] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
(EngineCore_1 pid=515701) WARNING 08-05 11:16:46 [registry.py:430] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
(EngineCore_1 pid=515701) WARNING 08-05 11:16:46 [registry.py:430] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
(EngineCore_1 pid=515701) WARNING 08-05 11:16:46 [registry.py:430] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3:CustomDeepseekV3ForCausalLM.
(EngineCore_1 pid=515701) WARNING 08-05 11:16:46 [registry.py:430] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
(EngineCore_1 pid=515701) WARNING 08-05 11:16:46 [registry.py:430] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
(EngineCore_1 pid=515701) INFO 08-05 11:16:46 [core.py:71] Initializing a V1 LLM engine (v0.9.2.dev301+g3c545c0c3) with config: model='/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
INFO 08-05 11:17:04 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:17:04 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:17:04 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:17:04 [__init__.py:226] Platform plugin ascend is activated
INFO 08-05 11:17:05 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:17:05 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:17:05 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:17:05 [__init__.py:226] Platform plugin ascend is activated
WARNING 08-05 11:17:07 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
WARNING 08-05 11:17:08 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
(EngineCore_0 pid=515697) INFO 08-05 11:17:11 [parallel_state.py:961] Adjusting world_size=2 rank=0 distributed_init_method=tcp://127.0.0.1:52674 for DP
(EngineCore_1 pid=515701) INFO 08-05 11:17:11 [parallel_state.py:961] Adjusting world_size=2 rank=1 distributed_init_method=tcp://127.0.0.1:52674 for DP
(EngineCore_1 pid=515701) INFO 08-05 11:17:12 [communicator.py:42] Using naive all2all manager.
(EngineCore_0 pid=515697) INFO 08-05 11:17:12 [communicator.py:42] Using naive all2all manager.
(EngineCore_1 pid=515701) INFO 08-05 11:17:12 [parallel_state.py:1102] rank 1 in world size 2 is assigned as DP rank 1, PP rank 0, TP rank 0, EP rank 1
(EngineCore_0 pid=515697) INFO 08-05 11:17:12 [parallel_state.py:1102] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_0 pid=515697) INFO 08-05 11:17:12 [model_runner_v1.py:2040] Starting to load model /home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B...
(EngineCore_1 pid=515701) INFO 08-05 11:17:12 [model_runner_v1.py:2040] Starting to load model /home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B...
Loading safetensors checkpoint shards:   0% Completed | 0/16 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   6% Completed | 1/16 [00:00<00:10,  1.40it/s]
Loading safetensors checkpoint shards:  12% Completed | 2/16 [00:01<00:06,  2.10it/s]
Loading safetensors checkpoint shards:  19% Completed | 3/16 [00:01<00:08,  1.61it/s]
Loading safetensors checkpoint shards:  25% Completed | 4/16 [00:02<00:08,  1.34it/s]
Loading safetensors checkpoint shards:  31% Completed | 5/16 [00:03<00:08,  1.28it/s]
Loading safetensors checkpoint shards:  38% Completed | 6/16 [00:04<00:07,  1.27it/s]
Loading safetensors checkpoint shards:  44% Completed | 7/16 [00:05<00:07,  1.28it/s]
Loading safetensors checkpoint shards:  50% Completed | 8/16 [00:06<00:06,  1.25it/s]
Loading safetensors checkpoint shards:  56% Completed | 9/16 [00:06<00:05,  1.21it/s]
Loading safetensors checkpoint shards:  62% Completed | 10/16 [00:07<00:05,  1.20it/s]
Loading safetensors checkpoint shards:  69% Completed | 11/16 [00:08<00:04,  1.17it/s]
Loading safetensors checkpoint shards:  75% Completed | 12/16 [00:09<00:03,  1.17it/s]
Loading safetensors checkpoint shards:  81% Completed | 13/16 [00:10<00:02,  1.18it/s]
Loading safetensors checkpoint shards:  88% Completed | 14/16 [00:11<00:01,  1.18it/s]
Loading safetensors checkpoint shards:  94% Completed | 15/16 [00:12<00:00,  1.15it/s]
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:12<00:00,  1.15it/s]
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:12<00:00,  1.23it/s]
(EngineCore_0 pid=515697) 
(EngineCore_0 pid=515697) INFO 08-05 11:17:27 [default_loader.py:262] Loading weights took 13.17 seconds
(EngineCore_1 pid=515701) INFO 08-05 11:17:27 [default_loader.py:262] Loading weights took 13.48 seconds
(EngineCore_0 pid=515697) INFO 08-05 11:17:28 [model_runner_v1.py:2070] Loading model weights took 29.8816 GB
(EngineCore_1 pid=515701) INFO 08-05 11:17:28 [model_runner_v1.py:2070] Loading model weights took 29.8816 GB
[rank0]:[W805 11:17:33.926755169 compiler_depend.ts:28] Warning: The oprator of MoeInitRouting will be removed from Pytorch and switch to AscendSpeed after 630. (function operator())
[rank1]:[W805 11:17:33.927160482 compiler_depend.ts:28] Warning: The oprator of MoeInitRouting will be removed from Pytorch and switch to AscendSpeed after 630. (function operator())
[rank0]:[W805 11:17:34.352018249 compiler_depend.ts:26] Warning: Warning: kernel [ArgSort] can not support dtype int32 or int64 on AiCore, Now this kernel is running on AiCpu.If you are more concerned about high-performance execution,please cast dtype to float32. (function operator())
[rank1]:[W805 11:17:34.353874789 compiler_depend.ts:26] Warning: Warning: kernel [ArgSort] can not support dtype int32 or int64 on AiCore, Now this kernel is running on AiCpu.If you are more concerned about high-performance execution,please cast dtype to float32. (function operator())
(EngineCore_0 pid=515697) INFO 08-05 11:17:36 [worker_v1.py:186] Available memory: 23550048256, total memory: 65452113920
(EngineCore_0 pid=515697) INFO 08-05 11:17:36 [kv_cache_utils.py:833] GPU KV cache size: 239,488 tokens
(EngineCore_0 pid=515697) INFO 08-05 11:17:36 [kv_cache_utils.py:837] Maximum concurrency for 40,960 tokens per request: 5.85x
(EngineCore_1 pid=515701) INFO 08-05 11:17:36 [worker_v1.py:186] Available memory: 23547757056, total memory: 65452113920
(EngineCore_1 pid=515701) INFO 08-05 11:17:36 [kv_cache_utils.py:833] GPU KV cache size: 239,488 tokens
(EngineCore_1 pid=515701) INFO 08-05 11:17:36 [kv_cache_utils.py:837] Maximum concurrency for 40,960 tokens per request: 5.85x
(EngineCore_0 pid=515697) INFO 08-05 11:17:36 [core.py:193] init engine (profile, create kv cache, warmup model) took 8.27 seconds
(EngineCore_1 pid=515701) INFO 08-05 11:17:36 [core.py:193] init engine (profile, create kv cache, warmup model) took 7.77 seconds
(EngineCore_0 pid=515697) INFO 08-05 11:17:37 [platform.py:144] Compilation disabled, using eager mode by default
Adding requests:   0%|                                                                                                                           | 0/200 [00:00<?, ?it/s](EngineCore_1 pid=515701) INFO 08-05 11:17:37 [platform.py:144] Compilation disabled, using eager mode by default
Adding requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 1786.83it/s]
Adding requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 1954.54it/s]
Processed prompts: 100%|███████████████████████████████████████████████████████| 200/200 [00:09<00:00, 20.28it/s, est. speed input: 172.61 toks/s, output: 627.66 toks/s]

DP rank 0, Prompt: 'Hello, my name is', Generated text: ' Lucy and I am from the UK. I have a few questions about the German'
DP rank 1, Prompt: 'Hello, my name is', Generated text: ' Lucy and I am from the UK. I have a few questions about the German language. First,'
DP rank 0, Prompt: 'The president of the United States is', Generated text: ' a powerful leader, but he or she is not all-powerful. What does'
DP rank 1, Prompt: 'The president of the United States is', Generated text: ' a powerful leader, but he or she is not all-powerful. What does this mean, and'
DP rank 0, Prompt: 'The capital of France is', Generated text: ' Paris, and the capital of Germany is Berlin. So, the capital of Italy'
DP rank 1, Prompt: 'The capital of France is', Generated text: ' Paris, and the capital of Germany is Berlin. So, the capital of Italy is...? The'
DP rank 0, Prompt: 'The future of AI is', Generated text: ' bright, but there are still challenges to be addressed. One of the main challenges'
DP rank 1, Prompt: 'The future of AI is', Generated text: ' bright, but there are still challenges to be addressed. One of the main challenges is the issue of'
DP rank 0, Prompt: 'Hello, my name is', Generated text: " Phoebe. I'm 13 and I live in Chicago. I"
DP rank 1, Prompt: 'Hello, my name is', Generated text: " Phoebe. I'm 13 and I live in Chicago. I have a brother and"
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Qwen3-30B-A3B failed with aclgraph + dp + ep

(atb) xxx@xxx-docker:~/code/vllm-ascend$ bash scripts/dp.sh 
INFO 08-05 11:21:04 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:21:04 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:21:04 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:21:04 [__init__.py:226] Platform plugin ascend is activated
WARNING 08-05 11:21:07 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
DP rank 0 needs to process 200 prompts
DP rank 1 needs to process 200 prompts
INFO 08-05 11:21:08 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
INFO 08-05 11:21:08 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 08-05 11:21:09 [registry.py:430] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 08-05 11:21:09 [registry.py:430] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 08-05 11:21:09 [registry.py:430] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 08-05 11:21:09 [registry.py:430] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 08-05 11:21:09 [registry.py:430] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3:CustomDeepseekV3ForCausalLM.
WARNING 08-05 11:21:09 [registry.py:430] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
WARNING 08-05 11:21:09 [registry.py:430] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
WARNING 08-05 11:21:10 [registry.py:430] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 08-05 11:21:10 [registry.py:430] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 08-05 11:21:10 [registry.py:430] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 08-05 11:21:10 [registry.py:430] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 08-05 11:21:10 [registry.py:430] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3:CustomDeepseekV3ForCausalLM.
WARNING 08-05 11:21:10 [registry.py:430] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
WARNING 08-05 11:21:10 [registry.py:430] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
INFO 08-05 11:21:27 [config.py:1604] Using max model len 40960
INFO 08-05 11:21:27 [config.py:2434] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 08-05 11:21:27 [platform.py:162] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
INFO 08-05 11:21:27 [utils.py:333] Calculated maximum supported batch sizes for ACL graph: 39
INFO 08-05 11:21:27 [utils.py:348] Adjusted ACL graph batch sizes for Qwen3MoeForCausalLM model (layers: 48): 67 → 39 sizes
INFO 08-05 11:21:28 [config.py:1604] Using max model len 40960
INFO 08-05 11:21:28 [config.py:2434] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 08-05 11:21:28 [platform.py:162] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
INFO 08-05 11:21:28 [utils.py:333] Calculated maximum supported batch sizes for ACL graph: 39
INFO 08-05 11:21:28 [utils.py:348] Adjusted ACL graph batch sizes for Qwen3MoeForCausalLM model (layers: 48): 67 → 39 sizes
INFO 08-05 11:21:41 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:21:41 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:21:41 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:21:41 [__init__.py:226] Platform plugin ascend is activated
INFO 08-05 11:21:42 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:21:42 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:21:42 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:21:42 [__init__.py:226] Platform plugin ascend is activated
WARNING 08-05 11:21:44 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
(EngineCore_0 pid=519998) INFO 08-05 11:21:44 [core.py:572] Waiting for init message from front-end.
WARNING 08-05 11:21:45 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
(EngineCore_1 pid=520079) INFO 08-05 11:21:46 [core.py:572] Waiting for init message from front-end.
(EngineCore_0 pid=519998) INFO 08-05 11:21:46 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
(EngineCore_1 pid=520079) INFO 08-05 11:21:46 [importing.py:63] Triton not installed or not compatible; certain GPU-related functions will not be available.
(EngineCore_0 pid=519998) WARNING 08-05 11:21:47 [registry.py:430] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
(EngineCore_0 pid=519998) WARNING 08-05 11:21:47 [registry.py:430] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
(EngineCore_0 pid=519998) WARNING 08-05 11:21:47 [registry.py:430] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
(EngineCore_0 pid=519998) WARNING 08-05 11:21:47 [registry.py:430] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
(EngineCore_0 pid=519998) WARNING 08-05 11:21:47 [registry.py:430] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3:CustomDeepseekV3ForCausalLM.
(EngineCore_0 pid=519998) WARNING 08-05 11:21:47 [registry.py:430] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
(EngineCore_0 pid=519998) WARNING 08-05 11:21:47 [registry.py:430] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
(EngineCore_0 pid=519998) INFO 08-05 11:21:47 [core.py:71] Initializing a V1 LLM engine (v0.9.2.dev301+g3c545c0c3) with config: model='/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.unified_ascend_attention_with_output"],"use_inductor":false,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,496,488,472,456,440,432,416,400,384,376,360,344,328,320,304,288,272,264,248,232,224,208,192,176,168,152,136,120,112,96,80,64,56,40,24,8,4,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
(EngineCore_1 pid=520079) WARNING 08-05 11:21:47 [registry.py:430] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
(EngineCore_1 pid=520079) WARNING 08-05 11:21:47 [registry.py:430] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
(EngineCore_1 pid=520079) WARNING 08-05 11:21:47 [registry.py:430] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
(EngineCore_1 pid=520079) WARNING 08-05 11:21:47 [registry.py:430] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
(EngineCore_1 pid=520079) WARNING 08-05 11:21:47 [registry.py:430] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v3:CustomDeepseekV3ForCausalLM.
(EngineCore_1 pid=520079) WARNING 08-05 11:21:47 [registry.py:430] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
(EngineCore_1 pid=520079) WARNING 08-05 11:21:47 [registry.py:430] Model architecture Qwen3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3:CustomQwen3ForCausalLM.
(EngineCore_1 pid=520079) INFO 08-05 11:21:47 [core.py:71] Initializing a V1 LLM engine (v0.9.2.dev301+g3c545c0c3) with config: model='/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["all"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.unified_ascend_attention_with_output"],"use_inductor":false,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,496,488,472,456,440,432,416,400,384,376,360,344,328,320,304,288,272,264,248,232,224,208,192,176,168,152,136,120,112,96,80,64,56,40,24,8,4,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
INFO 08-05 11:22:06 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:22:06 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:22:06 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:22:06 [__init__.py:226] Platform plugin ascend is activated
INFO 08-05 11:22:06 [__init__.py:38] Available plugins for group vllm.platform_plugins:
INFO 08-05 11:22:06 [__init__.py:40] - ascend -> vllm_ascend:register
INFO 08-05 11:22:06 [__init__.py:43] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 08-05 11:22:06 [__init__.py:226] Platform plugin ascend is activated
WARNING 08-05 11:22:09 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
WARNING 08-05 11:22:09 [_custom_ops.py:20] Failed to import from vllm._C with ImportError('libnuma.so.1: cannot open shared object file: No such file or directory')
(EngineCore_0 pid=519998) INFO 08-05 11:22:13 [parallel_state.py:961] Adjusting world_size=2 rank=0 distributed_init_method=tcp://127.0.0.1:40546 for DP
(EngineCore_1 pid=520079) INFO 08-05 11:22:13 [parallel_state.py:961] Adjusting world_size=2 rank=1 distributed_init_method=tcp://127.0.0.1:40546 for DP
(EngineCore_1 pid=520079) INFO 08-05 11:22:13 [communicator.py:42] Using naive all2all manager.
(EngineCore_0 pid=519998) INFO 08-05 11:22:13 [communicator.py:42] Using naive all2all manager.
(EngineCore_1 pid=520079) INFO 08-05 11:22:13 [parallel_state.py:1102] rank 1 in world size 2 is assigned as DP rank 1, PP rank 0, TP rank 0, EP rank 1
(EngineCore_0 pid=519998) INFO 08-05 11:22:13 [parallel_state.py:1102] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_0 pid=519998) INFO 08-05 11:22:13 [model_runner_v1.py:2040] Starting to load model /home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B...
(EngineCore_1 pid=520079) INFO 08-05 11:22:13 [model_runner_v1.py:2040] Starting to load model /home/xxx/cache/modelscope/models/Qwen/Qwen3-30B-A3B...
Loading safetensors checkpoint shards:   0% Completed | 0/16 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   6% Completed | 1/16 [00:00<00:10,  1.36it/s]
Loading safetensors checkpoint shards:  12% Completed | 2/16 [00:01<00:06,  2.08it/s]
Loading safetensors checkpoint shards:  19% Completed | 3/16 [00:01<00:08,  1.62it/s]
Loading safetensors checkpoint shards:  25% Completed | 4/16 [00:02<00:08,  1.42it/s]
Loading safetensors checkpoint shards:  31% Completed | 5/16 [00:03<00:08,  1.35it/s]
Loading safetensors checkpoint shards:  38% Completed | 6/16 [00:04<00:07,  1.29it/s]
Loading safetensors checkpoint shards:  44% Completed | 7/16 [00:05<00:07,  1.23it/s]
Loading safetensors checkpoint shards:  50% Completed | 8/16 [00:06<00:06,  1.18it/s]
Loading safetensors checkpoint shards:  56% Completed | 9/16 [00:07<00:06,  1.13it/s]
Loading safetensors checkpoint shards:  62% Completed | 10/16 [00:08<00:05,  1.09it/s]
Loading safetensors checkpoint shards:  69% Completed | 11/16 [00:09<00:04,  1.08it/s]
Loading safetensors checkpoint shards:  75% Completed | 12/16 [00:09<00:03,  1.11it/s]
Loading safetensors checkpoint shards:  81% Completed | 13/16 [00:10<00:02,  1.14it/s]
Loading safetensors checkpoint shards:  88% Completed | 14/16 [00:11<00:01,  1.14it/s]
Loading safetensors checkpoint shards:  94% Completed | 15/16 [00:12<00:00,  1.10it/s]
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:13<00:00,  1.09it/s]
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:13<00:00,  1.19it/s]
(EngineCore_0 pid=519998) 
(EngineCore_1 pid=520079) INFO 08-05 11:22:28 [default_loader.py:262] Loading weights took 13.49 seconds
(EngineCore_0 pid=519998) INFO 08-05 11:22:28 [default_loader.py:262] Loading weights took 13.68 seconds
(EngineCore_0 pid=519998) INFO 08-05 11:22:29 [model_runner_v1.py:2070] Loading model weights took 29.8816 GB
(EngineCore_1 pid=520079) INFO 08-05 11:22:30 [model_runner_v1.py:2070] Loading model weights took 29.8816 GB
(EngineCore_1 pid=520079) INFO 08-05 11:22:42 [backends.py:530] Using cache directory: /home/xxx/.cache/vllm/torch_compile_cache/443de98178/rank_0_1/backbone for vLLM's torch.compile
(EngineCore_1 pid=520079) INFO 08-05 11:22:42 [backends.py:541] Dynamo bytecode transform time: 11.56 s
(EngineCore_0 pid=519998) INFO 08-05 11:22:43 [backends.py:530] Using cache directory: /home/xxx/.cache/vllm/torch_compile_cache/b9248653d8/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_0 pid=519998) INFO 08-05 11:22:43 [backends.py:541] Dynamo bytecode transform time: 12.04 s
(EngineCore_1 pid=520079) INFO 08-05 11:22:46 [backends.py:215] Compiling a graph for dynamic shape takes 3.42 s
(EngineCore_0 pid=519998) INFO 08-05 11:22:47 [backends.py:215] Compiling a graph for dynamic shape takes 3.37 s
(EngineCore_0 pid=519998) INFO 08-05 11:22:55 [monitor.py:34] torch.compile takes 15.41 s in total
(EngineCore_1 pid=520079) INFO 08-05 11:22:55 [monitor.py:34] torch.compile takes 14.99 s in total
(EngineCore_0 pid=519998) INFO 08-05 11:22:57 [worker_v1.py:186] Available memory: 21729865728, total memory: 65452113920
(EngineCore_0 pid=519998) INFO 08-05 11:22:57 [kv_cache_utils.py:833] GPU KV cache size: 220,928 tokens
(EngineCore_0 pid=519998) INFO 08-05 11:22:57 [kv_cache_utils.py:837] Maximum concurrency for 40,960 tokens per request: 5.39x
(EngineCore_1 pid=520079) INFO 08-05 11:22:57 [worker_v1.py:186] Available memory: 21732462592, total memory: 65452113920
(EngineCore_1 pid=520079) INFO 08-05 11:22:57 [kv_cache_utils.py:833] GPU KV cache size: 221,056 tokens
(EngineCore_1 pid=520079) INFO 08-05 11:22:57 [kv_cache_utils.py:837] Maximum concurrency for 40,960 tokens per request: 5.40x
[rank0]:[W805 11:23:04.368990660 compiler_depend.ts:149] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
[rank1]:[W805 11:23:04.390677935 compiler_depend.ts:149] Warning: Waiting for pending NCCL work to finish before starting graph capture. (function operator())
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632] EngineCore failed to start.
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632] Traceback (most recent call last):
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 621, in run_engine_core
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 881, in __init__
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     super().__init__(vllm_config, local_client, handshake_address,
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 441, in __init__
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 86, in __init__
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     self._initialize_kv_caches(vllm_config)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 190, in _initialize_kv_caches
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/executor/abstract.py", line 66, in initialize_from_config
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     self.collective_rpc("compile_or_warm_up_model")
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/utils/__init__.py", line 2985, in run_method
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return func(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 261, in compile_or_warm_up_model
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     self.model_runner.capture_model()
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2386, in capture_model
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     self._dummy_run(num_tokens)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return func(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1940, in _dummy_run
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     hidden_states = model(
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return forward_call(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen3_moe.py", line 527, in forward
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/compilation/decorators.py", line 279, in __call__
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     model_output = self.forward(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen3_moe.py", line 350, in forward
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     def forward(
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return forward_call(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return fn(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return self._wrapped_call(self, *args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 406, in __call__
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     raise e
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 393, in __call__
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     return forward_call(*args, **kwargs)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "<eval_with_key>.98", line 374, in forward
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     submod_8 = self.submod_8(getitem_18, s0, l_self_modules_layers_modules_3_modules_self_attn_modules_o_proj_parameters_weight_, getitem_19, l_self_modules_layers_modules_3_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_3_modules_mlp_modules_gate_parameters_weight_, l_self_modules_layers_modules_4_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, s1, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_18 = l_self_modules_layers_modules_3_modules_self_attn_modules_o_proj_parameters_weight_ = getitem_19 = l_self_modules_layers_modules_3_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_3_modules_mlp_modules_gate_parameters_weight_ = l_self_modules_layers_modules_4_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_k_norm_parameters_weight_ = None
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-ascend/vllm_ascend/compilation/piecewise_backend.py", line 192, in __call__
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     with torch.npu.graph(aclgraph, pool=self.graph_pool):
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 310, in __enter__
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     self.npu_graph.capture_begin(
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 210, in capture_begin
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]     super().capture_begin(pool=pool, capture_error_mode=capture_error_mode)
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632] RuntimeError: capture_begin:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:156 NPU function error: c10_npu::acl::AclmdlRICaptureBegin(capture_stream_, capture_mode), error code is 207005
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632] [ERROR] 2025-08-05-11:23:25 (PID:519998, Device:0, RankID:-1) ERR00100 PTA call acl api failed
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632] [Error]: Failed to apply for memory.
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         Check the remaining storage space in the hardware environment.
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632] EL0006: [PID: 519998] 2025-08-05-11:23:25.691.093 The resources are insufficient.
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         Solution: Close applications not in use.
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         TraceBack (most recent call last):
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         rtStreamBeginCapture execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         begin capture stream failed, runtime result = 207005[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         stream setup failed, retCode=0x7020023.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:6109]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         Sync get device msg failed, retCode=0x7020023.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6159]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632]         rtGetDevMsg execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_0 pid=519998) ERROR 08-05 11:23:25 [core.py:632] 
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632] EngineCore failed to start.
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632] Traceback (most recent call last):
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 621, in run_engine_core
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 881, in __init__
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     super().__init__(vllm_config, local_client, handshake_address,
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 441, in __init__
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 86, in __init__
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     self._initialize_kv_caches(vllm_config)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 190, in _initialize_kv_caches
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/executor/abstract.py", line 66, in initialize_from_config
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     self.collective_rpc("compile_or_warm_up_model")
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/utils/__init__.py", line 2985, in run_method
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return func(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 261, in compile_or_warm_up_model
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     self.model_runner.capture_model()
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2386, in capture_model
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     self._dummy_run(num_tokens)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return func(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1940, in _dummy_run
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     hidden_states = model(
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return self._call_impl(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return forward_call(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen3_moe.py", line 527, in forward
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/compilation/decorators.py", line 279, in __call__
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     model_output = self.forward(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen3_moe.py", line 350, in forward
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     def forward(
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return self._call_impl(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return forward_call(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return fn(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return self._wrapped_call(self, *args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 406, in __call__
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     raise e
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 393, in __call__
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return self._call_impl(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     return forward_call(*args, **kwargs)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "<eval_with_key>.98", line 374, in forward
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     submod_8 = self.submod_8(getitem_18, s0, l_self_modules_layers_modules_3_modules_self_attn_modules_o_proj_parameters_weight_, getitem_19, l_self_modules_layers_modules_3_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_3_modules_mlp_modules_gate_parameters_weight_, l_self_modules_layers_modules_4_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, s1, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_18 = l_self_modules_layers_modules_3_modules_self_attn_modules_o_proj_parameters_weight_ = getitem_19 = l_self_modules_layers_modules_3_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_3_modules_mlp_modules_gate_parameters_weight_ = l_self_modules_layers_modules_4_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_k_norm_parameters_weight_ = None
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/code/vllm-ascend/vllm_ascend/compilation/piecewise_backend.py", line 192, in __call__
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     with torch.npu.graph(aclgraph, pool=self.graph_pool):
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 310, in __enter__
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     self.npu_graph.capture_begin(
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 210, in capture_begin
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]     super().capture_begin(pool=pool, capture_error_mode=capture_error_mode)
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632] RuntimeError: capture_begin:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:156 NPU function error: c10_npu::acl::AclmdlRICaptureBegin(capture_stream_, capture_mode), error code is 207005
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632] [ERROR] 2025-08-05-11:23:25 (PID:520079, Device:0, RankID:-1) ERR00100 PTA call acl api failed
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632] [Error]: Failed to apply for memory.
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         Check the remaining storage space in the hardware environment.
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632] EL0006: [PID: 520079] 2025-08-05-11:23:25.780.040 The resources are insufficient.
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         Solution: Close applications not in use.
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         TraceBack (most recent call last):
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         rtStreamBeginCapture execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         begin capture stream failed, runtime result = 207005[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         stream setup failed, retCode=0x7020023.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:6109]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         Sync get device msg failed, retCode=0x7020023.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6159]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632]         rtGetDevMsg execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_1 pid=520079) ERROR 08-05 11:23:25 [core.py:632] 
(EngineCore_0 pid=519998) Process EngineCore_0:
(EngineCore_0 pid=519998) Traceback (most recent call last):
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_0 pid=519998)     self.run()
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_0 pid=519998)     self._target(*self._args, **self._kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 636, in run_engine_core
(EngineCore_0 pid=519998)     raise e
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 621, in run_engine_core
(EngineCore_0 pid=519998)     engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 881, in __init__
(EngineCore_0 pid=519998)     super().__init__(vllm_config, local_client, handshake_address,
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 441, in __init__
(EngineCore_0 pid=519998)     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 86, in __init__
(EngineCore_0 pid=519998)     self._initialize_kv_caches(vllm_config)
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 190, in _initialize_kv_caches
(EngineCore_0 pid=519998)     self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/executor/abstract.py", line 66, in initialize_from_config
(EngineCore_0 pid=519998)     self.collective_rpc("compile_or_warm_up_model")
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_0 pid=519998)     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/utils/__init__.py", line 2985, in run_method
(EngineCore_0 pid=519998)     return func(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 261, in compile_or_warm_up_model
(EngineCore_0 pid=519998)     self.model_runner.capture_model()
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2386, in capture_model
(EngineCore_0 pid=519998)     self._dummy_run(num_tokens)
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(EngineCore_0 pid=519998)     return func(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1940, in _dummy_run
(EngineCore_0 pid=519998)     hidden_states = model(
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_0 pid=519998)     return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_0 pid=519998)     return forward_call(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen3_moe.py", line 527, in forward
(EngineCore_0 pid=519998)     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/compilation/decorators.py", line 279, in __call__
(EngineCore_0 pid=519998)     model_output = self.forward(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen3_moe.py", line 350, in forward
(EngineCore_0 pid=519998)     def forward(
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_0 pid=519998)     return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_0 pid=519998)     return forward_call(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(EngineCore_0 pid=519998)     return fn(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(EngineCore_0 pid=519998)     return self._wrapped_call(self, *args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 406, in __call__
(EngineCore_0 pid=519998)     raise e
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 393, in __call__
(EngineCore_0 pid=519998)     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_0 pid=519998)     return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_0 pid=519998)     return forward_call(*args, **kwargs)
(EngineCore_0 pid=519998)   File "<eval_with_key>.98", line 374, in forward
(EngineCore_0 pid=519998)     submod_8 = self.submod_8(getitem_18, s0, l_self_modules_layers_modules_3_modules_self_attn_modules_o_proj_parameters_weight_, getitem_19, l_self_modules_layers_modules_3_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_3_modules_mlp_modules_gate_parameters_weight_, l_self_modules_layers_modules_4_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, s1, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_18 = l_self_modules_layers_modules_3_modules_self_attn_modules_o_proj_parameters_weight_ = getitem_19 = l_self_modules_layers_modules_3_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_3_modules_mlp_modules_gate_parameters_weight_ = l_self_modules_layers_modules_4_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_k_norm_parameters_weight_ = None
(EngineCore_0 pid=519998)   File "/home/xxx/code/vllm-ascend/vllm_ascend/compilation/piecewise_backend.py", line 192, in __call__
(EngineCore_0 pid=519998)     with torch.npu.graph(aclgraph, pool=self.graph_pool):
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 310, in __enter__
(EngineCore_0 pid=519998)     self.npu_graph.capture_begin(
(EngineCore_0 pid=519998)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 210, in capture_begin
(EngineCore_0 pid=519998)     super().capture_begin(pool=pool, capture_error_mode=capture_error_mode)
(EngineCore_0 pid=519998) RuntimeError: capture_begin:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:156 NPU function error: c10_npu::acl::AclmdlRICaptureBegin(capture_stream_, capture_mode), error code is 207005
(EngineCore_0 pid=519998) [ERROR] 2025-08-05-11:23:25 (PID:519998, Device:0, RankID:-1) ERR00100 PTA call acl api failed
(EngineCore_0 pid=519998) [Error]: Failed to apply for memory.
(EngineCore_0 pid=519998)         Check the remaining storage space in the hardware environment.
(EngineCore_0 pid=519998) EL0006: [PID: 519998] 2025-08-05-11:23:25.691.093 The resources are insufficient.
(EngineCore_0 pid=519998)         Solution: Close applications not in use.
(EngineCore_0 pid=519998)         TraceBack (most recent call last):
(EngineCore_0 pid=519998)         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_0 pid=519998)         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_0 pid=519998)         rtStreamBeginCapture execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_0 pid=519998)         begin capture stream failed, runtime result = 207005[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(EngineCore_0 pid=519998)         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_0 pid=519998)         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_0 pid=519998)         stream setup failed, retCode=0x7020023.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:6109]
(EngineCore_0 pid=519998)         Sync get device msg failed, retCode=0x7020023.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6159]
(EngineCore_0 pid=519998)         rtGetDevMsg execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_0 pid=519998) 
(EngineCore_1 pid=520079) Process EngineCore_1:
(EngineCore_1 pid=520079) Traceback (most recent call last):
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_1 pid=520079)     self.run()
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_1 pid=520079)     self._target(*self._args, **self._kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 636, in run_engine_core
(EngineCore_1 pid=520079)     raise e
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 621, in run_engine_core
(EngineCore_1 pid=520079)     engine_core = DPEngineCoreProc(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 881, in __init__
(EngineCore_1 pid=520079)     super().__init__(vllm_config, local_client, handshake_address,
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 441, in __init__
(EngineCore_1 pid=520079)     super().__init__(vllm_config, executor_class, log_stats,
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 86, in __init__
(EngineCore_1 pid=520079)     self._initialize_kv_caches(vllm_config)
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core.py", line 190, in _initialize_kv_caches
(EngineCore_1 pid=520079)     self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/executor/abstract.py", line 66, in initialize_from_config
(EngineCore_1 pid=520079)     self.collective_rpc("compile_or_warm_up_model")
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_1 pid=520079)     answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/utils/__init__.py", line 2985, in run_method
(EngineCore_1 pid=520079)     return func(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 261, in compile_or_warm_up_model
(EngineCore_1 pid=520079)     self.model_runner.capture_model()
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 2386, in capture_model
(EngineCore_1 pid=520079)     self._dummy_run(num_tokens)
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(EngineCore_1 pid=520079)     return func(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1940, in _dummy_run
(EngineCore_1 pid=520079)     hidden_states = model(
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_1 pid=520079)     return self._call_impl(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_1 pid=520079)     return forward_call(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen3_moe.py", line 527, in forward
(EngineCore_1 pid=520079)     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/compilation/decorators.py", line 279, in __call__
(EngineCore_1 pid=520079)     model_output = self.forward(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-cpu/vllm/vllm/model_executor/models/qwen3_moe.py", line 350, in forward
(EngineCore_1 pid=520079)     def forward(
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_1 pid=520079)     return self._call_impl(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_1 pid=520079)     return forward_call(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(EngineCore_1 pid=520079)     return fn(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(EngineCore_1 pid=520079)     return self._wrapped_call(self, *args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 406, in __call__
(EngineCore_1 pid=520079)     raise e
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/fx/graph_module.py", line 393, in __call__
(EngineCore_1 pid=520079)     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(EngineCore_1 pid=520079)     return self._call_impl(*args, **kwargs)
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(EngineCore_1 pid=520079)     return forward_call(*args, **kwargs)
(EngineCore_1 pid=520079)   File "<eval_with_key>.98", line 374, in forward
(EngineCore_1 pid=520079)     submod_8 = self.submod_8(getitem_18, s0, l_self_modules_layers_modules_3_modules_self_attn_modules_o_proj_parameters_weight_, getitem_19, l_self_modules_layers_modules_3_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_3_modules_mlp_modules_gate_parameters_weight_, l_self_modules_layers_modules_4_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_qkv_proj_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_4_modules_self_attn_modules_k_norm_parameters_weight_, l_positions_, s1, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_18 = l_self_modules_layers_modules_3_modules_self_attn_modules_o_proj_parameters_weight_ = getitem_19 = l_self_modules_layers_modules_3_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_3_modules_mlp_modules_gate_parameters_weight_ = l_self_modules_layers_modules_4_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_4_modules_self_attn_modules_k_norm_parameters_weight_ = None
(EngineCore_1 pid=520079)   File "/home/xxx/code/vllm-ascend/vllm_ascend/compilation/piecewise_backend.py", line 192, in __call__
(EngineCore_1 pid=520079)     with torch.npu.graph(aclgraph, pool=self.graph_pool):
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 310, in __enter__
(EngineCore_1 pid=520079)     self.npu_graph.capture_begin(
(EngineCore_1 pid=520079)   File "/home/xxx/miniconda3/envs/atb/lib/python3.10/site-packages/torch_npu/npu/graphs.py", line 210, in capture_begin
(EngineCore_1 pid=520079)     super().capture_begin(pool=pool, capture_error_mode=capture_error_mode)
(EngineCore_1 pid=520079) RuntimeError: capture_begin:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:156 NPU function error: c10_npu::acl::AclmdlRICaptureBegin(capture_stream_, capture_mode), error code is 207005
(EngineCore_1 pid=520079) [ERROR] 2025-08-05-11:23:25 (PID:520079, Device:0, RankID:-1) ERR00100 PTA call acl api failed
(EngineCore_1 pid=520079) [Error]: Failed to apply for memory.
(EngineCore_1 pid=520079)         Check the remaining storage space in the hardware environment.
(EngineCore_1 pid=520079) EL0006: [PID: 520079] 2025-08-05-11:23:25.780.040 The resources are insufficient.
(EngineCore_1 pid=520079)         Solution: Close applications not in use.
(EngineCore_1 pid=520079)         TraceBack (most recent call last):
(EngineCore_1 pid=520079)         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_1 pid=520079)         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_1 pid=520079)         rtStreamBeginCapture execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_1 pid=520079)         begin capture stream failed, runtime result = 207005[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(EngineCore_1 pid=520079)         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_1 pid=520079)         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_1 pid=520079)         stream setup failed, retCode=0x7020023.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:6109]
(EngineCore_1 pid=520079)         Sync get device msg failed, retCode=0x7020023.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6159]
(EngineCore_1 pid=520079)         rtGetDevMsg execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_1 pid=520079) 
Process Process-1:
Traceback (most recent call last):
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xxx/code/vllm-ascend/examples/offline_data_parallel.py", line 182, in main
    llm = LLM(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/entrypoints/llm.py", line 273, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/engine/llm_engine.py", line 497, in from_engine_args
    return engine_cls.from_vllm_config(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/llm_engine.py", line 126, in from_vllm_config
    return cls(vllm_config=vllm_config,
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/llm_engine.py", line 103, in __init__
    self.engine_core = EngineCoreClient.make_client(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core_client.py", line 77, in make_client
    return SyncMPClient(vllm_config, executor_class, log_stats)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core_client.py", line 514, in __init__
    super().__init__(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core_client.py", line 408, in __init__
    with launch_core_engines(vllm_config, executor_class,
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/utils.py", line 697, in launch_core_engines
    wait_for_engine_startup(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/utils.py", line 750, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Process Process-2:
Traceback (most recent call last):
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xxx/code/vllm-ascend/examples/offline_data_parallel.py", line 182, in main
    llm = LLM(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/entrypoints/llm.py", line 273, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/engine/llm_engine.py", line 497, in from_engine_args
    return engine_cls.from_vllm_config(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/llm_engine.py", line 126, in from_vllm_config
    return cls(vllm_config=vllm_config,
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/llm_engine.py", line 103, in __init__
    self.engine_core = EngineCoreClient.make_client(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core_client.py", line 77, in make_client
    return SyncMPClient(vllm_config, executor_class, log_stats)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core_client.py", line 514, in __init__
    super().__init__(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/core_client.py", line 408, in __init__
    with launch_core_engines(vllm_config, executor_class,
  File "/home/xxx/miniconda3/envs/atb/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/utils.py", line 697, in launch_core_engines
    wait_for_engine_startup(
  File "/home/xxx/code/vllm-cpu/vllm/vllm/v1/engine/utils.py", line 750, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

[22dimensions]

Qwen3-8B-W4A8 test report:

model file url is https://www.modelscope.cn/models/vllm-ascend/Qwen3-8B-W4A8 which is converted by master version modelslim

test script:

import torch

from vllm import LLM, SamplingParams

import os
os.environ["VLLM_USE_V1"] = "1"

prompts = [
    "Hello, my name is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=40)

llm = LLM(model="/root/.cache/modelscope/hub/models/vllm-ascend/Qwen3-8B-W4A8",
          max_model_len=2048,
          trust_remote_code=True,
          quantization="ascend")

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

test result:

INFO 08-05 12:24:16 [default_loader.py:262] Loading weights took 3.95 seconds
INFO 08-05 12:24:44 [model_runner_v1.py:2015] Loading model weights took 8.2380 GB
INFO 08-05 12:24:55 [backends.py:530] Using cache directory: /root/.cache/vllm/torch_compile_cache/6955c13859/rank_0_0/backbone for vLLM's torch.compile
INFO 08-05 12:24:55 [backends.py:541] Dynamo bytecode transform time: 9.74 s
INFO 08-05 12:24:59 [backends.py:215] Compiling a graph for dynamic shape takes 2.84 s
INFO 08-05 12:25:10 [monitor.py:34] torch.compile takes 12.58 s in total
INFO 08-05 12:25:12 [worker_v1.py:186] Available memory: 47774801612, total memory: 65464696832
INFO 08-05 12:25:12 [kv_cache_utils.py:833] GPU KV cache size: 323,968 tokens
INFO 08-05 12:25:12 [kv_cache_utils.py:837] Maximum concurrency for 2,048 tokens per request: 158.19x
INFO 08-05 12:25:48 [model_runner_v1.py:2340] Graph capturing finished in 36 secs, took 0.47 GiB
INFO 08-05 12:25:48 [core.py:193] init engine (profile, create kv cache, warmup model) took 63.64 seconds
INFO 08-05 12:25:49 [platform.py:162] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
INFO 08-05 12:25:49 [utils.py:333] Calculated maximum supported batch sizes for ACL graph: 51
INFO 08-05 12:25:49 [utils.py:359] No adjustment needed for ACL graph batch sizes: Qwen3ForCausalLM model (layers: 36) with 51 sizes
Adding requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 101.95it/s]
Processed prompts: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  4.03it/s, est. speed input: 20.18 toks/s, output: 64.59 toks/s]
Prompt: 'Hello, my name is', Generated text: " [Your Name] and I'm a [Your Profession] with a passion for"
Prompt: 'The future of AI is', Generated text: ' a topic that has been the subject of much discussion and speculation. While some believe'

precision test:

command:

lm_eval \
  --model vllm \
  --model_args pretrained=/root/.cache/modelscope/hub/models/vllm-ascend/Qwen3-8B-W4A8,max_model_len=2048,tensor_parallel_size=1,quantization=ascend \
  --tasks gsm8k\
  --batch_size 8

result:

vllm (pretrained=/root/.cache/modelscope/hub/models/vllm-ascend/Qwen3-8B-W4A8,max_model_len=2048,tensor_parallel_size=1,quantization=ascend), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 8
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8673|±  |0.0093|
|     |       |strict-match    |     5|exact_match|↑  |0.8620|±  |0.0095|

[zhangxinyuehfad]

✅Disaggregate prefill on multi-node + deepseek + torchair graph mode + AscendScheduler (disabling chunked prefill) + dp + ep + tp

command:

cd examples/disaggregate_prefill_v1/

bash gen_ranktable.sh --ips 172.22.0.188 172.22.0.212  \
  --npus-per-node 16 --network-card-name enp23s0f3 --prefill-device-cnt 16 --decode-device-cnt 16
  

prefill

#!/bin/sh

export HCCL_IF_IP=172.22.0.188  # node ip
export GLOO_SOCKET_IFNAME="enp23s0f3"  # network card name
export TP_SOCKET_IFNAME="enp23s0f3"
export HCCL_SOCKET_IFNAME="enp23s0f3"
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
  --host 0.0.0.0 \
  --port 20002 \
  --data-parallel-size 2 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --seed 1024 \
  --trust-remote-code \
  --served-model-name deepseek_v3 \
  --quantization ascend \
  --gpu-memory-utilization 0.9  \
  --max-model-len 32768 \
  --additional-config '{"ascend_scheduler_config":{"enabled":true, "enable_chunked_prefill": false},"torchair_graph_config":{"enabled":true}}' \
  --kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
  "kv_buffer_device": "npu",
  "kv_role": "kv_producer",
  "kv_parallel_size": 1,
  "kv_port": "20001",
  "engine_id": "0",
  "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }'  &
 

decode

#!/bin/sh

export HCCL_IF_IP=172.22.0.212
export GLOO_SOCKET_IFNAME="enp23s0f3"
export TP_SOCKET_IFNAME="enp23s0f3"
export HCCL_SOCKET_IFNAME="enp23s0f3"
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=600

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
  --host 0.0.0.0 \
  --port 20002 \
  --data-parallel-size 4 \
  --api-server-count 2 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --served-model-name deepseek_v3 \
  --seed 1024 \
  --max-model-len 32768  \
  --max-num-batched-tokens 256  \
  --max-num-seqs 256 \
  --quantization ascend \
  --trust-remote-code \
  --gpu-memory-utilization 0.9  \
   --additional-config '{"ascend_scheduler_config":{"enabled":true, "enable_chunked_prefill": false},"torchair_graph_config":{"enabled":true}}' \
  --kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
  "kv_buffer_device": "npu",
  "kv_role": "kv_consumer",
  "kv_parallel_size": 1,
  "kv_port": "20001",
  "engine_id": "0",
  "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }'  &

proxy

python load_balance_proxy_server_example.py --host 0.0.0.0  --port 1025 --prefiller-hosts 172.22.0.188 --prefiller-port 20002 --decoder-hosts 172.22.0.212 --decoder-ports 20002

output：

root@liteserver-for-vllm-ascend-00003:/workspace# #!/bin/bash
N=10

for i in $(seq 1 $N); do
  curl http://localhost:1025/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek_v3",
        "prompt": "The future of AI is",
        "max_tokens": 100,
        "temperature": 0.5
    }'
done

wait
{"id":"cmpl-2","object":"text_completion","created":1754470911,"model":"deepseek_v3","choices":[{"index":0,"text":" here, and it’s changing the way we live, work, and learn. In this article, we explore the latest advancements in AI technology, its impact on various industries, and what the future holds for this rapidly evolving field.\n\nArtificial intelligence (AI) has been a buzzword for years, but recent advancements in technology have brought us closer than ever to realizing its full potential. From self-driving cars to virtual assistants, AI is transforming the way we live, work, and learn. In this","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-3","object":"text_completion","created":1754470921,"model":"deepseek_v3","choices":[{"index":0,"text":" a topic of much speculation and debate. Some experts believe that AI will eventually surpass human intelligence, leading to a technological singularity. Others believe that AI will continue to be a tool that humans use to augment their own capabilities.\n\nThere is no doubt that AI is already having a major impact on our world. AI-powered systems are being used in a wide range of industries, from healthcare to finance to transportation. As AI continues to develop, it is likely to have an even greater impact on our lives.\n\nHere","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-4","object":"text_completion","created":1754470929,"model":"deepseek_v3","choices":[{"index":0,"text":" a topic of much debate and speculation. Some experts believe that AI will eventually surpass human intelligence and lead to a technological singularity, while others believe that AI will continue to be a tool that is used to augment human capabilities.\n\nThere is no doubt that AI is already having a major impact on our world. AI-powered systems are being used to automate tasks, improve decision-making, and create new products and services. As AI continues to develop, it is likely to have an even greater impact on our lives.\n\n","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-5","object":"text_completion","created":1754470937,"model":"deepseek_v3","choices":[{"index":0,"text":" not just about technological advancements but also about its integration into our daily lives and industries. As AI continues to evolve, it will undoubtedly shape the future in ways we are only beginning to understand.\n\nArtificial Intelligence (AI) has rapidly evolved from a futuristic concept to a transformative force across various industries. As we look ahead, the future of AI promises even more groundbreaking advancements that will reshape our world. This article explores the key trends, potential applications, and challenges that lie ahead in the realm of AI.\n\n","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-6","object":"text_completion","created":1754470945,"model":"deepseek_v3","choices":[{"index":0,"text":" here, and it’s called ChatGPT. This revolutionary technology has the potential to change the way we interact with machines forever. But what exactly is ChatGPT, and how does it work? In this article, we’ll take a closer look at this cutting-edge technology and explore its many applications.\n\nWhat is ChatGPT?\n\nChatGPT is a natural language processing (NLP) model developed by OpenAI. It’s based on the GPT (Generative Pre-trained Transformer) architecture, which is a type of","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-7","object":"text_completion","created":1754470953,"model":"deepseek_v3","choices":[{"index":0,"text":" not just about technological advancements but also about how these technologies integrate into and enhance our daily lives. As we look ahead, several key trends and predictions stand out:\n\n### 1. Ubiquitous AI Integration\n\nAI will become seamlessly integrated into everyday life, much like electricity or the internet. From smart homes to personalized education, AI will enhance efficiency and convenience across various sectors. This integration will be so natural that users may not even be aware they are interacting with AI.\n\n### 2. Ethical AI","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-8","object":"text_completion","created":1754470961,"model":"deepseek_v3","choices":[{"index":0,"text":" not just about smarter machines; it’s about creating a symbiotic relationship between humans and technology. As AI continues to evolve, it will increasingly augment human capabilities, enabling us to solve complex problems and make more informed decisions. This partnership will redefine industries, enhance creativity, and improve quality of life, making the future of AI a collaborative journey towards unprecedented possibilities.\n\nArtificial Intelligence (AI) has become a cornerstone of modern technology, influencing various aspects of our daily lives and reshaping industries. From healthcare to finance,","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-9","object":"text_completion","created":1754470969,"model":"deepseek_v3","choices":[{"index":0,"text":" here, and it’s called ChatGPT. This powerful language model developed by OpenAI is revolutionizing the way we interact with technology. With its ability to understand and respond to natural language, ChatGPT is changing the game for businesses, developers, and consumers alike.\n\nChatGPT, or Generative Pre-trained Transformer, is a state-of-the-art language model that uses deep learning to generate human-like text. It’s trained on a massive dataset of diverse text sources, allowing it to understand and respond to a wide range","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-10","object":"text_completion","created":1754470977,"model":"deepseek_v3","choices":[{"index":0,"text":" a topic of much speculation and debate. Some experts believe that AI will eventually surpass human intelligence, while others believe that it will remain a tool that humans use to augment their own capabilities.\n\nThere are a number of factors that will likely shape the future of AI. These include:\n\n- The development of new AI algorithms and techniques: As AI research continues, we can expect to see the development of new algorithms and techniques that will make AI more powerful and versatile. For example, researchers are working on developing AI","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}{"id":"cmpl-11","object":"text_completion","created":1754470985,"model":"deepseek_v3","choices":[{"index":0,"text":" here, and it’s called ChatGPT. This revolutionary technology has the potential to transform the way we interact with machines, and it’s already making waves in industries ranging from healthcare to finance. But what exactly is ChatGPT, and how does it work? In this article, we’ll explore the ins and outs of this cutting-edge technology, from its origins to its potential applications. We’ll also take a closer look at the benefits and drawbacks of ChatGPT, and what it means for the future","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}root@liteserver-for-vllm-ascend-00003:/workspace#

[shen-shanshan]

✅ Test Ngram

Run:

vllm serve /shared/cache/modelscope/hub/models/LLM-Research/Meta-Llama-3.1-8B-Instruct \
--max-model-len 1024 \
--speculative-config '{"method": "ngram", "num_speculative_tokens": 3, "prompt_lookup_max": 5, "prompt_lookup_min": 3}' \
--gpu_memory_utilization 0.9 \
--trust-remote-code \
--enforce-eager

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/shared/cache/modelscope/hub/models/LLM-Research/Meta-Llama-3.1-8B-Instruct",
        "prompt": "The future of AI is",
        "max_tokens": 100,
        "temperature": 0
    }'

Output:

{"id":"cmpl-052159faf86245f899b93b5eb6ef7309","object":"text_completion","created":1754463011,"model":"/shared/cache/modelscope/hub/models/LLM-Research/Meta-Llama-3.1-8B-Instruct","choices":[{"index":0,"text":" bright, but it also raises concerns about bias, accountability, and the potential for AI to exacerbate existing social inequalities. To address these concerns, researchers and policymakers are exploring ways to develop more transparent, explainable, and fair AI systems.\nOne approach is to use techniques such as model interpretability and explainability to provide insights into how AI systems make decisions. This can help identify potential biases and ensure that AI systems are fair and transparent.\nAnother approach is to use data that is representative of diverse populations","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

✅ Test Eagle3

Run:

vllm serve /shared/cache/modelscope/hub/models/LLM-Research/Meta-Llama-3.1-8B-Instruct \
--max-model-len 1024 \
--speculative-config '{"method": "eagle3", "num_speculative_tokens": 2, "max_model_len": 128, "model": "/home/sss/models/models/models/vllm-ascend/EAGLE3-LLaMA3.1-Instruct-8B"}' \
--gpu_memory_utilization 0.9 \
--trust-remote-code \
--enforce-eager

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/shared/cache/modelscope/hub/models/LLM-Research/Meta-Llama-3.1-8B-Instruct",
        "prompt": "The future of AI is",
        "max_tokens": 100,
        "temperature": 0
    }'

Output:

{"id":"cmpl-741eca1b3bc6495a9ca3f81f54e51704","object":"text_completion","created":1754463381,"model":"/shared/cache/modelscope/hub/models/LLM-Research/Meta-Llama-3.1-8B-Instruct","choices":[{"index":0,"text":" bright, but it also raises concerns about bias, accountability, and the potential for AI to exacerbate existing social inequalities. To address these concerns, researchers and policymakers are exploring ways to develop more transparent, explainable, and fair AI systems.\nOne approach is to use techniques such as model interpretability and explainability to provide insights into how AI systems make decisions. This can help identify potential biases and ensure that AI systems are fair and transparent.\nAnother approach is to use data that is representative of diverse populations","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

Test Mtp

Run:

git pr 2145

vllm serve /home/sss/models/models/models/vllm-ascend/DeepSeek-R1-W8A8 \
--max-model-len 1024 \
--tensor-parallel-size 4 \
--speculative-config '{"method":"deepseek_mtp", "num_speculative_tokens": 1}' \
--quantization ascend \
--additional-config '{"ascend_scheduler_config": {"enabled": true, "enable_chunked_prefill": false}, "torchair_graph_config": {"enabled": true, "graph_batch_sizes": [16]}, "enable_weight_nz_layout": true}' \
--gpu_memory_utilization 0.9 \
--no-enable-prefix-caching \
--trust-remote-code

Output:

Logs

[Potabk]

Disaggregate prefill

✅ multi-node + deepseek + torchair graph mode + V1 scheduler + dp + ep + tp

details:

cd vllm-ascend/examples/disaggregate_prefill_v1/
bash gen_ranktable.sh --ips 172.22.0.218 172.22.0.155  \
  --npus-per-node 16 --network-card-name enp23s0f3 --prefill-device-cnt 16 --decode-device-cnt 16

prefill

#!/bin/sh

export HCCL_IF_IP=172.22.0.218  # node ip
export GLOO_SOCKET_IFNAME="enp23s0f3"  # network card name
export TP_SOCKET_IFNAME="enp23s0f3"
export HCCL_SOCKET_IFNAME="enp23s0f3"
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export OMP_PROC_BIND=false
export HCCL_BUFFSIZE=1024
export OMP_NUM_THREADS=100

vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
  --host 0.0.0.0 \
  --port 20002 \
  --data-parallel-size 2 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --seed 1024 \
  --enforce-eager \
  --trust-remote-code \
  --quantization ascend \
  --gpu-memory-utilization 0.9  \
  --kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
  "kv_buffer_device": "npu",
  "kv_role": "kv_producer",
  "kv_parallel_size": 1,
  "kv_port": "20001",
  "engine_id": "0",
  "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }'  \
  --additional-config \
  '{"chunked_prefill_for_mla":true}' 

decode

#!/bin/sh

export HCCL_IF_IP=172.22.0.155
export GLOO_SOCKET_IFNAME="enp23s0f3"
export TP_SOCKET_IFNAME="enp23s0f3"
export HCCL_SOCKET_IFNAME="enp23s0f3"
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024

vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
  --host 0.0.0.0 \
  --port 20002 \
  --data-parallel-size 4 \
  --api-server-count 2 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --seed 1024 \
  --max-model-len 8192  \
  --max-num-seqs 125 \
  --quantization ascend \
  --trust-remote-code \
  --gpu-memory-utilization 0.9  \
  --kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
  "kv_buffer_device": "npu",
  "kv_role": "kv_consumer",
  "kv_parallel_size": 1,
  "kv_port": "20001",
  "engine_id": "0",
  "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }'  \
  --additional-config \
  '{"torchair_graph_config": {"enabled":true}}' 

proxy

python load_balance_proxy_server_example.py --host 0.0.0.0  --port 1025 --prefiller-hosts 172.22.0.218 --prefiller-port 20002 --decoder-hosts 172.22.0.155 --decoder-ports 20002

result

curl http://localhost:1025/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8",
        "prompt": "Who are you?",
        "max_tokens": 100,
        "temperature": 0
    }'
{"id":"cmpl-1","object":"text_completion","created":1754469116,"model":"/home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8","choices":[{"index":0,"text":" What do you do? How long have you been at it?\n\nMy name is Chris and I’m a 3D artist and animator. I’ve been working professionally in the field for about 5 years now. I’ve worked on a variety of projects ranging from video games to TV commercials to feature films.\n\nWhat inspired you to get into this line of work?\n\nI’ve always been interested in art and animation, ever since I was a kid. I remember watching cartoons and","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":5,"total_tokens":105,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

performance
qps 1

vllm bench serve --model /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8  \
--dataset-name random --random-input-len 128 --random-output-len 128 \
--num-prompts 200  --trust-remote-code --base-url "http://localhost:1025" --request-rate 1

============ Serving Benchmark Result ============
Successful requests:                     200
Benchmark duration (s):                  206.17
Total input tokens:                      25291
Total generated tokens:                  4886
Request throughput (req/s):              0.97
Output token throughput (tok/s):         23.70
Total Token throughput (tok/s):          146.37
---------------Time to First Token----------------
Mean TTFT (ms):                          703.75
Median TTFT (ms):                        700.86
P99 TTFT (ms):                           890.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          74.90
Median TPOT (ms):                        75.30
P99 TPOT (ms):                           96.70
---------------Inter-token Latency----------------
Mean ITL (ms):                           76.51
Median ITL (ms):                         76.02
P99 ITL (ms):                            116.05
==================================================

qps 10

vllm bench serve --model /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8  \
--dataset-name random --random-input-len 128 --random-output-len 128 \
--num-prompts 200  --trust-remote-code --base-url "http://localhost:1025" --request-rate 10

============ Serving Benchmark Result ============
Successful requests:                     200
Benchmark duration (s):                  33.76
Total input tokens:                      25291
Total generated tokens:                  4517
Request throughput (req/s):              5.92
Output token throughput (tok/s):         133.81
Total Token throughput (tok/s):          883.00
---------------Time to First Token----------------
Mean TTFT (ms):                          9425.87
Median TTFT (ms):                        9868.82
P99 TTFT (ms):                           19101.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          137.85
Median TPOT (ms):                        106.72
P99 TPOT (ms):                           316.91
---------------Inter-token Latency----------------
Mean ITL (ms):                           103.81
Median ITL (ms):                         104.02
P99 ITL (ms):                            245.72
==================================================

[Potabk]

❌ Disaggregate prefill on multi-node + qwen3 235B + aclgraph + V1 scheduler + dp + ep + tp

❌ Disaggregate prefill on single-node + qwen3 235B + aclgraph + V1 scheduler + dp + ep + tp

✅ Disaggregate prefill on multi-node + qwen3 235B + aclgraph + V1 scheduler + ep + tp(without dp)

tp+ep+aclgraph success

Disaggregate prefill on multi-node + qwen3 235B + aclgraph + V1 scheduler + dp + ep + tp

#!/bin/sh

export HCCL_IF_IP=172.22.0.218  # node ip
export GLOO_SOCKET_IFNAME="enp23s0f3"  # network card name
export TP_SOCKET_IFNAME="enp23s0f3"
export HCCL_SOCKET_IFNAME="enp23s0f3"
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export OMP_PROC_BIND=false
export HCCL_BUFFSIZE=1024
export OMP_NUM_THREADS=100

vllm serve /home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B \
  --host 0.0.0.0 \
  --port 20002 \
  --data-parallel-size 2 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --seed 1024 \
  --enforce-eager \
  --max-model-len 8192  \
  --max-num-seqs 125 \
  --trust-remote-code \
  --gpu-memory-utilization 0.9  \
  --kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
  "kv_buffer_device": "npu",
  "kv_role": "kv_producer",
  "kv_parallel_size": 1,
  "kv_port": "20001",
  "engine_id": "0",
  "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }' 

decode

#!/bin/sh

export HCCL_IF_IP=172.22.0.155
export GLOO_SOCKET_IFNAME="enp23s0f3"
export TP_SOCKET_IFNAME="enp23s0f3"
export HCCL_SOCKET_IFNAME="enp23s0f3"
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024

vllm serve /home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B \
  --host 0.0.0.0 \
  --port 20002 \
  --data-parallel-size 4 \
  --api-server-count 2 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --seed 1024 \
  --max-model-len 8192  \
  --max-num-seqs 125 \
  --trust-remote-code \
  --gpu-memory-utilization 0.9  \
  --kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
  "kv_buffer_device": "npu",
  "kv_role": "kv_consumer",
  "kv_parallel_size": 1,
  "kv_port": "20001",
  "engine_id": "0",
  "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }'

proxy

python load_balance_proxy_server_example.py --host 0.0.0.0  --port 1025 --prefiller-hosts 172.22.0.218 --prefiller-port 20002 --decoder-hosts 172.22.0.155 --decoder-ports 20002

curl http://localhost:1025/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B",
        "prompt": "tell me how to sleep well",
        "max_tokens": 100,
        "temperature": 0
    }'

{"id":"cmpl-2","object":"text_completion","created":1754539184,"model":"/home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B","choices":[{"index":0,"text":"\nImproving sleep quality involves a combination of good sleep hygiene, lifestyle adjustments, and creating a conducive environment. Here's a comprehensive guide to help you sleep better:\n\n---\n\n### **1. Stick to a Consistent Sleep Schedule**\n- **Go to bed and wake up at the same time every day**, even on weekends. This helps regulate your body's internal clock (circadian rhythm).\n- Aim for **7–9 hours of sleep** per night (varies by age and individual needs).\n\n","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

note: this is not the fault of disaggregate pd, it is the qwen moe's issue

error detail

#!/bin/sh

export HCCL_IF_IP=172.22.0.155
export GLOO_SOCKET_IFNAME="enp23s0f3"
export TP_SOCKET_IFNAME="enp23s0f3"
export HCCL_SOCKET_IFNAME="enp23s0f3"
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/ranktable.json
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024

vllm serve /home/cache/modelscope/hub/models/Qwen/Qwen3-235B-A22B \
  --host 0.0.0.0 \
  --port 20002 \
  --data-parallel-size 4 \
  --api-server-count 2 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --seed 1024 \
  --max-model-len 8192  \
  --max-num-seqs 125 \
  --trust-remote-code \
  --gpu-memory-utilization 0.9

error

(EngineCore_3 pid=168013) [ERROR] 2025-08-07-03:03:47 (PID:169090, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
(EngineCore_3 pid=168013) EL0006: [PID: 169090] 2025-08-07-03:03:44.021.097 The resources are insufficient.
(EngineCore_3 pid=168013)         Solution: Close applications not in use.
(EngineCore_3 pid=168013)         TraceBack (most recent call last):
(EngineCore_3 pid=168013)         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_3 pid=168013)         [SqCqManage]Alloc sq cq fail, stream_id=1984, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_3 pid=168013)         New capture stream failed, stream_id=16, event_id=[-1->12387].[FUNC:GetCaptureStream][FILE:capture_model_utils.cc][LINE:92]
(EngineCore_3 pid=168013)         Create capture event failed, stream_id=16, event_id=-1, error=117571619.[FUNC:CaptureEventWait][FILE:api_impl.cc][LINE:1439]
(EngineCore_3 pid=168013)         Capture wait event failed.[FUNC:StreamWaitEvent][FILE:api_impl.cc][LINE:1493]
(EngineCore_3 pid=168013)         rtStreamWaitEvent execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
(EngineCore_3 pid=168013)         stream wait event failed, runtime result = 207005[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
(EngineCore_3 pid=168013)         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:78]
(EngineCore_3 pid=168013)         [SqCqManage]Alloc sq cq fail, stream_id=1985, retCode=0x7020023.[FUNC:Setup][FILE:stream.cc][LINE:655]
(EngineCore_3 pid=168013)         stream setup failed, retCode=0x7020023.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:6109]
(EngineCore_3 pid=168013)         Sync get device msg failed, retCode=0x7020023.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6159]
(EngineCore_3 pid=168013)         rtGetDevMsg execute failed, reason=[driver error:resource alloc fail][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

[liuchenbing]

✅Lora : After torch.ops._C.bgmv_expand and torch.ops._C.sgmv_shrink fusion operator are enabled, the performance (output token throughput) of the Qwen2.5-7B model is improved by 60% using the Lora feature based on vllm-ascend. (Input: 128; Output: 100; Concurrent: 40; Request Rate: 5.1)

command:

export VLLM_LOG_LEVEL=ERROR
export VLLM_LOG_TO_STDOUT=1
export VLLM_CONFIGURE_LOGGING=0
vllm serve  xxx/Qwen2.5-7B-Instruct --trust-remote-code --port=8099 --served-model-name=qwen --enable-prefix-caching --enforce-eager --tensor-parallel-size=1 --host=0.0.0.0 --enable-lora --max-lora-rank=16 --max-loras=32 --max-cpu-loras=32 --lora-modules lora1= xxx

performance

python3 benchmark_serving.py --backend openai-chat --dataset-name random --random-input-len 128 --random-output-len 100 --num-prompts 120 --ignore-eos --tokenizer xxx/Qwen2.5-7B-Instruct --host xxx --port xxx --endpoint /v1/chat/completions --max-concurrency 40 --percentile-metrics "ttft,tpot,e2el,itl" --metric-percentiles "10,90,95,99" --model lora1 --request-rate 5.1

Before：

After：

[Potabk]

✅ Disaggregate prefill on single-node

bash gen_ranktable.sh --ips 172.22.0.218  --network-card-name enp23s0f3 --prefill-device-cnt 1 --decode-device-cnt 1

prefill

ASCEND_RT_VISIBLE_DEVICES=0 VLLM_LLMDD_RPC_PORT=5559 vllm serve /home/cache/modelscope/hub/models/Qwen/Qwen3-8B \
  --port 8001 \
  --seed 1024 \
  --enforce-eager \
  --gpu-memory-utilization 0.8 \
  --kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
  "kv_buffer_device": "npu",
  "kv_role": "kv_producer",
  "kv_parallel_size": 1,
  "kv_port": "20001",
  "engine_id": "0",
  "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }'

decode

ASCEND_RT_VISIBLE_DEVICES=1 VLLM_LLMDD_RPC_PORT=6000 vllm serve /home/cache/modelscope/hub/models/Qwen/Qwen3-8B \
  --port 8000 \
  --seed 1024 \
  --gpu-memory-utilization 0.8 \
  --kv-transfer-config  \
  '{"kv_connector": "LLMDataDistCMgrConnector",
  "kv_buffer_device": "npu",
  "kv_role": "kv_consumer",
  "kv_parallel_size": 1,
  "kv_port": "20001",
  "engine_id": "0",
  "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
  }'

proxy

python load_balance_proxy_server_example.py --port 8192 --prefiller-ports 8001  --decoder-ports 8000

test

curl http://localhost:8192/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/home/cache/modelscope/hub/models/Qwen/Qwen3-8B",
        "prompt": "tell me how to sleep well",
        "max_tokens": 100,
        "temperature": 0
    }'

{"id":"cmpl-1","object":"text_completion","created":1754549607,"model":"/home/cache/modelscope/hub/models/Qwen/Qwen3-8B","choices":[{"index":0,"text":" at night\n\nSleeping well is essential for your physical and mental health. Here are some practical tips to help you improve your sleep quality:\n\n### 1. **Stick to a Sleep Schedule**\n   - Go to bed and wake up at the same time every day, even on weekends.\n   - This helps regulate your body's internal clock (circadian rhythm).\n\n### 2. **Create a Relaxing Bedtime Routine**\n   - Wind down with calming activities 1–2 hours before bed","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":6,"total_tokens":106,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}