[Release]: Release checklist for v0.9.1

[wangxiyuan]

Release Checklist

Release Version: v0.9.1
Release Branch: v0.9.1-dev
Release Date: 2025/08/30
Release Manager: @wangxiyuan

Prepare Release Note

• Create a new issue for release feedback [v0.9.1] FAQ / Feedback | 问题/反馈 #2643

• Write the release note PR. [Doc] add v0.9.1 release note #2646 （should backport to 0.9.1-dev branch then）

  • Update the feedback issue link in docs/source/faqs.md

  • Add release note to docs/source/user_guide/release_notes.md

  • Update version info in docs/source/community/versioning_policy.md

  • Update contributor info in docs/source/community/contributors.md

  • Update package version in docs/conf.py

PR need Merge

we should make sure all 0.9.1-dev related PR is done. https://github.com/vllm-project/vllm-ascend/pulls?page=1&q=is%3Apr+is%3Aopen+base%3Av0.9.1-dev

Merge/Close/Keep as it is(should leave the reason here)

TO be merge:

• [v0.9.1][Doc]Large Scale Ep doc update #2595
• [0.9.1][doc]Update doc for 0.9.1 #2648
• [0.9.1][Bugfix][Aclgraph] Fix qwen3-moe + aclgraph + tp #2647
• [v0.9.1][Bugfix] Fix guided decoding invalid backend #2645
• [v0.9.1][bugfix] disable the chunked prefill feature in Non-MLA LLMs #2659
• rm ptach run_engine_core in dense case #2665
• [v0.9.1] Add CN docs #2657

Fix in post1 release later:

• [0.9.1][Fixbug] Fix num_hidden_layers on Qwen2-Audio and Qwen2.5-Omni #2656
• [0.9.1][PromptLogprobs][V1] Support prompt logprobs to fix ceval accuracy in V1 #2654

Functional Test

we should make sure 0.9.1 is well tested. The core function is:

• PD disaggreagate with LLMDatadist @Potabk
• aclgraph @MengqingCao
• spec decode @wxsIcey Ngram, MTP works. Eagle doesn't work.
• guided decode @shen-shanshan
• quantization @22dimensions
• vlm @zhangxinyuehfad
• rlhf @leo-pony
• loar @wxsIcey
• ascend scheduler @wxsIcey

Doc Test

• Tutorial is updated.
• User Guide is updated.
• Developer Guide is updated.
• Check V1 env in 0.9.1-dev branch

Prepare Artifacts

• Docker image is ready.
• Wheel package is ready.

Release Step

• Release note PR is merged.
• Post the release on GitHub release page.
• Generate official doc page on https://app.readthedocs.org/dashboard/
• Wait for the wheel package to be available on https://pypi.org/project/vllm-ascend
• Wait for the docker image to be available on https://quay.io/ascend/vllm-ascend
• Upload 310p wheel to Github release page
• Broadcast the release news (By message, blog , etc)
• Close this issue

[Yikun]

Known issue should mention PP not work, please use DP instead of?

[1] related: #2027 #1557

[Yikun]

Test Note:
# vllm serve
export MODEL=Qwen/Qwen3-8B
VLLM_USE_V1=1 VLLM_USE_MODELSCOPE=true python3 -m vllm.entrypoints.openai.api_server --model $MODEL \
         --tensor-parallel-size 1 --swap-space 16 --disable-log-stats \
         --disable-log-requests  --load-format dummy

# Run benchmark
export MODEL=Qwen/Qwen3-8B
export VLLM_USE_MODELSCOPE=true
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip install -r /vllm-workspace/vllm-ascend/benchmarks/requirements-bench.txt
python3 /vllm-workspace/vllm/benchmarks/benchmark_serving.py --model $MODEL --dataset-name random \
         --random-input-len 200 --num-prompts 200 --request-rate 1 \
         --save-result --result-dir ./

---

❌ Found a perf regression (TTFT ↑ 200ms / TPOT ↑ 8ms): #2326

---

❌ Found 2nd perf regression (TTFT ↑ 10~15ms): #2502

[shen-shanshan]

Test Guided Decoding

pytest -sv tests/singlecard/test_guided_decoding.py

❌ Get errors:

FAILED tests/singlecard/test_guided_decoding.py::test_guided_json_completion[guidance:disable-any-whitespace] - NotImplementedError: VLLM_USE_V1=1 is not supported with --guided-decoding-backend=guidance:disable-any-whitespace.
FAILED tests/singlecard/test_guided_decoding.py::test_guided_regex[guidance:disable-any-whitespace] - NotImplementedError: VLLM_USE_V1=1 is not supported with --guided-decoding-backend=guidance:disable-any-whitespace.

Backend guidance:disable-any-whitespace is invalid, replace to guidance.

Fix PR: #2645.

Test Spec Decode

❌ Ngram:

Run:

vllm serve /shared/cache/modelscope/hub/models/LLM-Research/Meta-Llama-3.1-8B-Instruct \
--max-model-len 1024 \
--speculative-config '{"method": "ngram", "num_speculative_tokens": 3, "prompt_lookup_max": 5, "prompt_lookup_min": 3}' \
--gpu_memory_utilization 0.9 \
--trust-remote-code \
--enforce-eager \
-tp 2

Get errors:

logs

[rank1]:[E830 02:48:23.701330865 compiler_depend.ts:429]  setup failed!
Exception raised from OperationSetup at build/third_party/op-plugin/op_plugin/CMakeFiles/op_plugin_atb.dir/compiler_depend.ts:151 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0xb8 (0xffffaedcc908 in /home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffffaed7b404 in /home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: atb::OperationSetup(atb::VariantPack, atb::Operation*, atb::Context*) + 0xc8 (0xffff8fe8dc3c in /home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #3: <unknown function> + 0x8dce4 (0xffff8fe8dce4 in /home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #4: <unknown function> + 0x212d600 (0xffffa276d600 in /home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x825e54 (0xffffa0e65e54 in /home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x828084 (0xffffa0e68084 in /home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x8249fc (0xffffa0e649fc in /home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #8: <unknown function> + 0x4c9e4c (0xffffaee09e4c in /home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x8595c (0xffffb9c5595c in /lib/aarch64-linux-gnu/libc.so.6)
frame #10: <unknown function> + 0xeb7dc (0xffffb9cbb7dc in /lib/aarch64-linux-gnu/libc.so.6)

(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527] WorkerProc hit an exception.
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527] Traceback (most recent call last):
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm/vllm/v1/executor/multiproc_executor.py", line 522, in worker_busy_loop
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     output = func(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 224, in execute_model
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     output = self.model_runner.execute_model(scheduler_output)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return func(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1402, in execute_model
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     finished_recving) = (self._process_reqs(scheduler_output,
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1164, in _process_reqs
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     hidden_states = self.model(
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm/vllm/model_executor/models/llama.py", line 581, in forward
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm/vllm/compilation/decorators.py", line 173, in __call__
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return self.forward(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm/vllm/model_executor/models/llama.py", line 392, in forward
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     hidden_states, residual = layer(positions, hidden_states, residual)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm/vllm/model_executor/models/llama.py", line 303, in forward
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     hidden_states, residual = self.input_layernorm(
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm/vllm/model_executor/custom_op.py", line 24, in forward
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return self._forward_method(*args, **kwargs)
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/github/vllm-v0.9.1/vllm-ascend/vllm_ascend/ops/layernorm.py", line 69, in forward_oot
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     x, _, residual = torch_npu.npu_add_rms_norm(x, residual, self.weight,
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]   File "/home/sss/miniconda3/envs/vllm-v0.9.1/lib/python3.10/site-packages/torch/_ops.py", line 1116, in __call__
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527]     return self._op(*args, **(kwargs or {}))
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is PagedAttentionOperation.
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527] Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527] Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527] [ERROR] 2025-08-30-02:48:23 (PID:31790, Device:1, RankID:-1) ERR00100 PTA call acl api failed.
(VllmWorker rank=1 pid=31790) ERROR 08-30 02:48:23 [multiproc_executor.py:527] 
ERROR 08-30 02:48:24 [multiproc_executor.py:140] Worker proc VllmWorker-0 died unexpectedly, shutting down executor.

❌ Eagle3:

Run:

vllm serve /shared/cache/modelscope/hub/models/LLM-Research/Meta-Llama-3.1-8B-Instruct \
--max-model-len 1024 \
--speculative-config '{"method": "eagle", "num_speculative_tokens": 2, "model": "/home/sss/models/models/models/vllm-ascend/EAGLE3-LLaMA3.1-Instruct-8B"}' \
--gpu_memory_utilization 0.9 \
--trust-remote-code \
--enforce-eager \
-tp 2

Get errors:

logs

(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492] WorkerProc failed to start.
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492] Traceback (most recent call last):
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492]   File "/home/sss/github/vllm-v0.9.1/vllm/vllm/v1/executor/multiproc_executor.py", line 466, in worker_main
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492]     worker = WorkerProc(*args, **kwargs)
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492]   File "/home/sss/github/vllm-v0.9.1/vllm/vllm/v1/executor/multiproc_executor.py", line 363, in __init__
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492]     self.worker.load_model()
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492]   File "/home/sss/github/vllm-v0.9.1/vllm-ascend/vllm_ascend/worker/worker_v1.py", line 238, in load_model
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492]     self.model_runner.load_model()
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492]   File "/home/sss/github/vllm-v0.9.1/vllm-ascend/vllm_ascend/worker/model_runner_v1.py", line 1905, in load_model
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492]     self.drafter.load_model()
(VllmWorker rank=0 pid=47212) ERROR 08-30 03:25:12 [multiproc_executor.py:492] TypeError: EagleProposer.load_model() missing 1 required positional argument: 'target_model'
EngineCore failed to start.

[Potabk]

PD disaggreagate with LLMDatadist test good

Take the example of four A3 machines running PD separation, use 2 nodes as prefill nodes, 2 nodes as decode nodes:
prefill * 2: dp2 tp8
decode * 1: dp32

generate ranktable

cd vllm-ascend/examples/disaggregate_prefill_v1/
bash gen_ranktable.sh --ips 172.22.0.218 172.22.0.155 172.22.0.188 172.22.0.212 \
  --npus-per-node  16 --network-card-name enp23s0f3 --prefill-device-cnt 32 --decode-device-cnt 32

run prefill

Run the following python scripts on different nodes

import multiprocessing
import os
import sys
dp_size = 2 # total number of DP engines for decode/prefill
dp_size_local = 2 # number of DP engines on the current node
dp_rank_start = 0 # starting DP rank for the current node
dp_ip = 172.22.0.218 # master node ip for DP communication
dp_port = 13395 # port used for DP communication
engine_port = 9000 # starting port for all DP groups on the current node
template_path = "./run_dp_template.sh"
if not os.path.exists(template_path):
  print(f"Template file {template_path} does not exist.")
  sys.exit(1)
def run_command(dp_rank_local, dp_rank, engine_port_):
  command = f"bash ./run_dp_template.sh {dp_size} {dp_ip} {dp_port} {dp_rank_local} {dp_rank} {engine_port_} {dp_size_local}"
  os.system(command)
processes = []
for i in range(dp_size_local):
  dp_rank = dp_rank_start + i
  dp_rank_local = i
  engine_port_ = engine_port + i
  process = multiprocessing.Process(target=run_command, args=(dp_rank_local, dp_rank, engine_port_))
  processes.append(process)
  process.start()
for process in processes:
  process.join()

prefill node1 corresponding template script

# run_dp_template.sh
#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="enp23s0f3"
local_ip="172.22.0.218"

# basic configuration for HCCL and connection
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export HCCL_BUFFSIZE=256
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH='/workspace/code/vllm-ascend/examples/disaggregate_prefill_v1/ranktable.json'

# obtain parameters from distributed DP server
export VLLM_DP_SIZE=$1
export VLLM_DP_MASTER_IP=$2
export VLLM_DP_MASTER_PORT=$3
export VLLM_DP_RANK_LOCAL=$4
export VLLM_DP_RANK=$5
export VLLM_DP_SIZE_LOCAL=$7

#pytorch_npu settings and vllm settings
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export VLLM_USE_V1=1

# enable the distributed DP server 
export VLLM_WORKER_MULTIPROC_METHOD="fork"
export VLLM_ASCEND_EXTERNAL_DP_LB_ENABLED=1

# The w8a8 weight can obtained from https://www.modelscope.cn/models/vllm-ascend/DeepSeek-R1-W8A8
# "--additional-config" is used to enable characteristics from vllm-ascend
vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
    --host 0.0.0.0 \
    --port $6 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name deepseek_v3 \
    --max-model-len 17000 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --max-num-seqs 4 \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
    --enforce-eager \
    --kv-transfer-config \
    '{"kv_connector": "LLMDataDistCMgrConnector",
      "kv_buffer_device": "npu",
      "kv_role": "kv_producer",
      "kv_parallel_size": "1",
      "kv_port": "20001",
      "engine_id": "0",
      "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
    }'
    --additional-config '{"ascend_scheduler_config":{"enabled":false}, "torchair_graph_config":{"enabled":false},"enable_weight_nz_layout":true,"enable_prefill_optimizations":true}'

prefill node2 corresponding template script

# run_dp_template.sh
#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="enp23s0f3"
local_ip="172.22.0.155"

# basic configuration for HCCL and connection
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export HCCL_BUFFSIZE=256
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH='/workspace/code/vllm-ascend/examples/disaggregate_prefill_v1/ranktable.json'

# obtain parameters from distributed DP server
export VLLM_DP_SIZE=$1
export VLLM_DP_MASTER_IP=$2
export VLLM_DP_MASTER_PORT=$3
export VLLM_DP_RANK_LOCAL=$4
export VLLM_DP_RANK=$5
export VLLM_DP_SIZE_LOCAL=$7

#pytorch_npu settings and vllm settings
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export VLLM_USE_V1=1

# enable the distributed DP server 
export VLLM_WORKER_MULTIPROC_METHOD="fork"
export VLLM_ASCEND_EXTERNAL_DP_LB_ENABLED=1

# The w8a8 weight can obtained from https://www.modelscope.cn/models/vllm-ascend/DeepSeek-R1-W8A8
# "--additional-config" is used to enable characteristics from vllm-ascend
vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
    --host 0.0.0.0 \
    --port $6 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name deepseek_v3 \
    --max-model-len 17000 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --max-num-seqs 4 \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
    --enforce-eager \
    --kv-transfer-config \
    '{"kv_connector": "LLMDataDistCMgrConnector",
      "kv_buffer_device": "npu",
      "kv_role": "kv_producer",
      "kv_parallel_size": "1",
      "kv_port": "20001",
      "engine_id": "0",
      "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
    }'
    --additional-config '{"ascend_scheduler_config":{"enabled":false}, "torchair_graph_config":{"enabled":false},"enable_weight_nz_layout":true,"enable_prefill_optimizations":true}'

run decode

Run the following python scripts on different nodes

import multiprocessing
import os
import sys
dp_size = 32 # total number of DP engines for decode/prefill
dp_size_local = 16 # number of DP engines on the current node
dp_rank_start = 0 # starting DP rank for the current node
dp_ip = 172.22.0.188 # master node ip for DP communication
dp_port = 13395 # port used for DP communication
engine_port = 9000 # starting port for all DP groups on the current node
template_path = "./run_dp_template.sh"
if not os.path.exists(template_path):
  print(f"Template file {template_path} does not exist.")
  sys.exit(1)
def run_command(dp_rank_local, dp_rank, engine_port_):
  command = f"bash ./run_dp_template.sh {dp_size} {dp_ip} {dp_port} {dp_rank_local} {dp_rank} {engine_port_} {dp_size_local}"
  os.system(command)
processes = []
for i in range(dp_size_local):
  dp_rank = dp_rank_start + i
  dp_rank_local = i
  engine_port_ = engine_port + i
  process = multiprocessing.Process(target=run_command, args=(dp_rank_local, dp_rank, engine_port_))
  processes.append(process)
  process.start()
for process in processes:
  process.join()

import multiprocessing
import os
import sys
dp_size = 32 # total number of DP engines for decode/prefill
dp_size_local = 16 # number of DP engines on the current node
dp_rank_start = 16 # starting DP rank for the current node
dp_ip = 172.22.0.188 # master node ip for DP communication
dp_port = 13395 # port used for DP communication
engine_port = 9000 # starting port for all DP groups on the current node
template_path = "./run_dp_template.sh"
if not os.path.exists(template_path):
  print(f"Template file {template_path} does not exist.")
  sys.exit(1)
def run_command(dp_rank_local, dp_rank, engine_port_):
  command = f"bash ./run_dp_template.sh {dp_size} {dp_ip} {dp_port} {dp_rank_local} {dp_rank} {engine_port_} {dp_size_local}"
  os.system(command)
processes = []
for i in range(dp_size_local):
  dp_rank = dp_rank_start + i
  dp_rank_local = i
  engine_port_ = engine_port + i
  process = multiprocessing.Process(target=run_command, args=(dp_rank_local, dp_rank, engine_port_))
  processes.append(process)
  process.start()
for process in processes:
  process.join()

decode node1 corresponding script

# run_dp_template.sh
#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="enp23s0f3"
local_ip="172.22.0.188"

# basic configuration for HCCL and connection
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export HCCL_BUFFSIZE=1024
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH='/workspace/code/vllm-ascend/examples/disaggregate_prefill_v1/ranktable.json'

# obtain parameters from distributed DP server
export VLLM_DP_SIZE=$1
export VLLM_DP_MASTER_IP=$2
export VLLM_DP_MASTER_PORT=$3
export VLLM_DP_RANK_LOCAL=$4
export VLLM_DP_RANK=$5
export VLLM_DP_SIZE_LOCAL=$7

#pytorch_npu settings and vllm settings
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export VLLM_USE_V1=1

# enable the distributed DP server 
export VLLM_WORKER_MULTIPROC_METHOD="fork"
export VLLM_ASCEND_EXTERNAL_DP_LB_ENABLED=1

# The w8a8 weight can obtained from https://www.modelscope.cn/models/vllm-ascend/DeepSeek-R1-W8A8
# "--additional-config" is used to enable characteristics from vllm-ascend
vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
    --host 0.0.0.0 \
    --port $6 \
    --tensor-parallel-size 1 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name deepseek_v3 \
    --max-model-len 17000 \
    --max-num-batched-tokens 256 \
    --trust-remote-code \
    --max-num-seqs 28 \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
    --kv-transfer-config \
        '{"kv_connector": "LLMDataDistCMgrConnector",
        "kv_buffer_device": "npu",
        "kv_role": "kv_consumer",
        "kv_parallel_size": "1",
        "kv_port": "20001",
        "engine_id": "0",
        "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
        }' \
    --additional-config '{"ascend_scheduler_config":{"enabled":false}, "torchair_graph_config":{"enabled":true,"enable_multistream_mla":true,"enable_multistream_moe":true,"graph_batch_sizes":[28], "enable_super_kernel":true, "use_cached_graph":true},"enable_weight_nz_layout":true}'

decode node2 corresponding script

# run_dp_template.sh
#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="enp23s0f3"
local_ip="172.22.0.212"

# basic configuration for HCCL and connection
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export HCCL_BUFFSIZE=1024
export DISAGGREGATED_PREFILL_RANK_TABLE_PATH='/workspace/code/vllm-ascend/examples/disaggregate_prefill_v1/ranktable.json'

# obtain parameters from distributed DP server
export VLLM_DP_SIZE=$1
export VLLM_DP_MASTER_IP=$2
export VLLM_DP_MASTER_PORT=$3
export VLLM_DP_RANK_LOCAL=$4
export VLLM_DP_RANK=$5
export VLLM_DP_SIZE_LOCAL=$7

#pytorch_npu settings and vllm settings
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export VLLM_USE_V1=1

# enable the distributed DP server 
export VLLM_WORKER_MULTIPROC_METHOD="fork"
export VLLM_ASCEND_EXTERNAL_DP_LB_ENABLED=1

# The w8a8 weight can obtained from https://www.modelscope.cn/models/vllm-ascend/DeepSeek-R1-W8A8
# "--additional-config" is used to enable characteristics from vllm-ascend
vllm serve /home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-V3-W8A8 \
    --host 0.0.0.0 \
    --port $6 \
    --tensor-parallel-size 1 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name deepseek_v3 \
    --max-model-len 17000 \
    --max-num-batched-tokens 256 \
    --trust-remote-code \
    --max-num-seqs 28 \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
    --kv-transfer-config \
        '{"kv_connector": "LLMDataDistCMgrConnector",
        "kv_buffer_device": "npu",
        "kv_role": "kv_consumer",
        "kv_parallel_size": "1",
        "kv_port": "20001",
        "engine_id": "0",
        "kv_connector_module_path": "vllm_ascend.distributed.llmdatadist_c_mgr_connector"
        }' \
    --additional-config '{"ascend_scheduler_config":{"enabled":false}, "torchair_graph_config":{"enabled":true,"enable_multistream_mla":true,"enable_multistream_moe":true,"graph_batch_sizes":[28], "enable_super_kernel":true, "use_cached_graph":true},"enable_weight_nz_layout":true}'

run proxy

python load_balance_proxy_server_example.py \
  --port 8000 \
  --host 0.0.0.0 \
  --prefiller-hosts \
    172.22.0.218 \
    172.22.0.155 \
  --prefiller-hosts-num \
    2 2 \
  --prefiller-ports \
    9000 9000 \
  --prefiller-ports-inc \
    2 2\
  --decoder-hosts \
    172.22.0.188 \
    172.22.0.212  \
  --decoder-hosts-num \
    16 16 \
  --decoder-ports  \
    9000 9000 \
  --decoder-ports-inc \
    16 16

curl http://0.0.0.0:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek_v3",
        "prompt": "tell me how to sleep well",
        "max_tokens": 100,
        "temperature": 0
    }'

{"id":"cmpl-c0e765b3-4dff-4882-aca3-de0ef1b3ebe1","object":"text_completion","created":1756526789,"model":"deepseek_v3","choices":[{"index":0,"text":" at night\n\nGetting a good night's sleep is essential for your overall health and well-being. Here are some tips to help you sleep well at night:\n\n### 1. **Establish a Consistent Sleep Schedule**\n   - Go to bed and wake up at the same time every day, even on weekends.\n   - This helps regulate your body's internal clock.\n\n### 2. **Create a Relaxing Bedtime Routine**\n   - Engage in calming activities before bed, such as reading, taking a","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":7,"total_tokens":107,"completion_tokens":100,"prompt_tokens_details":null},"kv_transfer_params":null}

[zhangxinyuehfad]

✅Test Qwen2.5 VL pad and unpad

pad:

V0:

VLLM_USE_MODELSCOPE=True vllm serve Qwen/Qwen2.5-VL-7B-Instruct --tensor_parallel_size 1  --max_model_len 8192 &

output:

root@57caeb7510d1:/workspace# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-d3df2bb9ada74541b97c353ced08f300","object":"chat.completion","created":1756522107,"model":"Qwen/Qwen2.5-VL-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"The future of Artificial Intelligence (AI) is highly anticipated and expected to bring significant changes across various sectors, including technology, healthcare, finance, transportation, and more. Here are some key trends and possibilities for the future of AI:\n\n1. **Enhanced Human-AI Collaboration**: AI will increasingly work in tandem with humans, enhancing productivity and decision-making processes. This collaboration could lead to more efficient workflows and better outcomes.\n\n2. **Advanced Machine Learning**: AI systems will become more sophisticated, capable of learning from vast amounts of data and making decisions based on complex patterns and insights that are not easily discernible by humans.\n\n3. **Ethical AI Development**: There will be a greater emphasis on developing AI systems that are transparent, accountable, and ethical. This includes ensuring that AI does not perpetuate biases or harm individuals or groups.\n\n4. **AI in Healthcare**: AI will play a crucial role in diagnosing diseases, personalizing treatment plans, and improving patient outcomes. It can also help in drug discovery and personalized medicine.\n\n5. **Autonomous Systems**: The development of fully autonomous vehicles, drones, and robots will continue to advance, potentially transforming industries such as logistics, manufacturing, and agriculture.\n\n6. **AI in Education**: AI will revolutionize education by providing personalized learning experiences, adaptive assessments, and intelligent tutoring systems.\n\n7. **AI in Finance**: AI will be used for risk management, fraud detection, and investment strategies, leading to more efficient and secure financial systems.\n\n8. **AI in Entertainment**: AI will enhance content creation, recommendation systems, and interactive experiences, making entertainment more engaging and tailored to individual preferences.\n\n9. **AI in Smart Cities**: AI will be integral to managing urban infrastructure, improving public services, and enhancing the quality of life for citizens through smart traffic management, energy efficiency, and waste reduction.\n\n10. **AI in Environmental Sustainability**: AI will help in monitoring environmental conditions, predicting natural disasters, and optimizing resource use, contributing to sustainable practices.\n\n11. **AI in Customer Service**: AI-powered chatbots and virtual assistants will continue to improve customer service, offering faster and more personalized support.\n\n12. **AI in Manufacturing**: AI will drive automation and optimization in manufacturing processes, leading to increased efficiency and reduced costs.\n\n13. **AI in Space Exploration**: AI will be essential for analyzing data from space missions, planning trajectories, and controlling spacecraft autonomously.\n\n14. **AI in Agriculture**: AI will help in precision farming, crop monitoring, and livestock management, improving agricultural productivity and sustainability.\n\n15. **AI in Legal and Compliance**: AI will assist in legal research, document analysis, and compliance monitoring, helping professionals stay updated with changing regulations and laws.\n\nThe future of AI is promising but also presents challenges related to job displacement, privacy concerns, and ethical considerations. As AI continues to evolve, it will be crucial to address these issues proactively to ensure that its benefits are widely accessible and that it is developed responsibly.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":24,"total_tokens":625,"completion_tokens":601,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

V1:

VLLM_USE_V1=1 VLLM_USE_MODELSCOPE=True vllm serve Qwen/Qwen2.5-VL-7B-Instruct --tensor_parallel_size 1  --max_model_len 8192 &

output

root@57caeb7510d1:/workspace# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-121d8ae67cb44f36bbc797cddf30bdf7","object":"chat.completion","created":1756522361,"model":"Qwen/Qwen2.5-VL-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"The future of Artificial Intelligence (AI) is highly anticipated and expected to be transformative across numerous sectors, including technology, healthcare, finance, transportation, and more. Here are some key trends and possibilities for the future of AI:\n\n1. **Enhanced Human-AI Collaboration**: AI will increasingly work in tandem with humans, enhancing productivity and decision-making capabilities. This could lead to more personalized experiences, better decision support systems, and improved efficiency in various industries.\n\n2. **Advanced Machine Learning and Deep Learning**: We can expect significant advancements in machine learning algorithms that will enable AI systems to learn from vast amounts of data more efficiently, leading to more sophisticated and adaptive AI applications.\n\n3. **Ethical AI Development**: There will be a greater emphasis on developing AI systems that are transparent, explainable, and ethical. This includes ensuring that AI systems respect privacy, avoid bias, and operate within ethical frameworks.\n\n4. **Autonomous Systems**: Autonomous vehicles, robots, and drones will become more prevalent, revolutionizing industries like manufacturing, logistics, and healthcare. These systems will be designed to make decisions based on complex data sets, improving accuracy and reducing errors.\n\n5. **AI in Healthcare**: AI will play a crucial role in diagnosing diseases, predicting patient outcomes, and optimizing treatments. AI will help in drug discovery, medical diagnostics, and personalized medicine.\n\n6. **AI in Education**: AI will be used to personalize education, adapt to individual learning styles, and assist teachers and students alike.\n\n7. **AI in Agriculture**: AI will optimize crop yields, manage resources more sustainably, and improve food security by automating tasks and improving agricultural practices.\n\n8. **AI in Personalized Medicine**: AI will aid in diagnosing diseases, predicting health risks, and improving patient care through predictive analytics and personalized treatment plans.\n\n9. **AI in Transportation and Logistics**: AI will optimize supply chains, reduce costs, and increase efficiency in logistics and transportation, potentially revolutionizing how we approach healthcare, agriculture, and other fields where human expertise and AI can complement each other, leading to more efficient resource allocation and management, leading to better health outcomes and sustainability.\n\n10. **AI in Smart Cities**: AI will help cities become smarter, managing resources, and improve urban planning and infrastructure management.\n\n11. **AI in Finance**: AI will transform financial services, fraud detection, and risk assessment.\n\n12. **AI in Entertainment**: AI will help in content creation, recommendation systems, and enhance user experience.\n\n13. **AI in Manufacturing**: AI will automate repetitive tasks, reduce waste, and improve safety standards.\n\n14. **AI in Entertainment**: AI will help in environmental sustainability, climate change, and sustainability.\n\n15. **AI in Entertainment**: AI will help in content creation, content curation, and content creation.\n\n16. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n17. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n18. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n19. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n16. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n17. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n18. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n19. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n20. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n21. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n22. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n23. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n24. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n25. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n26. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n22. **AI in Tourism**: AI will help in content creation, content creation, and content creation.\n\n23. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n24. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n25. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n26. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n27. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n28. **AI in Trade**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n26. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n27. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n28. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n26 **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n28. **AI in Trade**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n28. **AI in Tech**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Trade**: AI will help in content creation, content creation, and content creation.\n\n30. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n31. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n32. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n33. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n34. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n35. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n36. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n37. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n38. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n39. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n30. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n31. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n32. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n33. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n34. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n35. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n36. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n37. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n38. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n39. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n30. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n31. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n32. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n33. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n34. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n35. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n36. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n37. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n38. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n39. **AI in Travel**: AI will help in content creation, content creation.\n\n30. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n31. **AI in Travel**: AI will help in content creation, content creation.\n\n32. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n333. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n34. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n36. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n37. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n38. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n39. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n340. **AI in Travel**:","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":24,"total_tokens":2198,"completion_tokens":2174,"prompt_tokens_details":null},

unpad:

V0:

USE_OPTIMIZED_MODEL=0 VLLM_USE_MODELSCOPE=True vllm serve Qwen/Qwen2.5-VL-7B-Instruct --tensor_parallel_size 1  --max_model_len 8192 &

output:

root@57caeb7510d1:/workspace# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-dbc2e41822d84270af11d178109ab8b1","object":"chat.completion","created":1756534404,"model":"Qwen/Qwen2.5-VL-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"The future of Artificial Intelligence (AI) is highly anticipated and expected to bring significant changes across various sectors, including technology, healthcare, finance, transportation, and more. Here are some key trends and possibilities for the future of AI:\n\n1. **Enhanced Human-AI Collaboration**: AI will increasingly work in tandem with humans, enhancing productivity and decision-making processes. This collaboration could lead to more efficient workflows and better outcomes.\n\n2. **Advanced Machine Learning**: AI systems will become more sophisticated, capable of learning from vast amounts of data and making decisions based on complex patterns and insights that are not easily discernible by humans.\n\n3. **Ethical AI Development**: There will be a greater emphasis on developing AI systems that are transparent, accountable, and ethical. This includes ensuring that AI does not perpetuate biases or harm individuals or groups.\n\n4. **AI in Healthcare**: AI will play a crucial role in diagnosing diseases, personalizing treatment plans, and improving patient outcomes. It can also help in drug discovery and personalized medicine.\n\n5. **Autonomous Systems**: The development of fully autonomous vehicles, drones, and robots will continue to advance, potentially transforming industries such as logistics, manufacturing, and agriculture.\n\n6. **AI in Education**: AI will revolutionize education by providing personalized learning experiences, adaptive assessments, and intelligent tutoring systems.\n\n7. **AI in Finance**: AI will be used for risk management, fraud detection, and investment strategies, leading to more efficient and secure financial systems.\n\n8. **AI in Entertainment**: AI will enhance content creation, recommendation systems, and interactive experiences, making entertainment more engaging and tailored to individual preferences.\n\n9. **AI in Smart Cities**: AI will be integral to managing urban infrastructure, improving public services, and enhancing the quality of life for citizens through smart traffic management, energy efficiency, and waste reduction.\n\n10. **AI in Environmental Sustainability**: AI will help in monitoring environmental conditions, predicting natural disasters, and optimizing resource use, contributing to sustainable practices.\n\n11. **AI in Customer Service**: AI-powered chatbots and virtual assistants will continue to improve customer service, offering faster and more personalized support.\n\n12. **AI in Manufacturing**: AI will drive automation and optimization in manufacturing processes, leading to increased efficiency and reduced costs.\n\n13. **AI in Space Exploration**: AI will be essential for analyzing data from space missions, planning trajectories, and controlling spacecraft autonomously.\n\n14. **AI in Agriculture**: AI will help in precision farming, crop monitoring, and livestock management, improving agricultural productivity and sustainability.\n\n15. **AI in Legal and Compliance**: AI will assist in legal research, document analysis, and compliance monitoring, helping professionals stay updated with changing regulations and laws.\n\nThe future of AI is promising but also presents challenges related to job displacement, privacy concerns, and ethical considerations. As AI continues to evolve, it will be crucial to address these issues proactively to ensure that its benefits are widely accessible and that it is developed responsibly.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":24,"total_tokens":625,"completion_tokens":601,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

V1:

USE_OPTIMIZED_MODEL=0 VLLM_USE_V1=1 VLLM_USE_MODELSCOPE=True vllm serve Qwen/Qwen2.5-VL-7B-Instruct --tensor_parallel_size 1  --max_model_len 8192 &

output

{"id":"chatcmpl-b75faafb26064cc184b16d6960c6a8b5","object":"chat.completion","created":1756534939,"model":"Qwen/Qwen2.5-VL-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"The future of Artificial Intelligence (AI) is highly anticipated and expected to be transformative across numerous sectors, including technology, healthcare, finance, transportation, and more. Here are some key trends and possibilities for the future of AI:\n\n1. **Enhanced Human-AI Collaboration**: AI will increasingly work in tandem with humans, enhancing productivity and decision-making capabilities. This could lead to more personalized experiences, better decision support systems, and improved efficiency in various industries.\n\n2. **Advanced Machine Learning and Deep Learning**: We can expect significant advancements in machine learning algorithms that will enable AI systems to learn from vast amounts of data more efficiently, leading to more sophisticated and adaptive AI applications.\n\n3. **Ethical AI Development**: There will be a greater emphasis on developing AI systems that are transparent, explainable, and ethical. This includes ensuring that AI systems respect privacy, avoid bias, and operate within ethical frameworks.\n\n4. **Autonomous Systems**: Autonomous vehicles, robots, and drones will become more prevalent, revolutionizing industries like manufacturing, logistics, and healthcare. These systems will be designed to make decisions based on complex data sets, improving accuracy and reducing errors.\n\n5. **AI in Healthcare**: AI will play a crucial role in diagnosing diseases, predicting patient outcomes, and optimizing treatments. AI will help in drug discovery, medical diagnostics, and personalized medicine.\n\n6. **AI in Education**: AI will be used to personalize education, adapt to individual learning styles, and assist teachers and students alike.\n\n7. **AI in Agriculture**: AI will optimize crop yields, manage resources more sustainably, and improve food security by automating tasks and improving agricultural practices.\n\n8. **AI in Personalized Medicine**: AI will aid in diagnosing diseases, predicting health risks, and improving patient care through predictive analytics and personalized treatment plans.\n\n9. **AI in Transportation and Logistics**: AI will optimize supply chains, reduce costs, and increase efficiency in logistics and transportation, potentially revolutionizing how we approach healthcare, agriculture, and other fields where human expertise and AI can complement each other, leading to more efficient resource allocation and management, leading to better health outcomes and sustainability.\n\n10. **AI in Smart Cities**: AI will help cities become smarter, managing resources, and improve urban planning and infrastructure management.\n\n11. **AI in Finance**: AI will transform financial services, fraud detection, and risk assessment.\n\n12. **AI in Entertainment**: AI will help in content creation, recommendation systems, and enhance user experience.\n\n13. **AI in Manufacturing**: AI will automate repetitive tasks, reduce waste, and improve safety standards.\n\n14. **AI in Entertainment**: AI will help in environmental sustainability, climate change, and sustainability.\n\n15. **AI in Entertainment**: AI will help in content creation, content curation, and content creation.\n\n16. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n17. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n18. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n19. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n16. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n17. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n18. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n19. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n20. **AI in Entertainment**: AI will help in content creation, content creation, and content creation.\n\n21. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n22. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n23. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n24. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n25. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n26. **AI in Technology**: AI will help in content creation, content creation, and content creation.\n\n22. **AI in Tourism**: AI will help in content creation, content creation, and content creation.\n\n23. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n24. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n25. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n26. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n27. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n28. **AI in Trade**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n26. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n27. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n28. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n26 **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n28. **AI in Trade**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Transportation**: AI will help in content creation, content creation, and content creation.\n\n28. **AI in Tech**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n29. **AI in Trade**: AI will help in content creation, content creation, and content creation.\n\n30. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n31. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n32. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n33. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n34. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n35. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n36. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n37. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n38. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n39. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n30. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n31. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n32. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n33. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n34. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n35. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n36. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n37. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n38. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n39. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n30. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n31. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n32. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n33. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n34. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n35. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n36. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n37. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n38. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n39. **AI in Travel**: AI will help in content creation, content creation.\n\n30. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n31. **AI in Travel**: AI will help in content creation, content creation.\n\n32. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n333. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n34. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n36. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n37. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n38. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n39. **AI in Travel**: AI will help in content creation, content creation, and content creation.\n\n340. **AI in Travel**:","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":24,"total_tokens":2198,"completion_tokens":2174,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}r

✅Test Qwen2.5-Omni-7B pad and unpad (Fixed by #2656)

pad:

V0:

VLLM_USE_MODELSCOPE=True vllm serve Qwen/Qwen2.5-Omni-7B  &

output:

root@57caeb7510d1:~/.cache/modelscope/hub/models/Qwen# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-1ff4d725eac147ab96dd03addd53af7d","object":"chat.completion","created":1756541999,"model":"Qwen/Qwen2.5-Omni-7B","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"It is difficult to predict the exact future of AI with certainty, but it is certainly a rapidly evolving field with the potential to bring many new advancements and changes. Some possible ways that AI might continue to develop include:\n\n  1. Increased integration with other technologies: AI is likely to become more integrated with other technologies, such as robotics and the internet of things (IoT). This could result in the creation of more intelligent, interconnected machines that can work together to achieve a common goal.\n  2. Broader applications: AI is currently used in a wide range of industries, such as healthcare, finance, and transportation. As the technology continues to improve, it is likely to be applied to even more fields, including education, entertainment, and government.\n  3. Increased sophistication: AI systems are likely to become increasingly sophisticated, with the ability to learn and adapt to new situations more quickly and accurately.\n  4. Greater human-computer collaboration: As AI systems become more capable, they may be designed to work together with humans in a collaborative way, rather than simply replacing human workers.\n  5. Ethical and societal implications: The development of AI has the potential to bring many new challenges and ethical considerations, such as issues related to privacy, security, and bias. These are important areas that will need to be addressed as the technology continues to develop.\n\nOf course, many of these developments are still speculative, and the future of AI is likely to be shaped by a number of factors, including technological advancements, societal values, and regulatory developments.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":24,"total_tokens":339,"completion_tokens":315,"prompt_tokens_details":null}

V1:

VLLM_USE_V1=1 VLLM_USE_MODELSCOPE=True vllm serve Qwen/Qwen2.5-Omni-7B  &

output

root@57caeb7510d1:~/.cache/modelscope/hub/models/Qwen# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-483d0f6bd8cc49ffaef57a3ae587b669","object":"chat.completion","created":1756539935,"model":"Qwen/Qwen2.5-Omni-7B","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"It is difficult to predict the exact future of AI with certainty, but it is certainly a rapidly evolving field with the potential to bring many new advancements and changes. Some possible ways that AI might continue to develop include:\n\n  1. Increased integration with other technologies: AI is likely to become more integrated with other technologies, such as robotics and the internet of things (IoT). This could result in the creation of more intelligent, interconnected machines that can work with humans to solve complex problems and improve human life in many different areas.\n  2. Continual improvements in artificial intelligence: AI is likely to continue improving in terms of its capabilities and applications. With the amount of processing power and data available to researchers and developers, it is possible that we will see significant advancements in areas such as natural language processing, computer vision, and automation, among others.\n  3. increased autonomy: AI systems may become increasingly autonomous and self-governing, allowing for greater efficiency and human productivity.\n  4. New applications of AI: As research and funding continue to advance, more and more AI applications and systems will be developed and deployed. They may help to provide safety, efficiency, and even solve complex Problems that humans can't solve.\n  5. impacts on the workforce: AI could have a significant impact on the job market, as machines and robots perform tasks that were previously done by humans.\n  6. Enhance decision-making: AI is likely to become more common and used in fields such as healthcare and education, as well as manufacturing and military.\n  7. Ethical and policy policies: We can do more faster, more efficiently and with less oversight.\n\n  8. Awareness-based AI: AI will be applied to more industries, depending on the ethical and policy implications, such as safety, security and privacy.\n 9. ethics đảmй top Hispanic- andeer theories. The ethical implications of AI raising concerns over time, especially with automation and the general improvement of machines, resulting in society.\n\n 10. Risks: AI can also bring risks of AI-driven socio-pol Chíniine, automating dangerous tasks.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":24,"total_tokens":457,"completion_tokens":433,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}r

unpad:

V0:

USE_OPTIMIZED_MODEL=0 VLLM_USE_MODELSCOPE=True vllm serve Qwen/Qwen2.5-Omni-7B &

output:

root@57caeb7510d1:/vllm-workspace/vllm-ascend/vllm_ascend# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-8e781fdfed324f649850b614845cfa89","object":"chat.completion","created":1756545605,"model":"Qwen/Qwen2.5-Omni-7B","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"It is difficult to predict the exact future of AI with certainty, but it is certainly a rapidly evolving field with the potential to bring many new advancements and changes. Some possible ways that AI might continue to develop include:\n\n  1. Increased integration with other technologies: AI is likely to become more integrated with other technologies, such as robotics and the internet of things (IoT). This could result in the creation of more intelligent, interconnected machines that can work together to achieve a common goal.\n  2. Broader applications: AI is currently used in a wide range of industries, such as healthcare, finance, and transportation. As the technology continues to improve, it is likely to be applied to even more fields, including education, entertainment, and government.\n  3. Increased sophistication: AI systems are likely to become increasingly sophisticated, with the ability to learn and adapt to new situations more quickly and accurately.\n  4. Greater human-computer collaboration: As AI systems become more capable, they may be designed to work together with humans in a collaborative way, rather than simply replacing human workers.\n  5. Ethical and societal implications: The development of AI has the potential to bring many new challenges and ethical considerations, such as issues related to privacy, security, and bias. These are important areas that will need to be addressed as the technology continues to develop.\n\nOf course, many of these developments are still speculative, and the future of AI is likely to be shaped by a number of factors, including technological advancements, societal values, and regulatory developments.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":24,"total_tokens":339,"completion_tokens":315,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}r

V1:

USE_OPTIMIZED_MODEL=0 VLLM_USE_V1=1 VLLM_USE_MODELSCOPE=True vllm serve Qwen/Qwen2.5-Omni-7B &

output

root@57caeb7510d1:~/.cache/modelscope/hub/models/Qwen# curl -X POST "http://0.0.0.0:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{
  "messages": [
    {"role": "user", "content": "The future of AI is"}
  ]
}'
{"id":"chatcmpl-8df45c7b99654a679fccb20c42774390","object":"chat.completion","created":1756540621,"model":"Qwen/Qwen2.5-Omni-7B","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"It is difficult to predict the exact future of AI with certainty, but it is certainly a rapidly evolving field with the potential to bring many new advancements and changes. Some possible ways that AI might continue to develop include:\n\n  1. Increased integration with other technologies: AI is likely to become more integrated with other technologies, such as robotics and the internet of things (IoT). This could result in the creation of more intelligent, interconnected machines that can work with humans to solve complex problems and improve human life in many different areas.\n  2. Continual improvements in artificial intelligence: AI is likely to continue improving in terms of its capabilities and applications. With the amount of processing power and data available to researchers and developers, it is possible that we will see significant advancements in areas such as natural language processing, computer vision, and automation, among others.\n  3. increased autonomy: AI systems may become increasingly autonomous and self-governing, allowing for greater efficiency and human productivity.\n  4. New applications of AI: As research and funding continue to advance, more and more AI applications and systems will be developed and deployed. They may help to provide safety, efficiency, and even solve complex Problems that humans can't solve.\n  5. impacts on the workforce: AI could have a significant impact on the job market, as machines and robots perform tasks that were previously done by humans.\n  6. Enhance decision-making: AI is likely to become more common and used in fields such as healthcare and education, as well as manufacturing and military.\n  7. Ethical and policy policies: We can do more faster, more efficiently and with less oversight.\n\n  8. Awareness-based AI: AI will be applied to more industries, depending on the ethical and policy implications, such as safety, security and privacy.\n 9. ethics đảmй top Hispanic- andeer theories. The ethical implications of AI raising concerns over time, especially with automation and the general improvement of machines, resulting in society.\n\n 10. Risks: AI can also bring risks of AI-driven socio-pol Chíniine, automating dangerous tasks.","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":24,"total_tokens":457,"completion_tokens":433,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

[MengqingCao]

Test aclgraph on Qwen3-30B-A3B:

• only support tp after [0.9.1][Bugfix][Aclgraph] Fix qwen3-moe + aclgraph + tp #2647
• not support ep/dp/pp

[wxsIcey]

✅Test Ascend Scheduler (eager and aclgraph): passed

• chunked prefill + prefix cache
• prefix cache
• no chunked prefile, no prefix cache

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# Create an LLM.
llm = LLM(
            model="Qwen/Qwen2.5-7B-Instruct",
            max_model_len=1024,
            max_num_seqs=3,
            gpu_memory_utilization=0.9,
            additional_config={
                'ascend_scheduler_config': {
                    'enabled': True,
                    # 'enable_chunked_prefill': True,
                    'enable_prefix_caching': False,
                },
            }
        )

# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

✅Test Lora (eager and aclgraph): passed

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# Create an LLM.
llm = LLM(
            model="hmellor/Ilama-3.2-1B",
            gpu_memory_utilization=0.9,
            enable_lora=True,
            max_loras=4,
            max_model_len=1024,
            # enforce_eager=True,
            trust_remote_code=True,
            
        )

# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

[22dimensions]

Quantization test:

test script:

from vllm import LLM, SamplingParams
import os

os.environ["VLLM_USE_V1"] = "1"

prompts = [
    "Hello, my name is",
    "The future of AI is",
]

sampling_params = SamplingParams(temperature=0)

llm = LLM(model="/root/.cache/DeepSeek-V2-Lite-W8A8",
          max_model_len=2048,
          trust_remote_code=True,
          enforce_eager=True,
          tensor_parallel_size=1,
          quantization="ascend")
print("start infer")
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Qwen2.5-0.5B-Instruct-W8A8
test result:

start infer
Adding requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 139.13it/s]
Processed prompts: 100%|████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  3.69it/s, est. speed input: 18.47 toks/s, output: 59.10 toks/s]
Prompt: 'Hello, my name is', Generated text: ' John and I am a software developer. I have been working on a project for'
Prompt: 'The future of AI is', Generated text: " in the hands of the people who will build it. That's why we're"

Qwen3-8B-W4A8:
test result:

start infer
Adding requests: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 117.18it/s]
Processed prompts: 100%|████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  2.32it/s, est. speed input: 11.60 toks/s, output: 37.13 toks/s]
Prompt: 'Hello, my name is', Generated text: ' [Your Name], and I am the [Your Position] at [Company Name'
Prompt: 'The future of AI is', Generated text: ' a topic of much discussion, with some predicting it will revolutionize the world and'

DeepSeek-V2-Lite-W8A8:
test result:

start infer
Adding requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 105.88it/s]
Processed prompts:   0%|                                                                                | 0/2 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]/vllm-workspace/vllm-ascend/vllm_ascend/attention/mla_v1.py:860: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  seqlen=torch.tensor(
Processed prompts: 100%|████████████████████████████████████████████████████████████████████████| 2/2 [00:10<00:00,  5.44s/it, est. speed input: 1.10 toks/s, output: 2.94 toks/s]
Prompt: 'Hello, my name is', Generated text: '\nI am a 20 year old student from the UK. I am'
Prompt: 'The future of AI is', Generated text: ' in the hands of the people\nThe future of AI is in the hands of'

DeepSeek-R1-w4a8-pruning:
test script (eager mode):

from vllm import LLM, SamplingParams
import os

os.environ["VLLM_USE_V1"] = "1"
os.environ["VLLM_ASCEND_MLA_PA"]="1"

prompts = [
    "Hello, my name is",
    "The future of AI is",
]

sampling_params = SamplingParams(temperature=0, max_tokens=5)

llm = LLM(model="/home/cache/modelscope/hub/models/vllm-ascend/DeepSeek-R1-w4a8-pruning",
          max_model_len=2048,
                  dtype="bfloat16",
                  tensor_parallel_size=2,
                  quantization="ascend",
                  enforce_eager=True,
                  enable_expert_parallel=True,
                  additional_config={
                          "torchair_graph_config": {
                                  "enabled": False,
                          },
                          "ascend_scheduler_config": {
                                  "enabled": True,
                          }
                  })
print("start infer")
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

result:

start infer
Adding requests: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 103.10it/s]
Processed prompts:   0%|                                                                                | 0/2 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](VllmWorker rank=1 pid=87009) /workspace/code/vllm-ascend/vllm_ascend/attention/mla_v1.py:860: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
(VllmWorker rank=1 pid=87009)   seqlen=torch.tensor(
(VllmWorker rank=0 pid=87005) /workspace/code/vllm-ascend/vllm_ascend/attention/mla_v1.py:860: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
(VllmWorker rank=0 pid=87005)   seqlen=torch.tensor(
Processed prompts: 100%|████████████████████████████████████████████████████████████████████████| 2/2 [00:09<00:00,  4.86s/it, est. speed input: 1.24 toks/s, output: 1.03 toks/s]
Prompt: 'Hello, my name is', Generated text: '逸研究发现IPPudsimentary'
Prompt: 'The future of AI is', Generated text: '逸 Ban Corporealistically'

[leo-pony]

RLHF

Test Cases running Ok:

pytest tests/singlecard/test_camem.py
pytest tests/singlecard/test_pyhccl.py
pytest tests/multicard/test_pyhccl_distributed.py

Verl GRPO Trainning OK:
Running steps see in doc:
https://github.com/volcengine/verl/blob/main/docs/ascend_tutorial/ascend_quick_start.rst

(TaskRunner pid=110880) local_global_step_folder: checkpoints/verl_grpo_example_gsm8k/qwen2_7b_function_rm/global_step_58
(WorkerDict pid=114492) Downloading Model from https://www.modelscope.cn to directory: /shared/cache/modelscope/models/Qwen/Qwen2.5-0.5B-Instruct
(WorkerDict pid=114492) 2025-08-30 07:17:34,096 - modelscope - INFO - Target directory already exists, skipping creation.
(TaskRunner pid=110880) step:58 - global_seqlen/min:30283 - global_seqlen/max:32829 - global_seqlen/minmax_diff:2546 - global_seqlen/balanced_min:31835 - global_seqlen/balanced_max:31835 - global_seqlen/mean:31835.0 - actor/entropy:0.2552359104156494 - actor/kl_loss:0.043474738486111164 - actor/kl_coef:0.001 - actor/pg_loss:-0.007866104424465448 - actor/pg_clipfrac:0.001218357589095831 - actor/ppo_kl:-0.0004338123035267927 - actor/pg_clipfrac_lower:0.0 - actor/grad_norm:1.477298378944397 - perf/mfu/actor:0.029526387023333194 - perf/max_memory_allocated_gb:32.682217597961426 - perf/max_memory_reserved_gb:43.072265625 - perf/cpu_memory_used_gb:69.03737258911133 - actor/lr:5e-07 - val-core/openai/gsm8k/reward/mean@1:0.38362395754359363 - training/global_step:58 - training/epoch:0 - critic/score/mean:0.3984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.3984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.008147225715219975 - critic/advantages/max:1.7888504266738892 - critic/advantages/min:-1.7888504266738892 - critic/returns/mean:-0.008147225715219975 - critic/returns/max:1.7888504266738892 - critic/returns/min:-1.7888504266738892 - response_length/mean:93.203125 - response_length/max:128.0 - response_length/min:35.0 - response_length/clip_ratio:0.20468750596046448 - response_length_non_aborted/mean:93.203125 - response_length_non_aborted/max:128.0 - response_length_non_aborted/min:35.0 - response_length_non_aborted/clip_ratio:0.20468750596046448 - response/aborted_ratio:0.0 - prompt_length/mean:105.765625 - prompt_length/max:232.0 - prompt_length/min:66.0 - prompt_length/clip_ratio:0.0 - timing_s/start_profile:8.52709636092186e-05 - timing_s/generate_sequences:5.539670467376709 - timing_s/reshard:2.6130850315093994 - timing_s/generation_timing/max:5.569456577301025 - timing_s/generation_timing/min:5.508663177490234 - timing_s/generation_timing/topk_ratio:0.25 - timing_s/gen:9.28225199598819 - timing_s/reward:0.2662604786455631 - timing_s/old_log_prob:2.1357194492593408 - timing_s/ref:1.875957471318543 - timing_s/adv:0.033633374609053135 - timing_s/update_actor:11.775554296560585 - timing_s/step:25.392320052720606 - timing_s/testing:18.106994600966573 - timing_s/save_checkpoint:7.030189113691449 - timing_s/stop_profile:0.00014534126967191696 - timing_per_token_ms/gen:0.15561193622779868 - timing_per_token_ms/adv:0.0002641226213998204 - timing_per_token_ms/update_actor:0.09247333356809004 - timing_per_token_ms/ref:0.014731878995747943 - perf/total_num_tokens:127340 - perf/time_per_step:25.392320052720606 - perf/throughput:1253.7255333070325
(TaskRunner pid=110880) ("Final validation metrics: {'val-core/openai/gsm8k/reward/mean@1': "
(TaskRunner pid=110880)  '0.38362395754359363}')
Training Progress: 100%|██████████| 58/58 [27:47<00:00, 28.74s/it]

[Potabk]

PD disaggreagate perf

using ais_bench

1. install

git clone https://gitee.com/aisbench/benchmark.git
cd benchmark/
pip3 install -e ./

2. prepare dataset(gsm8k)

wget http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/gsm8k.zip
unzip gsm8k.zip -d benchmark/ais_bench/datasets/gsm8k/

3. config for vllm api
   You can change the configuration the dir :benchmark/ais_bench/benchmark/configs/models/vllm_api Take the vllm_api_stream_chat.py for examples

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChatStream,
        abbr='vllm-api-stream-chat',
        path="/root/.cache/ds_r1",
        model="dsr1",
        request_rate = 28,
        retry = 2,
        host_ip = "192.0.0.1", # Proxy service host IP
        host_port = 8000,  # Proxy service Port
        max_out_len = 10,
        batch_size=1536,
        trust_remote_code=True,
        generation_kwargs = dict(
            temperature = 0,
            seed = 1024,
            ignore_eos=False,
        )
    )
]

4. run perf

╒══════════════════════════╤═════════╤════════════════╤════════════════╤════════════════╤════════════════╤════════════════╤════════════════╤════════════════╤══════╕
│ Performance Parameters   │ Stage   │ Average        │ Min            │ Max            │ Median         │ P75            │ P90            │ P99            │  N   │
╞══════════════════════════╪═════════╪════════════════╪════════════════╪════════════════╪════════════════╪════════════════╪════════════════╪════════════════╪══════╡
│ E2EL                     │ total   │ 12044.1894 ms  │ 1033.005 ms    │ 22593.0985 ms  │ 11827.8399 ms  │ 17982.5084 ms  │ 21259.5598 ms  │ 22196.1517 ms  │ 1280 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼──────┤
│ TTFT                     │ total   │ 11795.5236 ms  │ 794.2698 ms    │ 22313.4215 ms  │ 11573.1799 ms  │ 17735.4871 ms  │ 21011.4372 ms  │ 21944.7155 ms  │ 1280 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼──────┤
│ TPOT                     │ total   │ 27.6295 ms     │ 0.0414 ms      │ 33.7424 ms     │ 27.8192 ms     │ 28.0268 ms     │ 28.5011 ms     │ 32.8085 ms     │ 1280 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼──────┤
│ ITL                      │ total   │ 35.3829 ms     │ 0.0083 ms      │ 71.2453 ms     │ 49.2118 ms     │ 50.5129 ms     │ 51.4635 ms     │ 56.3023 ms     │ 1280 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼──────┤
│ InputTokens              │ total   │ 62.9633        │ 26.0           │ 189.0          │ 59.0           │ 74.0           │ 93.0           │ 128.0          │ 1280 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼──────┤
│ OutputTokens             │ total   │ 10.0           │ 10.0           │ 10.0           │ 10.0           │ 10.0           │ 10.0           │ 10.0           │ 1280 │
├──────────────────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼──────┤
│ OutputTokenThroughput    │ total   │ 1.4391 token/s │ 0.4426 token/s │ 9.6805 token/s │ 0.8454 token/s │ 1.6172 token/s │ 3.3202 token/s │ 7.8689 token/s │ 1280 │
╘══════════════════════════╧═════════╧════════════════╧════════════════╧════════════════╧════════════════╧════════════════╧════════════════╧════════════════╧══════╛
╒══════════════════════════╤═════════╤═══════════════════╕
│ Common Metric            │ Stage   │ Value             │
╞══════════════════════════╪═════════╪═══════════════════╡
│ Benchmark Duration       │ total   │ 69521.7306 ms     │
├──────────────────────────┼─────────┼───────────────────┤
│ Total Requests           │ total   │ 1319              │
├──────────────────────────┼─────────┼───────────────────┤
│ Failed Requests          │ total   │ 39                │
├──────────────────────────┼─────────┼───────────────────┤
│ Success Requests         │ total   │ 1280              │
├──────────────────────────┼─────────┼───────────────────┤
│ Concurrency              │ total   │ 221.7517          │
├──────────────────────────┼─────────┼───────────────────┤
│ Max Concurrency          │ total   │ 1536              │
├──────────────────────────┼─────────┼───────────────────┤
│ Request Throughput       │ total   │ 18.4115 req/s     │
├──────────────────────────┼─────────┼───────────────────┤
│ Total Input Tokens       │ total   │ 80593             │
├──────────────────────────┼─────────┼───────────────────┤
│ Prefill Token Throughput │ total   │ 5.3379 token/s    │
├──────────────────────────┼─────────┼───────────────────┤
│ Total generated tokens   │ total   │ 12800             │
├──────────────────────────┼─────────┼───────────────────┤
│ Input Token Throughput   │ total   │ 1159.249 token/s  │
├──────────────────────────┼─────────┼───────────────────┤
│ Output Token Throughput  │ total   │ 184.1151 token/s  │
├──────────────────────────┼─────────┼───────────────────┤
│ Total Token Throughput   │ total   │ 1343.3641 token/s │

[wangxiyuan]

@shen-shanshan ngram test on my machine works well.

test code:

import gc

import torch

from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import (destroy_distributed_environment,
                                             destroy_model_parallel)

def clean_up():
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    torch.npu.empty_cache()

prompts = [
    "Hello, my name is",
    "The future of AI is",
]
if __name__ == "__main__":
    sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=40)
    llm = LLM(model="LLM-Research/Meta-Llama-3.1-8B-Instruct",
              speculative_config={"method": "ngram", "num_speculative_tokens": 3, "prompt_lookup_max": 5, "prompt_lookup_min": 3},
              trust_remote_code=True,
              enforce_eager=True,
              tensor_parallel_size=2,
              max_model_len=4096,
              )

    outputs = llm.generate(prompts, sampling_params)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

    del llm
    clean_up()

[Yikun]

vllm-ascend/docs/source/_templates/sections/header.html

Line 57 in 3584306

<p>You are viewing the latest developer preview docs. <a href="https://vllm-ascend.readthedocs.io/en/v0.7.3-dev">Click here</a> to view docs for the latest stable release(v0.7.3.post1).</p>

This should be updated.

[wangxiyuan]

@Yikun It's done in the release note PR.