vLLM Ascend Roadmap Q3 2025

[wangxiyuan]

This is a living document! We are eager to know what do you want for vLLM Ascend in Q3 2025. Any feedback is welcome.

Welcome to join vLLM Ascend Weekly Meeting.

---

Release plan

Next release v0.10.3 (v0.10.3rc1) (about 2025.09.23): https://github.com/vllm-project/vllm-ascend/milestone/4

---

As a vital component of vLLM, the vLLM Ascend project is dedicated to providing an easy, fast, and cheap LLM serving for everyone on Ascend NPU, and to actively contribute to the enrichment of vLLM.

In 2025 Q2, we have focused on 4 themes: vLLM Ascend for Production, Performance Optimization, Key Features, Ecosystem Connect. In 2025 Q3, we will focus on: Default V1 Engine、Quality and Production ready、User / Developer Experience、Competitive for Key Workflow.

1. Default V1 Engine

• Stable plugin architecture for hardware platforms [RFC]: Clear and stable interface for platform in vLLM vllm#22082
• V1 Engine fully supports and cleanup V0 code path: [Feature]: Enable V1 by default and cleanup V0 code #1620
• Enable CustomOP register: [CustomOP][Refactor] Register CustomOP instead of overwrite forward_oot #1647
• V1 feature support enhancement
  • Enc-Dec models
  • V1 PP supports [V1][PP] Support pp with ray backend in V1 #1800
  • xPyD Disaggregate prefill for kv cache register style #950
  • V1 scheduler

2. Quality and Production ready

• Unit test coverage enhancement: [RFC]: Unit test coverage improvement #1298
  • Coverage report: https://app.codecov.io/gh/vllm-project/vllm-ascend
• E2e Test coverage
• Model Support vLLM Ascend Model Support Priority #1608
• Benchmark Test
• Accuracy Test
• Module refactor
  • model arch
  • ops
  • attention
  • torchair
  • quantization

3. User / Developer Experience

• Users doc: [RFC]: Doc enhancement #1248
• Developer Design doc: [RFC]: Doc enhancement #1248
• Distributions
• Perf Dashboard and Accuracy report
• Developer Experience
  • vLLM commit hash recording: Record vLLM commit in PR description #1623

4. Competitive for Key Workflow

• Large Scale Serving

  • EPLB
    • Dynamic EPLB: [RFC]: Dynamic Expert Load Balance with Zero-like-overhead vllm#22246
    • static EPLB: Add static EPLB #1116
  • Qwen series (Qwen3 / Qwen3 MoE) optimization [Perf] Optimize perf of Qwen3 #1245
  • Qwen series (Qwen3 MoE) optimization: [Bugfix] Support Qwen3-MOE on aclgraph mode #1381
  • Disaggregated Prefilling
    • LLMDataDist: Disaggregate prefill for kv cache register style #950
    • Mooncake: [P/D] Mooncake Connector for v1 distributed #1568
  • CP/SP [RFC]: Context Parallelism && Sequence Parallelism vllm#22693
  • AF Disaggregated [RFC]: ATTN-FFN Disaggregation for MoE Models vllm#22799

• RLHF

  • Performance improvements
  • Parallel support

• Graph mode

  • Support Full Graph with multiple attention kernels: [RFC]: Support Full Graph with multiple attention kernels #1649
  • Automatic Kernel Fusion via torch.fx.graph and graph rewriter for vLLM-Ascend: [RFC]: Automatic Kernel Fusion via torch.fx.graph and graph rewriter for vLLM-Ascend #2386

• Model

  • Qwen/DeepSeek/Qwen VL series
  • Gemma3
  • K2
  • New model support vLLM Ascend Model Support Priority #1608
    - New trending models like: minimax / hunyuan / ERNIE
  • Quantization support: w4a16/w4a8 for Dense model
  • Quantization support: w4a16/w4a8 for MoE model
  • Model format support: awq, gguf

• Others

  • Atlas 300I series experimental support and perf enhancement: [Performance] Disable JIT and nd2nz to improve performance for Altlas 300I series #1591
  • LoRA performance enhancement. Add Custom Kernels For LoRA Performance #1884

---

If any of the items you wanted is not on the roadmap, your suggestion and contribution is strongly welcomed! Please feel free to comment in this thread, open feature request, or create an RFC.

Historical Roadmap:

• vLLM Ascend Roadmap Q2 2025 #448
• vLLM Ascend Roadmap Q1 2025 #71

[huyz-git]

vllm-ascend/vllm_ascend/utils.py

Lines 50 to 55 in 0060886

	# NOTE: Currently, we can only capture 1920 graphs at most,
	# due to the limitation of ACL graph. This number is bounded by
	# the number of streams, which is 2048, we save 128 streams
	# as a buffer.
	# Maximum number of graphs that can be captured by ACL Graph
	MAX_CAPTURE_SIZE = 1920

Are there any plan to enlarge the number of graphs that can be captured? 1920 is somehow too small for large models.

For example, even a 40 layers 14B model cannot have all the default 67 batch sizes be captured (40*67=2680). For a 64 layers 32B model running on two NPUs, it can only capture 1920/64/2=15 batch sizes.

In my experience, on NPU the performance gap between eager mode and ACL graph mode is large, and if the gap between each pair of batch sizes be captured is large, then the performance of the batch sizes between them will be impacted a lot. For example, when I running the Qwen3-235B model on 8 NPUs, it can only capture 1920/94/3=6 batch sizes, and the engine choose these sizes: [512,408,304,192,88,1]. In this case, the gap between 1 and 88 is large. I can get around 29 token/s for single request, but only ~20 token/s total for a batch of 2 requests (10 token/s each) and ~97 token/s for a batch of 10 requests (~9.7 token/s each). Although this is still faster than eager mode, batch of 2 requests has less total throughput than single request is a significant performance degradation.

In comparison, if I manually specific [32, 24, 16, 8, 4, 1] as the sizes, although batch size 2 and 10 is not captured, since the gap between 1 and 4 and between 8 and 16 is not too large, I can still get ~45 token/s total for a batch of 2 requests (~22 token/s each) and ~140 token/s total for a batch of 10 requests (~14 token/s each). If I manually specific [16 10 8 4 2 1] as the sizes, I can get ~49 and ~160 token/s respectively.

Mode	Graph BS	Single request	Batch size 2	Batch size 10
Eager	-	~5	~9	~44
ACL Graph	[512,408,304,192,88,1] (2 and 10 not captured, and gap 1-88 is large)	~29	~20	~97
ACL Graph	[32, 24, 16, 8, 4, 1] (2 and 10 not captured, but gap 1-4 / 8-16 is small)	~29	~44	~140
ACL Graph	[16 10 8 4 2 1] (2 and 10 captured)	~29	~49	~160

[yiz-liu]

Are there any plan to enlarge the number of graphs that can be captured? 1920 is somehow too small for large models.

For example, even a 40 layers 14B model cannot have all the default 67 batch sizes be captured (40*67=2680). For a 64 layers 32B model running on two NPUs, it can only capture 1920/64/2=15 batch sizes.

In my experience, on NPU the performance gap between eager mode and ACL graph mode is large, and if the gap between each pair of batch sizes be captured is large, then the performance of the batch sizes between them will be impacted a lot. For example, when I running the Qwen3-235B model on 8 NPUs, it can only capture 1920/94/3=6 batch sizes, and the engine choose these sizes: [512,408,304,192,88,1]. In this case, the gap between 1 and 88 is large. I can get around 29 token/s for single request, but only ~20 token/s total for a batch of 2 requests (10 token/s each) and ~97 token/s for a batch of 10 requests (~9.7 token/s each). Although this is still faster than eager mode, batch of 2 requests has less total throughput than single request is a significant performance degradation.

In comparison, if I manually specific [32, 24, 16, 8, 4, 1] as the sizes, although batch size 2 and 10 is not captured, since the gap between 1 and 4 and between 8 and 16 is not too large, I can still get ~45 token/s total for a batch of 2 requests (~22 token/s each) and ~140 token/s total for a batch of 10 requests (~14 token/s each). If I manually specific [16 10 8 4 2 1] as the sizes, I can get ~49 and ~160 token/s respectively.

Mode Graph BS Single request Batch size 2 Batch size 10
Eager - ~5 ~9 ~44
ACL Graph [512,408,304,192,88,1] (2 and 10 not captured, and gap 1-88 is large) ~29 ~20 ~97
ACL Graph [32, 24, 16, 8, 4, 1] (2 and 10 not captured, but gap 1-4 / 8-16 is small) ~29 ~44 ~140
ACL Graph [16 10 8 4 2 1] (2 and 10 captured) ~29 ~49 ~160

Yes, we have noted this and are actively working to drive improvements on this matter. If we establish any concrete timelines or plans, we will share them here.

[huangfude]

能实现EPLB吗？

[erixyao]

I do hope that vllm-ascend can officially Ascend 300I Duo soon. My customers are quite eager to deploy it!
Thank you for considering it!

[jianzs]

能实现EPLB吗？

@huangfude Static EPLB is ready #1116 . Dynamic EPLB is in development. #1391

[hikariming]

能增加对最近的新模型：minimax hunyuan、百度的支持吗～

[huyz-git]

Yes, we have noted this and are actively working to drive improvements on this matter. If we establish any concrete timelines or plans, we will share them here.

May I ask another question: What is the difference between the ACL Graph used by vllm-ascend and the graph mode used by MindIE?

In MindIE, I can set the maxBatchSize parameter to whatever I want, and it seems that the model will always run in graph mode, and I have never seen such a performance gap between different batch sizes I requested. It seems that MindIE can handle all possible batch sizes with its graph mode, and is not capped by this 1920 limits.

[mjp9527]

May I ask about the progress of xPyD?

[shanying2017]

what't the plan for support gguf ?

[shen-shanshan]

Refactor for TorchAir: #2273.