[RFC]: Support Full Graph with multiple attention kernels

[yiz-liu]

Motivation

Compared to piecewise graph capture, the “full graph” approach offers three primary advantages:

• Reduced dispatch latency: By replaying the entire model execution graph at once, we cut overhead compared with multiple smaller replays.
• Stabilized multi-device performance: Captureing the whole model as one static graph also mitigates the dispatch fluctuations across devices.
• Stream/resource savings: Consolidating graph captures frees up streams, allowing more graphs to be captured.

Proposed Change

Our implementation will differ slightly from vLLM’s native mechanism for two main reasons:

1. Divergent Attention Operator Paths
   In vLLM‑Ascend, the attention operator path is not uniform and is selected at inference time based on the AttentionState. To accommodate this, we will hoist the control flow logic outside the inference graph. We will pre‑compile distinct graphs for each possible attention state, then dispatch the appropriate graph at each step.

2. Dynamic Attention Parameter Updates
   Parameters such as seq_lens and block_table must be updated at every decoding step to ensure correct tiling and maintain numerical accuracy. We will leverage graph_task_update to asynchronously re‑issue updated parameters for the attention operator on a separate stream, thereby minimizing compute bubbles caused by synchronous dispatch.

Implementation Plan

1. Decode‑Stage Full Graph
   Build the foundational framework and implement full‑graph capture for the DecodeOnly stage—where host dispatch latency is most critical. (See PRs [1/N][Feat] Implement primal full graph with limited scenario #1503 and [0.9.1][2/N][Feat] Restore paged attention kernel in Full Graph for performence #1677.)

2. Prefill‑Stage Full Graph
   Extend the graph‑dispatching framework to support the Prefill stage, and refactor the implementation into its long‑term location within the codebase.

3. Engineering Tasks
   Address additional concerns such as memory profiling, stream management, batch sizes adjusting, and integration testing.

Feedback Period.

2 weeks.

CC List.

@Yikun @wangxiyuan

Any Other Things.

No response

[yiz-liu]

I anticipate that adopting this RFC will significantly alleviate the problem described in vLLM‑Ascend issue #1168 (#1168 (comment)) since more graphs can be captured.

[xueliangyang-oeuler]

Are there any issues about Prefill‑Stage Full Graph？

[frankie-ys]

Could you provide me with the relevant startup command? I encountered a startup error during startup.VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 /usr/local/python3.11.13/bin/vllm serve /data/qwen3-32B --port 8080 --max-model-len 80000 -tp 2 --enable-prefix-caching --enable-chunked-prefill --served-model-name qwen3-32b-910b --complitation_config '{"full_cuda_graph": True}' . The error content is not recognized param of complitation_config.

[yiz-liu]

Due to a massive overhual of vLLM, full graph feature is temporarily broken on main branch, currently we do not have time to fix this, we will look into this ASAP.

[frankie-ys]

Could you provide me with the relevant startup command? I encountered a startup error during startup.VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 /usr/local/python3.11.13/bin/vllm serve /data/qwen3-32B --port 8080 --max-model-len 80000 -tp 2 --enable-prefix-caching --enable-chunked-prefill --served-model-name qwen3-32b-910b --complitation_config '{"full_cuda_graph": True}' . The error content is not recognized param of complitation_config.你能提供相关的启动命令吗？我在启动时遇到了启动错误。 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 /usr/local/python3.11.13/bin/vllm serve /data/qwen3-32B --port 8080 --max-model-len 80000 -tp 2 --enable-prefix-caching --enable-chunked-prefill --served-model-name qwen3-32b-910b --complitation_config '{"full_cuda_graph": True}' 。错误内容是未识别的参数 param of complitation_config。

When I set --complitation_config '{"full_cuda_graph": True}' to --complitation_config '{"full_cuda_graph": 1}', it can be work.

[yiz-liu]

Greetings!

I finally have time to get back to this outstanding issue and bring you up to speed on #2128.

Two major changes since v0.9.1:

1. graph_dispatcher now decides the graph mode instead of hard-coding it in the backend — this decouples Full Graph and Piecewise Graph and could make it possible to remove dynamo.
2. Updated to the new attn_group logic, but there’s a small hack left in update_graph_params; multi-attention models may not be fully supported yet (Mamba and Qwen3-Next for example).

For MLA models, they should be ready by the end of this month.

And as for FULL_AND_PIECEWISE mode, I'm not sure if this is a top-priority task.

[yiz-liu]

A quick follow up:

1. MLA models is supported now with [Feat][Graph]Support FULL_DECEDE_ONLY mode for MLA models #3125
2. MTP + FULL_DECODE_ONLY is under developement with [Feat]Make full graph mode compalible with MTP #3276