[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) #17864

qli88 · 2025-05-08T16:24:24Z

Changes to adapt new AITER MoE API;
Changes to adapt new AITER MLA API (MTP is not enabled)
Some other bug fixes;
This PR is for AITER API changes introduced by commit 939f741fc37f46694e48c32c7164f49eae2584c4 (merged on 04/20/2025). AITER versions after this require this patch to work.

Signed-off-by: Qiang Li <qiang.li2@amd.com>

github-actions · 2025-05-08T16:24:35Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

hongxiayang · 2025-05-08T16:37:01Z

Hi, @qli88:
Regarding "latest AITER(05/07/2025":
Is there a change related to the base docker image for aiter that caused the incompatibility? should that aiter commit change be included in this PR as well?

hongxiayang · 2025-05-08T16:38:49Z

Any tests in this PR?

Signed-off-by: Qiang Li <qiang.li2@amd.com>

qli88 · 2025-05-08T16:46:46Z

Hi, @qli88: Regarding "latest AITER(05/07/2025": Is there a change related to the base docker image for aiter that caused the incompatibility? should that aiter commit change be included in this PR as well?

The incompatibility is caused by mainly 2 things: 1. AITER MoE interface changes; 2. AITER MLA interface changes. So I think this is irrelevant to base docker image.

qli88 · 2025-05-08T16:49:26Z

Any tests in this PR?

This change is for ROCm only and I have tested on MI300X host with DeepSeek V3.

hongxiayang · 2025-05-08T17:14:37Z

Hi, @qli88: Regarding "latest AITER(05/07/2025": Is there a change related to the base docker image for aiter that caused the incompatibility? should that aiter commit change be included in this PR as well?

The incompatibility is caused by mainly 2 things: 1. AITER MoE interface changes; 2. AITER MLA interface changes. So I think this is irrelevant to base docker image.

The reason I am asking about from which commit of aiter introduced those changes. If the customer currently only uses upstream to build from source to serve Deepseek model, does the current aiter commit (https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base#L15) in Dockerfile.rocm_base works with these changes?

qli88 · 2025-05-08T17:24:06Z

Hi, @qli88: Regarding "latest AITER(05/07/2025": Is there a change related to the base docker image for aiter that caused the incompatibility? should that aiter commit change be included in this PR as well?

The incompatibility is caused by mainly 2 things: 1. AITER MoE interface changes; 2. AITER MLA interface changes. So I think this is irrelevant to base docker image.

The reason I am asking about from which commit of aiter introduced those changes. If the customer currently only uses upstream to build from source to serve Deepseek model, does the current aiter commit (https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base#L15) in Dockerfile.rocm_base works with these changes?

Thanks for the context! I think the new Docker image that will be released today (or tomorrow) will use the AITER commit I mentioned above (commit c1debd87ce0391aa27438d9e07e76e4fea7c4b70).

mgoin

Looks reasonable to me but will wait for @SageMoore to sign off

hmellor · 2025-05-09T15:31:28Z

This PR introduces two undefined names qo_indptr and max_seqlen_qo to main:

vllm/vllm/attention/ops/rocm_aiter_mla.py

Lines 63 to 72 in 200da9a

    
           mla_decode_fwd(q, 
        
                          kv_buffer.view(-1, 1, 1, q.shape[-1]), 
        
                          o, 
        
                          qo_indptr, 
        
                          kv_indptr, 
        
                          kv_indices, 
        
                          kv_last_page_lens, 
        
                          max_seqlen_qo, 
        
                          sm_scale=sm_scale, 
        
                          logit_cap=logit_cap)

https://github.com/vllm-project/vllm/actions/runs/14931789556

mgoin · 2025-05-09T15:49:54Z

Darn, why did the precommit not show up in the checks?

…17864)" This reverts commit 9f64e93.

hmellor · 2025-05-09T16:17:26Z

Yeah it's all green here, perhaps it was just a race condition with another PR that didn't trigger merge conflicts?

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com> Signed-off-by: David Xia <david@davidxia.com>

Looks like vllm-project#17864 had an outdated branch. So its [merge commit][1] caused `qo_indptr` and `max_seqlen_qo` to go into the function signature of `aiter_mla_decode_fwd()` where they're not used and into the body of `mla_decode_fwd_impl()` where they aren't defined. This PR fixes the discrepancies and call-sites. [1]: vllm-project@9f64e93#diff-88fd09f50e8cfc77678ade87483ab9a89ce58904203578f8816882763bd577c2 Signed-off-by: David Xia <david@davidxia.com>

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com> Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com>

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

Compatible patch for latest AITER(05/07/2025)

608260a

Signed-off-by: Qiang Li <qiang.li2@amd.com>

hongxiayang added the rocm Related to AMD ROCm label May 8, 2025

yapf

853ed2b

Signed-off-by: Qiang Li <qiang.li2@amd.com>

mgoin reviewed May 8, 2025

View reviewed changes

mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label May 8, 2025

SageMoore approved these changes May 9, 2025

View reviewed changes

mgoin approved these changes May 9, 2025

View reviewed changes

mgoin merged commit 9f64e93 into vllm-project:main May 9, 2025
77 checks passed

gshtras mentioned this pull request May 9, 2025

Aiter mla cherrypick ROCm/vllm#543

Merged

mgoin added a commit that referenced this pull request May 9, 2025

Revert "[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) (#…

e9c18f1

…17864)" This reverts commit 9f64e93.

mgoin mentioned this pull request May 9, 2025

Revert "[BugFix][AMD] Compatible patch for latest AITER(05/07/2025)" #17910

Merged

hmellor mentioned this pull request May 9, 2025

Handle error when str passed to /v1/audio/transcriptions #17909

Merged

davidxia mentioned this pull request May 9, 2025

[Bugfix] add missing function params to rocm_aiter_mla.py #17911

Closed

davidxia pushed a commit to davidxia/vllm that referenced this pull request May 9, 2025

[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) (vllm-pro…

ad296c7

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com> Signed-off-by: David Xia <david@davidxia.com>

tjtanaa mentioned this pull request May 10, 2025

[Feature] [ROCm]: AITER Kernel Integration #14964

Open

61 tasks

princepride pushed a commit to princepride/vllm that referenced this pull request May 10, 2025

[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) (vllm-pro…

a4ba0e8

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com> Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>

RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025

[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) (vllm-pro…

fcecfcc

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com> Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

tjtanaa mentioned this pull request May 13, 2025

[BugFix][AMD] Compatible patch for AITER lib after 04/20 #17912

Merged

mawong-amd pushed a commit to ROCm/vllm that referenced this pull request May 14, 2025

[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) (vllm-pro…

cf5161a

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com>

zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025

[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) (vllm-pro…

99bf818

…ject#17864) Signed-off-by: Qiang Li <qiang.li2@amd.com> Signed-off-by: Yuqi Zhang <yuqizhang@google.com>

vllmellm mentioned this pull request Aug 26, 2025

[Feature] [ROCm]: AITER Kernel Integration vllmellm/vllm#51

Open

61 tasks

Uh oh!

[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) #17864

[BugFix][AMD] Compatible patch for latest AITER(05/07/2025) #17864

Uh oh!

Conversation

qli88 commented May 8, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 8, 2025

Uh oh!

hongxiayang commented May 8, 2025

Uh oh!

hongxiayang commented May 8, 2025

Uh oh!

qli88 commented May 8, 2025

Uh oh!

qli88 commented May 8, 2025

Uh oh!

hongxiayang commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qli88 commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgoin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hmellor commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgoin commented May 9, 2025

Uh oh!

hmellor commented May 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

qli88 commented May 8, 2025 •

edited by github-actions bot

Loading

hongxiayang commented May 8, 2025 •

edited

Loading

qli88 commented May 8, 2025 •

edited

Loading

hmellor commented May 9, 2025 •

edited

Loading