Skip to content

Conversation

@sarckk
Copy link
Collaborator

@sarckk sarckk commented Nov 7, 2025

Purpose

#26969 added usage of AITER gemm_a16w16 for GEMM ops on AMD for valid shapes. However, it passes a bias term to gemm_a16w16 that was only added to AITER in ROCm/aiter#1148 (merged Oct 28).

Some are still using an older version of AITER. For backward compatibility, we should have an option to disable using this AITER triton gemm kernels.

Test Plan

VLLM_ROCM_USE_AITER_TRITON_GEMM=0 VLLM_ROCM_USE_AITER=1  vllm serve Qwen/Qwen3-8B

Test Result

Checked that VLLM_ROCM_USE_AITER_TRITON_GEMM=0 disables it and by default on.


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@sarckk sarckk force-pushed the aiter-triton-gemm branch from 20ac3b0 to c87094f Compare November 7, 2025 19:55
@mergify mergify bot added the rocm Related to AMD ROCm label Nov 7, 2025
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an environment variable VLLM_ROCM_USE_AITER_TRITON_GEMM to control the usage of AITER's Triton GEMM kernel, which is a good addition for backward compatibility. The implementation is straightforward. However, there is a critical issue: the new environment variable, which affects the computation graph, has not been added to the compute_hash function. This can lead to incorrect caching behavior. I've added a comment with details on how to fix this.

@sarckk
Copy link
Collaborator Author

sarckk commented Nov 7, 2025

cc: @gshtras @maleksan85

@sarckk sarckk requested review from gshtras and houseroad November 8, 2025 22:52
@22quinn 22quinn enabled auto-merge (squash) November 9, 2025 00:44
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 9, 2025
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
auto-merge was automatically disabled November 9, 2025 03:02

Head branch was pushed to by a user without write access

@sarckk sarckk force-pushed the aiter-triton-gemm branch from 668c0c6 to b5419b2 Compare November 9, 2025 03:02
@22quinn 22quinn merged commit de2b783 into vllm-project:main Nov 9, 2025
49 checks passed
@tjtanaa
Copy link
Collaborator

tjtanaa commented Nov 9, 2025

@sarckk May I know if you are evaluating against the ROCm aiter version that is specified in the Dockerfile.rocm_base? The aiter code that are merged into main are supposed to be based off that version. I will double check on my end if the gemm_a16w16 is suppored in the ROCm aiter defined in Dockerfile.rocm_base.

@tjtanaa
Copy link
Collaborator

tjtanaa commented Nov 9, 2025

@sarckk It seems that the API is valid for Dockerfile.rocm_base. Thank you for surfacing there are situation where we need to support backwards compatibility. We will find a way to go forward.
We will try to avoid introduce new flags as it is causing confusion for users as there are too many flags to tuned.

@sarckk
Copy link
Collaborator Author

sarckk commented Nov 10, 2025

@tjtanaa yea in general it would be good to have some option to easily toggle features that rely on the latest AITER version, thanks

@sarckk sarckk deleted the aiter-triton-gemm branch November 10, 2025 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants