Load tuned fused_moe_lora shrink and expand kernel configs separately… #21

yugong333 · 2025-10-21T15:05:19Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Loading the kernel configs of fused_moe_lora kernel
Separate shrink and expand kernel configs in the fused_moe_lora function
Adding num_stages and num_warps parameters in the configs

Note: Based on PR vllm-project#21229 and vllm-project#26319

Test Plan

Test Result

Together with vllm-project#26319 we can improve the OTPS 80% - 90% in GPT-OSS-120B when concurrency is 1 or 2.

(Optional) Documentation Update

Signed-off-by: Fardin Hoque <kfhfar@amazon.com>

…lm-project#26728) Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>

…dels (vllm-project#26526) Signed-off-by: Max de Bayser <mbayser@br.ibm.com> Co-authored-by: Ayush Singh <ayush1009208@gmail.com>

…24354) Signed-off-by: Lu Fang <fanglu@fb.com>

…t#26732) Signed-off-by: mgoin <mgoin64@gmail.com>

…cifying compile sizes (vllm-project#26681) Signed-off-by: angelayi <yiangela7@gmail.com>

…NSE (vllm-project#26742) Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>

Signed-off-by: youkaichao <youkaichao@gmail.com>

Signed-off-by: bk-201 <joy25810@foxmail.com>

Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com> Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

…-project#24024) Signed-off-by: n1ck-guo <heng.guo@intel.com> Signed-off-by: Heng Guo <heng.guo@intel.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Signed-off-by: bk-201 <joy25810@foxmail.com>

Signed-off-by: Danielle Robinson <dmmaddix@amazon.com>

…t#26602) Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

Update to default_act_function and pass as callable

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

…m-project#26723) Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

…d. (alternative PR) (vllm-project#26718) Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>

…#26758) Signed-off-by: Ryan Li <ryanli@ryanli.org>

…-project#26684) Signed-off-by: wangyafeng <wangyafeng@baidu.com>

…26750) Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>

… ` (vllm-project#20983) Signed-off-by: Max Wittig <max.wittig@siemens.com> Signed-off-by: Antoine Auger <antoineauger@users.noreply.github.com> Co-authored-by: Antoine Auger <antoineauger@users.noreply.github.com>

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…llm-project#27085) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com> Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

…m-project#27169) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: Chen Wu <cntryroa@gmail.com>

…for gpt-oss

Signed-off-by: Chen Wu <cntryroa@gmail.com>

enable passing activation func so act_wrapper in lora will be called …

…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <yvorott@gmail.com>

…ole` (vllm-project#27166) Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>

Signed-off-by: uyzhang <yi.zhang.4096@gmail.com> Signed-off-by: Yi Zhang <zhangyi970819@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Roger Wang <hey@rogerw.io>

Signed-off-by: Andy Lo <andy@mistral.ai> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>

Signed-off-by: Eugene Khvedchenia <ekhvedchenia@nvidia.com> Signed-off-by: Natan Bagrov <nbagrov@nvidia.com> Signed-off-by: Roger Wang <hey@rogerw.io> Co-authored-by: Natan Bagrov <nbagrov@nvidia.com> Co-authored-by: Roger Wang <hey@rogerw.io>

… H100 (FP8/BF16) (vllm-project#26268) Signed-off-by: Shivam <shivampr.dev@gmail.com>

…llm-project#27195) Signed-off-by: NickLucche <nlucches@redhat.com>

vllm-project#23812) Signed-off-by: n1ck-guo <heng.guo@intel.com> Signed-off-by: Heng Guo <heng.guo@intel.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

… from json files

github-actions · 2025-10-21T15:05:36Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

kfhfar and others added 30 commits October 13, 2025 23:08

Pruning kernel Core Tests (vllm-project#26727)

fa96fb9

Signed-off-by: Fardin Hoque <kfhfar@amazon.com>

[ResponseAPI] Further polish message serialization and unit tests (vl…

35bc22f

…lm-project#26728) Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>

Add tests for chunked prefill and prefix cache with causal pooling mo…

d8bebb0

…dels (vllm-project#26526) Signed-off-by: Max de Bayser <mbayser@br.ibm.com> Co-authored-by: Ayush Singh <ayush1009208@gmail.com>

[Misc][DP] support customized aggregated logger for dp (vllm-project#…

8317f72

…24354) Signed-off-by: Lu Fang <fanglu@fb.com>

[UX] Replace VLLM_ALL2ALL_BACKEND with --all2all-backend (vllm-projec…

3e051bd

…t#26732) Signed-off-by: mgoin <mgoin64@gmail.com>

[compile] Enable sequence parallelism for full cuda graph without spe…

b59dd19

…cifying compile sizes (vllm-project#26681) Signed-off-by: angelayi <yiangela7@gmail.com>

[Easy] Fix env type check errors from VLLM_DEBUG_LOG_API_SERVER_RESPO…

cfded80

…NSE (vllm-project#26742) Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>

Merge branch 'main' into fused_moe_lora

48b15f4

[build][torch.compile] upgrade depyf version (vllm-project#26702)

8a0af6a

Signed-off-by: youkaichao <youkaichao@gmail.com>

add unit testing

ed61b7a

Signed-off-by: bk-201 <joy25810@foxmail.com>

[torch.compile] Unwrap fused_marlin_moe custom op (vllm-project#26739)

8ae1692

Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com> Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

[Feature][Quantization] auto_round format add support for regex (vllm…

2935092

…-project#24024) Signed-off-by: n1ck-guo <heng.guo@intel.com> Signed-off-by: Heng Guo <heng.guo@intel.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Add test

8c3148f

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Adding test for gptoss

42c7abe

add test

355967e

Signed-off-by: bk-201 <joy25810@foxmail.com>

Update to default_act_function and pass as callable

1f1d5d5

remove None type

95fe27e

Update default_act function signature

e22782a

Signed-off-by: Danielle Robinson <dmmaddix@amazon.com>

Add support for the /rerank endpoint in vllm bench serve (vllm-projec…

fe3edb4

…t#26602) Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

Merge pull request #16 from dcmaddix/remove_torch_ops

a860253

Update to default_act_function and pass as callable

[Docs] Add a start tag to build.inc.md (vllm-project#26747)

2e36cdb

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

Fix lora tests failure in TPU CI due to the removal of LoRA bias (vll…

4497c8f

…m-project#26723) Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>

[CI] [ROCm] Automate CC list for ROCm related issue (vllm-project#26753)

4821ac1

Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

[ci] Adding the test-amd.yaml for test definitions for the AMD backen…

d3cc842

…d. (alternative PR) (vllm-project#26718) Signed-off-by: Alexei V. Ivanov <alexei.ivanov@amd.com>

scheduler.py: Update the name of the default scheduler. (vllm-project…

481545b

…#26758) Signed-off-by: Ryan Li <ryanli@ryanli.org>

[Model][Bugfix]fix ernie45 load failed due to ernie45 eplb code (vllm…

01ad27f

…-project#26684) Signed-off-by: wangyafeng <wangyafeng@baidu.com>

[CI/Build] Use 127.0.0.1 instead of localhost in utils (vllm-project#…

d32c611

…26750) Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>

[Plugin] Make plugin group clear (vllm-project#26757)

577d498

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

[Bugfix] Standardize merging multimodal embeddings (vllm-project#26771)

d2f816d

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 and others added 27 commits October 18, 2025 23:57

[Benchmark] Convenience script for multiple parameter combinations (v…

b3aba04

…llm-project#27085) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

output type conversion fix (vllm-project#27159)

221bf72

[Chore] Separate out vllm.utils.network_utils (vllm-project#27164)

7a6c8c3

Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com> Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>

Fix tes

449f41a

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Fix test

7af116d

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

[Misc] Move utils to avoid conflicts with stdlib, and move tests (vll…

d31f784

…m-project#27169) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

merge main into fused_moe_lora

e03c892

Signed-off-by: Chen Wu <cntryroa@gmail.com>

clean

528900f

Signed-off-by: Chen Wu <cntryroa@gmail.com>

enable passing activation func so act_wrapper in lora will be called …

0a29b72

…for gpt-oss

move default act to beginning of file

bb01436

clean

f0ba7dd

Signed-off-by: Chen Wu <cntryroa@gmail.com>

clean

07f7b7c

Signed-off-by: Chen Wu <cntryroa@gmail.com>

Merge pull request #20 from dcmaddix/test_memory_err

570df8c

enable passing activation func so act_wrapper in lora will be called …

[Bugfix] Fix error with penalties when speculative decoding and struc…

f6fdacd

…tural output are enabled (vllm-project#26586) Signed-off-by: southfreebird <yvorott@gmail.com>

Fix typo in ValueError message: use kv_role instead of `kv_disagg_r…

8a81d77

…ole` (vllm-project#27166) Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>

Merge branch 'main' into fused_moe_lora

4ac5bfd

[LoRA] LoRA cuda graph specialization (vllm-project#25914)

b63f214

Signed-off-by: Andy Lo <andy@mistral.ai> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>

[Kernel] Accelerate solve_tril with TMA (vllm-project#26746)

9fce7be

Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>

AArch64 CPU Docker pipeline (vllm-project#26931)

1c691f4

[Kernel][Model] Tune fused_moe Triton configs for Qwen3-30B A3/A3B on…

4d0f266

… H100 (FP8/BF16) (vllm-project#26268) Signed-off-by: Shivam <shivampr.dev@gmail.com>

[Bugfix][CI] Fix Distributed Tests (4 GPUs) async_sched+ray test (v…

f9e7ad5

…llm-project#27195) Signed-off-by: NickLucche <nlucches@redhat.com>

Merge branch 'main' into fused_moe_lora

5b7ae52

[Feature][Quantization] auto_round support for mixed bits quantization (

87778d5

vllm-project#23812) Signed-off-by: n1ck-guo <heng.guo@intel.com> Signed-off-by: Heng Guo <heng.guo@intel.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Merge branch 'main' into fused_moe_lora

6cba4c6

Load tuned fused_moe_lora shrink and expand kernel configs separately…

ee0ec49

… from json files

yugong333 changed the base branch from fused_moe_lora to main October 21, 2025 15:17

yugong333 closed this by deleting the head repository Oct 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Load tuned fused_moe_lora shrink and expand kernel configs separately… #21

Load tuned fused_moe_lora shrink and expand kernel configs separately… #21

Uh oh!

yugong333 commented Oct 21, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

116 participants

Load tuned fused_moe_lora shrink and expand kernel configs separately… #21

Load tuned fused_moe_lora shrink and expand kernel configs separately… #21

Uh oh!

Conversation

yugong333 commented Oct 21, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

github-actions bot commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

116 participants

yugong333 commented Oct 21, 2025 •

edited by github-actions bot

Loading