Skip to content

[Bugfix]: Assertion error when using FlashInfer backend #25933

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 5, 2025
Merged

[Bugfix]: Assertion error when using FlashInfer backend #25933

merged 2 commits into from
Oct 5, 2025

Conversation

@simondanielsson
Copy link
Contributor

@simondanielsson simondanielsson commented Sep 30, 2025 •
edited by github-actions bot
Loading

Purpose

Fixes #25928.

Test Plan

The following does not err any more:

OMP_NUM_THREADS=8 VLLM_USE_AITER_UNIFIED_ATTENTION=1 VLLM_ATTENTION_BACKEND=FLASHINFER VLLM_USE_FLASHINFER_MOE_FP8=1 vllm serve --async-scheduling --gpu-memory-utilization 0.8 --enable-auto-tool-choice --tool-call-parser hermes --model=Qwen/Qwen3-30B-A3B-Instruct-2507-FP8

and
with Qwen/Qwen3-4B-Instruct-2507-FP8.

Test Result

Runs successfully without error on a single L4 node.

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@simondanielsson
Update assertion to use boolean compare
17b81fb 
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
gemini-code-assist[bot]
gemini-code-assist bot reviewed Sep 30, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request correctly fixes an AssertionError that occurs when using the FlashInfer backend. The original assertion, assert self.block_quant is None, was incorrect as self.block_quant is a boolean value and can never be None. The change to assert not self.block_quant accurately reflects the requirement that block quantization should be disabled for this code path. The addition of a type hint for the block_quant attribute is also a good improvement for code clarity. The changes are sound and resolve the bug.

@jasl
Copy link
Contributor

jasl commented Sep 30, 2025

Thank you! I'm testing this

@simondanielsson
Copy link
Contributor Author

simondanielsson commented Sep 30, 2025

Works on my end on L4 with Qwen/Qwen3-4B-Instruct-2507-FP8 (don't have enough mem for 30B) - does it work for you @jasl?

@jasl
Copy link
Contributor

jasl commented Sep 30, 2025

Works on my end on L4 with Qwen/Qwen3-4B-Instruct-2507-FP8 (don't have enough mem for 30B) - does it work for you @jasl?

Sorry, I haven't finish the test, will continue later

@jasl
Copy link
Contributor

jasl commented Sep 30, 2025

Sorry for the late response. I can confirm that the fix works.
Thank you!

@simondanielsson
Copy link
Contributor Author

simondanielsson commented Oct 1, 2025

Great to hear, then this PR should be ready for review.

yewentao256
yewentao256 approved these changes Oct 2, 2025
View reviewed changes
Copy link
Member

@yewentao256 yewentao256 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the work!

@yewentao256 yewentao256 added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 2, 2025
@DarkLight1337 DarkLight1337 merged commit 432e1cb into vllm-project:main Oct 5, 2025
52 checks passed
tomeras91 pushed a commit to tomeras91/vllm that referenced this pull request Oct 6, 2025
@simondanielsson @DarkLight1337 @tomeras91
[Bugfix]: Assertion error when using FlashInfer backend (vllm-project…
2aa85d7 
…#25933)

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
karan pushed a commit to karan/vllm that referenced this pull request Oct 6, 2025
@simondanielsson @DarkLight1337 @karan
[Bugfix]: Assertion error when using FlashInfer backend (vllm-project…
6c43a9f 
…#25933)

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Signed-off-by: Karan Goel <3261985+karan@users.noreply.github.com>
southfreebird pushed a commit to southfreebird/vllm that referenced this pull request Oct 7, 2025
@simondanielsson @DarkLight1337 @southfreebird
[Bugfix]: Assertion error when using FlashInfer backend (vllm-project…
333f773 
…#25933)

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 10, 2025
@simondanielsson @DarkLight1337 @xuebwang-amd
[Bugfix]: Assertion error when using FlashInfer backend (vllm-project…
c73e34e 
…#25933)

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025
@simondanielsson @DarkLight1337 @lywa1998
[Bugfix]: Assertion error when using FlashInfer backend (vllm-project…
e2d14df 
…#25933)

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
alhridoy pushed a commit to alhridoy/vllm that referenced this pull request Oct 24, 2025
@simondanielsson @DarkLight1337 @alhridoy
[Bugfix]: Assertion error when using FlashInfer backend (vllm-project…
02bd3c8 
…#25933)

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 24, 2025
@simondanielsson @DarkLight1337 @xuebwang-amd
[Bugfix]: Assertion error when using FlashInfer backend (vllm-project…
0b75e66 
…#25933)

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
@micah-wil micah-wil mentioned this pull request Nov 3, 2025
Open
1 task
rtourgeman pushed a commit to rtourgeman/vllm that referenced this pull request Nov 10, 2025
@simondanielsson @DarkLight1337 @rtourgeman
[Bugfix]: Assertion error when using FlashInfer backend (vllm-project…
95112ac 
…#25933)

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yewentao256 yewentao256 yewentao256 approved these changes

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Assertion error when using FlashInfer backend

4 participants

@simondanielsson @jasl @yewentao256 @DarkLight1337