Skip to content

[BugFix] Fix cascade attention - RuntimeError: scheduler_metadata must have shape (metadata_size) #17283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Apr 28, 2025
Merged

[BugFix] Fix cascade attention - RuntimeError: scheduler_metadata must have shape (metadata_size) #17283

merged 1 commit into from
Apr 28, 2025

Conversation

@LucasWilkinson
Copy link
Collaborator

FIX #17276

@LucasWilkinson
fix cascade attention
01e1540 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@github-actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@LucasWilkinson LucasWilkinson mentioned this pull request Apr 28, 2025
Closed
@mergify mergify bot added the v1 label Apr 28, 2025
@LucasWilkinson LucasWilkinson added this to the v0.8.5 milestone Apr 28, 2025
@LucasWilkinson LucasWilkinson marked this pull request as ready for review April 28, 2025 13:43
tlrmchlsmth
tlrmchlsmth reviewed Apr 28, 2025
View reviewed changes
vllm/v1/attention/backends/flash_attn.py
self.runner.device)
prefix_scheduler_metadata = schedule(
batch_size=num_reqs,
batch_size=1,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain what was going on here a bit more?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prefix part of cascade attention is run as a single request (since its shared), this was a copy paste bug with the scheduling code

mgoin
mgoin reviewed Apr 28, 2025
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also check if v1 flashinfer has this issue?

@LucasWilkinson
Copy link
Collaborator Author

Could you also check if v1 flashinfer has this issue?

FlashInfer uses the cascade wrapper which doesnt have a batch size argument

@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 28, 2025
@simon-mo simon-mo merged commit cc5befb into vllm-project:main Apr 28, 2025
62 checks passed
jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Apr 29, 2025
@LucasWilkinson @dbyoung18
[BugFix] Fix cascade attention - RuntimeError: scheduler_metadata mus…
f27b4fb 
…t have shape (metadata_size) (vllm-project#17283)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025
@LucasWilkinson @lk-chen
[BugFix] Fix cascade attention - RuntimeError: scheduler_metadata mus…
bfffc90 
…t have shape (metadata_size) (vllm-project#17283)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@LucasWilkinson
[BugFix] Fix cascade attention - RuntimeError: scheduler_metadata mus…
0324d04 
…t have shape (metadata_size) (vllm-project#17283)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
@ckhordiasma ckhordiasma mentioned this pull request May 14, 2025
Merged
zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025
@LucasWilkinson
[BugFix] Fix cascade attention - RuntimeError: scheduler_metadata mus…
fbb0c3b 
…t have shape (metadata_size) (vllm-project#17283)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

@mgoin mgoin mgoin approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat

@njhill njhill Awaiting requested review from njhill

@ywang96 ywang96 Awaiting requested review from ywang96

@comaniac comaniac Awaiting requested review from comaniac

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed v1

Projects

None yet

Milestone

v0.8.5

Development

Successfully merging this pull request may close these issues.

[Bug]: nightly version: EngineCore encountered a fatal error.

4 participants

@LucasWilkinson @tlrmchlsmth @mgoin @simon-mo