[Core] Separate out attention metadata building logic from prepare inputs #26764
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LucasWilkinson
added
the
ready
LucasWilkinson
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
robertgshaw2-redhat and
ywang96
as code owners
LucasWilkinson
force-pushed
the
lwilkinson/seperate-build-attn-metadata
branch
from
3bdfe10 to
e48ab54
Compare
fhl2000
reviewed
Oct 22, 2025
Contributor
fhl2000
left a comment
fhl2000
left a comment
There was a problem hiding this comment.
Hi @LucasWilkinson, thank you for refactoring this! Just left one question, otherwise looking great.
LucasWilkinson
force-pushed
the
lwilkinson/seperate-build-attn-metadata
branch
from
db8f24c to
e9387ea
Compare
fhl2000
reviewed
Nov 3, 2025
Contributor
fhl2000
left a comment
fhl2000
left a comment
There was a problem hiding this comment.
Overall looks good! Any chance to also separate the attn_metadata building logic into a function for the Eagle drafter? So we can later reuse them for both the drafter execution and its dummy run (prepared for full CUDA graph support).
LucasWilkinson
force-pushed
the
lwilkinson/seperate-build-attn-metadata
branch
2 times, most recently
from
6d1a1d4 to
4787566
Compare
5 tasks
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
LucasWilkinson
force-pushed
the
lwilkinson/seperate-build-attn-metadata
branch
from
2814f5f to
651b9de
Compare
mgoin
approved these changes
Nov 8, 2025
Member
mgoin
left a comment
mgoin
left a comment
There was a problem hiding this comment.
LGTM, nothing clearly stood out as missed
xuebwang-amd
pushed a commit
to xuebwang-amd/vllm
that referenced
this pull request
Nov 13, 2025
…puts (vllm-project#26764) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Preparatory refactor PR for: #24002 (#23789)
Separate out the attention metadata building logic from prepare inputs so we can re-use it for dummy_runs and to setup for padding for cuda-graphs before building attention metadata (see #23789 for motivation)
Moving
_compute_cascade_attn_prefix_lensout of metadata building loops since whether or not we are using cascade attention is needed by cudagraph dispatcher that will happen before attention metadata building in #24002.NOTE: renamed
kv_cache_group_idtokv_cache_gid(common acronym in linux) to make more lines fit within the formatter max line width