[Multimodal] Improve max video embedding length estimation in V1 #24312
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ywang96
requested review from
DarkLight1337 and
Isotr0py
and removed request for
sighingnow
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request correctly improves the estimation of the maximum video embedding length for V1 models. By removing the subtraction of image token counts from the available sequence length, the calculation now accurately reflects the capabilities of the V1 architecture with chunked prefill, where video and image data do not need to fit into the context window simultaneously. The changes in llava_onevision.py and qwen2_vl.py are consistent and simplify the logic as intended. The code is cleaner and more aligned with the V1 processing pipeline. Overall, this is a good improvement.
Isotr0py
approved these changes
Sep 5, 2025
github-actions
bot
added
the
ready
1 task
eicherseiji
pushed a commit
to eicherseiji/vllm
that referenced
this pull request
Sep 9, 2025
…m-project#24312) Signed-off-by: Roger Wang <hey@rogerw.me> Co-authored-by: Roger Wang <hey@rogerw.me>
skyloevil
pushed a commit
to skyloevil/vllm
that referenced
this pull request
Sep 13, 2025
…m-project#24312) Signed-off-by: Roger Wang <hey@rogerw.me> Co-authored-by: Roger Wang <hey@rogerw.me>
5 tasks
FeiDaLI
pushed a commit
to FeiDaLI/vllm
that referenced
this pull request
Sep 25, 2025
…m-project#24312) Signed-off-by: Roger Wang <hey@rogerw.me> Co-authored-by: Roger Wang <hey@rogerw.me>
xuebwang-amd
pushed a commit
to xuebwang-amd/vllm
that referenced
this pull request
Oct 10, 2025
…m-project#24312) Signed-off-by: Roger Wang <hey@rogerw.me> Co-authored-by: Roger Wang <hey@rogerw.me> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
xuebwang-amd
pushed a commit
to xuebwang-amd/vllm
that referenced
this pull request
Oct 24, 2025
…m-project#24312) Signed-off-by: Roger Wang <hey@rogerw.me> Co-authored-by: Roger Wang <hey@rogerw.me> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
In V0, multimodal profiling was based on the assumption that the batch size (by default model context window) fits all number of modality items of the biggest sizes, therefore the max number of video frames was computed based on how much space there is after fitting all images.
In V1, we no longer require this assumption because of chunked prefill and therefore there's no need to consider image tokens to calculate the max number of frames of videos.
A follow up is to further simplify dummy data generation since in V1 we don't need to consider the number of modality items but just whether a certain modality is allowed or not.
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.