[V1] Scatter and gather placeholders in the model runner #16076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

DarkLight1337 merged 39 commits into main from v1-is-embed

Apr 8, 2025

Member

ywang96 commented Apr 4, 2025 •

edited by github-actions bot

Loading

Reopened from accidentally merged #15712

This PR is an attempt to move scatter_patch_features and gather_patch_feature into the model runner (outside of the model) to avoid interfering with TPU graph compilation.

Breaking change for model developers:

PromptUpdateDetails.features has been replaced with PromptUpdateDetails.is_embed. You can use the newly added factories PromptUpdateDetails.select_text and PromptUpdateDetails.select_token_id to generate is_embed based on the target text/token ID.
BaseProcessingInfo.get_num_image_tokens should now return the equivalent of PromptUpdateDetails.is_embed.sum() instead of the number of tokens in PromptUpdateDetails.features.

Originally authored by @DarkLight1337; co-authored by @mgoin

DarkLight1337 and others added 26 commits

March 31, 2025 16:41


          [V1] Scatter and gather placeholders in the model runner

9e4e312

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Fix placeholder token calculation

d2ad7e6

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Improve error message

776074b

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Loosen check

d8c10c8

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Fix Molmo

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Rename

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Fix gemma3

d1568ad

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Skip h2ovl for current transformers

a68228e

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Fix MiniCPM

cb3ccce

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Fix Fuyu

cff414e

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Merge branch 'main' into v1-is-embed

0cd2a54


          Fix NVLM

699c346

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Enable PlaceholderRange equality check

cd86861

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Fix

f9b92df

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Fix pos_info.is_embed

53f5507

Signed-off-by: mgoin <mgoin64@gmail.com>


          Fix Idefics3

f798d89

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Merge branch 'main' into v1-is-embed

7935a8d


          Remove embed_is_patch from new models

06bf57d

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Fix missing replacement

df174e6

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Fix Pixtral token calculation and update docs

1e9fb1b

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Merge remote-tracking branch 'upstream/main' into v1-is-embed

142baf4

Signed-off-by: Roger Wang <ywang@roblox.com>


          Fix CI

e3ec92d

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          increase default timeout for testing

6d51118

Signed-off-by: Roger Wang <ywang@roblox.com>


          typo

4940d16

Signed-off-by: Roger Wang <ywang@roblox.com>


          lower max model len

522833b

Signed-off-by: Roger Wang <ywang@roblox.com>


          update pixtral test

b64983c

Signed-off-by: Roger Wang <ywang@roblox.com>

ywang96 requested review from DarkLight1337, WoosukKwon, njhill and robertgshaw2-redhat as code owners

April 4, 2025 21:59

DarkLight1337 added 2 commits

April 5, 2025 11:07


          Avoid OOM

fbc9345

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Skip Phi-3.5-Vision tests

2b16f91

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

This was referenced Apr 5, 2025

[Model] Add smolvlm support #16017

Merged

[Model] Refactor Phi-4-multimodal to use merged processor and support V1 #15477

Merged

Member Author

ywang96 commented Apr 6, 2025

I'm holding on merging this PR until #16113 and #16117 are merged.

DarkLight1337 mentioned this pull request

[Model] Add SupportsMultiModal.get_language_model interface #16007

Merged

DarkLight1337 added a commit to houseroad/vllm that referenced this pull request


          Apply fixes from vllm-project#16076

d76eac1

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

DarkLight1337 added 2 commits

April 7, 2025 15:08


          Merge branch 'main' into v1-is-embed

4bd0025

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>


          Update llama4

edd66c6

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Member

DarkLight1337 commented Apr 7, 2025 •

edited

Loading

Verifying llama4 now...

DarkLight1337 added 4 commits

April 7, 2025 15:21


          Update tests

c0300d2

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Fix

f837467

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Fix

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Fix

9e2b427

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

mergify bot commented Apr 7, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ywang96.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify bot added the needs-rebase label


          Merge branch 'main' into v1-is-embed

988fc49

mergify bot removed the needs-rebase label

DarkLight1337 approved these changes

View reviewed changes

DarkLight1337 merged commit f2ebb6f into main

45 checks passed

DarkLight1337 deleted the v1-is-embed branch

April 8, 2025 02:43

github-project-automation bot moved this from In Progress to Done in Multi-modality Core

DarkLight1337 mentioned this pull request

[Model] Support VLMs with transformers backend #13754

Closed

yangw-dev pushed a commit to yangw-dev/vllm that referenced this pull request


          [V1] Scatter and gather placeholders in the model runner (vllm-projec…

1084d5e

…t#16076)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
Signed-off-by: Yang Wang <elainewy@meta.com>

jikunshang pushed a commit to jikunshang/vllm that referenced this pull request


          [V1] Scatter and gather placeholders in the model runner (vllm-projec…

cb1b3b4

…t#16076)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

lk-chen pushed a commit to lk-chen/vllm that referenced this pull request


          [V1] Scatter and gather placeholders in the model runner (vllm-projec…

8c15934

…t#16076)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>

RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request


          [V1] Scatter and gather placeholders in the model runner (vllm-projec…

ca6ebfa

…t#16076)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

DarkLight1337 DarkLight1337 approved these changes

WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

njhill Awaiting requested review from njhill njhill is a code owner

comaniac Awaiting requested review from comaniac comaniac is a code owner

alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

Labels

documentation multi-modality ready tpu v1