Skip to content

[VLM] Support pan-and-scan for Gemma3 multi-modal processor #14672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 74 commits into from
Mar 13, 2025
Merged

[VLM] Support pan-and-scan for Gemma3 multi-modal processor #14672

merged 74 commits into from
Mar 13, 2025

Conversation

@DarkLight1337
Copy link
Member

@DarkLight1337 DarkLight1337 commented Mar 12, 2025 •
edited by github-actions bot
Loading

Please note that you need to pass --hf-overrides '{"do_pan_and_scan": true}' to enable this.

  • Add correctness tests for Gemma 3 model.

Note: This is not enabled in CI yet because of the older transformers version used in test environment.

  • Fix unnecessary warnings when processor arguments are passed to **kwargs instead of named keyword arguments.

FIX #12856
FIX #13143
FIX #14325

WoosukKwon added 30 commits March 4, 2025 01:38
@WoosukKwon
Gemma3 1B working
046e05a 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Add modeling_gemma3.py
669ae5a 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
remove
e044645 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
[TMP] Add HF Gemma 3
bd78da5 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Fix config:
a50c2d5 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
[TMP] image input
d2562cb 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Update
017239e 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Update
de0136a 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
update
12b7e9d 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
minor
26b6199 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Remove
b99336a 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
fix
004dc92 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Add kwargs
2bb965b 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Hew HF
b119945 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Fix scaling
4c67573 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
bidirectional attn
d90d410 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
sliding window
366e4b5 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Remove HF
77b9dd7 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
revert
f0f8e9d 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
add placeholder str
28e757b 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Merge branch 'main' into woosuk-gemma3
a935b24
@WoosukKwon
minor
285ffc4 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Add comments
713766b 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Minor
6746086 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
cleanup
6fa0336 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
tmp
0384ceb 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
minor
2beb199 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Docs
64ef15f 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
ruff
d92c7c1 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@WoosukKwon
Update transformers version
82acdcd 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@DarkLight1337
Fix
164ff57 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Fix
c32d51e 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Silence warnings
e7b9be9 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Fix
acc0a06 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Fix
4e33fd0 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337 DarkLight1337 added this to the v0.8.0 milestone Mar 12, 2025
@DarkLight1337
Update examples
144687b 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
WoosukKwon
WoosukKwon reviewed Mar 13, 2025
View reviewed changes
@DarkLight1337
Fix example
7e0550e 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Apply pan and scan in tests
eeb1950 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
ywang96
ywang96 reviewed Mar 13, 2025
View reviewed changes
@DarkLight1337
Update docs and examples
9d21076 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Avoid diffs
bc3a44f 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Simplify dummy data
645b186 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Fix
f30789a 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Add note
6bcd29b 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 13, 2025
ywang96
ywang96 approved these changes Mar 13, 2025
View reviewed changes
Copy link
Member

@ywang96 ywang96 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM! I left some minor comments

@DarkLight1337
Address comments
926e422 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Copy link
Member Author

DarkLight1337 commented Mar 13, 2025

Merging this early since the relevant tests have passed over the last 2 commits.

@vllm-bot vllm-bot merged commit 3824039 into vllm-project:main Mar 13, 2025
8 of 10 checks passed
@DarkLight1337 DarkLight1337 deleted the gemma3-crops branch March 13, 2025 09:23
richardsliu pushed a commit to richardsliu/vllm that referenced this pull request Mar 14, 2025
@DarkLight1337 @WoosukKwon
@ywang96 @richardsliu
[VLM] Support pan-and-scan for Gemma3 multi-modal processor (vllm-pro…
9b53236 
…ject#14672)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Roger Wang <ywang@roblox.com>
Signed-off-by: Richard Liu <ricliu@google.com>
lulmer pushed a commit to lulmer/vllm that referenced this pull request Apr 7, 2025
@DarkLight1337 @WoosukKwon
@ywang96 @lulmer
[VLM] Support pan-and-scan for Gemma3 multi-modal processor (vllm-pro…
bb5d20a 
…ject#14672)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Roger Wang <ywang@roblox.com>
Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>
@ckhordiasma ckhordiasma mentioned this pull request Apr 17, 2025
Closed
shreyankg pushed a commit to shreyankg/vllm that referenced this pull request May 3, 2025
@DarkLight1337 @WoosukKwon
@ywang96 @shreyankg
[VLM] Support pan-and-scan for Gemma3 multi-modal processor (vllm-pro…
90ac908 
…ject#14672)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Roger Wang <ywang@roblox.com>
@BugReporterZ BugReporterZ mentioned this pull request May 7, 2025
Merged
8 tasks
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@DarkLight1337 @WoosukKwon @ywang96
[VLM] Support pan-and-scan for Gemma3 multi-modal processor (vllm-pro…
bab2aeb 
…ject#14672)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Roger Wang <ywang@roblox.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@WoosukKwon WoosukKwon WoosukKwon left review comments

@ywang96 ywang96 ywang96 approved these changes

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation frontend multi-modality Related to multi-modality (#4194) ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

v0.8.0

4 participants

@DarkLight1337 @WoosukKwon @ywang96 @vllm-bot