[Model] Add support for LightOnOCR #26916

staghado · 2025-10-15T15:12:28Z

Purpose

This PR adds support for LightOnOCR: a SOTA 1B OCR VLM built on top of Mistral3 ViT and Qwen3 LM decoder.

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

- Updated `supported_models.md` to include `LightOnOCRForConditionalGeneration`. - Implemented `run_lightonocr` function in `vision_language.py` for handling LightOnOCR model requests. - Registered LightOnOCR in the model registry for example usage. Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

github-actions · 2025-10-15T15:12:39Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

mergify · 2025-10-15T15:13:15Z

Documentation preview: https://vllm--26916.org.readthedocs.build/en/26916/

gemini-code-assist

Code Review

This pull request adds support for the LightOnOCR model. The implementation is well-structured and follows the existing patterns for multimodal models within the vLLM project. I have identified one high-severity issue in the example script related to an unused function parameter, which could be misleading. My review includes a code suggestion to address this by adhering to a common Python convention for unused variables, which will improve code clarity.

examples/offline_inference/vision_language.py

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

vllm/model_executor/models/lightonocr.py

gemini-code-assist

Code Review

This pull request adds support for the LightOnOCR model. The changes include updating documentation, adding an example, and implementing the model logic. The implementation is based on the Mistral3/Pixtral architecture. My review identified a critical bug in the input parsing logic that could lead to a crash, and a high-severity issue in token processing that might result in incorrect behavior. I have provided suggestions to address these issues. The other changes appear to be correct.

vllm/model_executor/models/lightonocr.py

vrdn-23 · 2025-10-15T19:09:42Z

Apologies if this obvious, but I can't seem to find the model on Huggingface. Could you provide a link to where the model is hosted?

staghado · 2025-10-15T19:54:43Z

That's normal as the model is not released yet! This PR aims to add support for it so it's supported from the launch day. I can also provide more info if needed or provide a dummy model for testing the code.

docs/models/supported_models.md

examples/offline_inference/vision_language.py

tests/models/registry.py

vllm/model_executor/models/lightonocr.py

DarkLight1337 · 2025-10-16T04:01:36Z

Thanks for adding your implementation to vLLM! Made some initial comments.

…rted_models.md Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

…mples Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com> Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Said Taghadouini <84044788+staghado@users.noreply.github.com>

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

… registry Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

staghado · 2025-10-16T12:42:08Z

Thanks for the quick feedback!
I tried to address the different points, let me know if there is something else to fix!

DarkLight1337 · 2025-10-16T12:48:51Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for the LightOnOCR model, a 1B parameter OCR Vision Language Model. The changes span documentation, examples, and the core model implementation, which appears to be well-integrated and follows the existing patterns for similar multimodal models in the repository. I've identified one critical issue in the new model implementation that could lead to a server crash under certain conditions and have provided a code suggestion to address it.

vllm/model_executor/models/lightonocr.py

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

DarkLight1337

Thanks for the quick turnaround, have you verified that the model works correctly? If so then we can merge it.

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

staghado · 2025-10-16T14:01:19Z

I have verified by launching the model locally and it still works as before 🚀

DarkLight1337

LGTM then, thanks for adding this!

staghado · 2025-10-16T14:23:29Z

I have one final hf_to_vllm_mapper issue to fix to ensure the Transformers impl also can load the weights from the same repo!
cc @DarkLight1337

DarkLight1337 · 2025-10-16T14:34:13Z

Sure, tell me when that's done

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

staghado · 2025-10-16T15:26:15Z

it should be good now, thanks for waiting!

tests/models/registry.py

Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com> Signed-off-by: Said Taghadouini <84044788+staghado@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com> Signed-off-by: Said Taghadouini <84044788+staghado@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com> Signed-off-by: Said Taghadouini <84044788+staghado@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com> Signed-off-by: Said Taghadouini <84044788+staghado@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com> Signed-off-by: Said Taghadouini <84044788+staghado@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com> Signed-off-by: Said Taghadouini <84044788+staghado@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

staghado and others added 4 commits October 15, 2025 14:12

[Model] Add LightOnOCR model implementation

5ddc62f

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

Merge branch 'vllm-project:main' into main

47d6774

remove unused question from run_lightonocr example

5acd7a7

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

staghado requested review from DarkLight1337 and ywang96 as code owners October 15, 2025 15:12

mergify bot added documentation Improvements or additions to documentation new-model Requests to new models labels Oct 15, 2025

gemini-code-assist bot reviewed Oct 15, 2025

View reviewed changes

examples/offline_inference/vision_language.py Outdated Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Oct 15, 2025

View reviewed changes

vllm/model_executor/models/lightonocr.py Outdated Show resolved Hide resolved

gemini-code-assist bot reviewed Oct 15, 2025

View reviewed changes

vllm/model_executor/models/lightonocr.py Outdated Show resolved Hide resolved

vllm/model_executor/models/lightonocr.py Outdated Show resolved Hide resolved

DarkLight1337 reviewed Oct 16, 2025

View reviewed changes

staghado and others added 6 commits October 16, 2025 09:51

move LightOnOCRForConditionalGeneration to multimodal models in suppo…

c8bd87f

…rted_models.md Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

Respect alphabetical order in both registry and offline inference exa…

937c6f0

…mples Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com> Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

Update vllm/model_executor/models/lightonocr.py

46ae8a7

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Signed-off-by: Said Taghadouini <84044788+staghado@users.noreply.github.com>

remove flatten_bn and use merge_by_field_config = True

4ef0c3d

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

Merge branch 'main' into main

f08673f

Reorder LightOnOCRForConditionalGeneration entry in multimodal models…

9ef9876

… registry Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

gemini-code-assist bot reviewed Oct 16, 2025

View reviewed changes

vllm/model_executor/models/lightonocr.py Show resolved Hide resolved

DarkLight1337 reviewed Oct 16, 2025

View reviewed changes

vllm/model_executor/models/lightonocr.py Outdated Show resolved Hide resolved

use inheritance to reduce LoC

f1ea50b

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

DarkLight1337 reviewed Oct 16, 2025

View reviewed changes

remove unused import of Mistral3ImagePixelInputs from lightonocr.py

77c3e70

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

DarkLight1337 approved these changes Oct 16, 2025

View reviewed changes

DarkLight1337 added this to the v0.11.1 milestone Oct 16, 2025

DarkLight1337 enabled auto-merge (squash) October 16, 2025 14:05

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 16, 2025

DarkLight1337 disabled auto-merge October 16, 2025 14:34

adapt hf_to_vllm_mapper to load from same repo as Transformers

93d7b43

Signed-off-by: Said Taghadouini <taghadouinisaid@gmail.com>

DarkLight1337 enabled auto-merge (squash) October 16, 2025 15:34

DarkLight1337 reviewed Oct 17, 2025

View reviewed changes

tests/models/registry.py Show resolved Hide resolved

Update tests/models/registry.py

d972279

Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

DarkLight1337 merged commit 3aeb19a into vllm-project:main Oct 17, 2025
54 checks passed

Uh oh!

[Model] Add support for LightOnOCR #26916

[Model] Add support for LightOnOCR #26916

Uh oh!

Conversation

staghado commented Oct 15, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

github-actions bot commented Oct 15, 2025

Uh oh!

mergify bot commented Oct 15, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

vrdn-23 commented Oct 15, 2025

Uh oh!

staghado commented Oct 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 commented Oct 16, 2025

Uh oh!

staghado commented Oct 16, 2025

Uh oh!

DarkLight1337 commented Oct 16, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

DarkLight1337 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

staghado commented Oct 16, 2025

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

staghado commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkLight1337 commented Oct 16, 2025

Uh oh!

staghado commented Oct 16, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

staghado commented Oct 15, 2025 •

edited by github-actions bot

Loading

DarkLight1337 left a comment •

edited

Loading

staghado commented Oct 16, 2025 •

edited

Loading