-
-
Notifications
You must be signed in to change notification settings - Fork 11.2k
[Model] Use merge_by_field_config for MM models (H-L)
#26230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Model] Use merge_by_field_config for MM models (H-L)
#26230
Conversation
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request refactors several multi-modal models (Idefics3, Keye, Keye-VL 1.5, Kimi-VL) to use a new centralized mechanism for handling multi-modal inputs, enabled by the merge_by_field_config flag. This is a good maintainability improvement as it removes duplicated validation and reshaping code from individual model files. However, I've found a couple of potential issues. For Kimi-VL, specific reshaping logic has been removed, which is likely to break the model since it could not be tested. For Idefics3, enabling merge_by_field_config will use an existing configuration function that appears to be incorrect for image_embeds and num_patches. I've left detailed comments on these issues. The changes for other models and the improvements to error messages in tensor_schema.py are positive.
|
cc @Isotr0py |
…#26230) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
…#26230) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Karan Goel <3261985+karan@users.noreply.github.com>
…#26230) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…#26230) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
…#26230) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…#26230) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…#26230) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
…#26230) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Purpose
Part of #26149
Test Plan
Tested all models other than Kimi-VL (doesn't work with latest Transformers)
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.