-
-
Notifications
You must be signed in to change notification settings - Fork 11.3k
[Model] Enable BNB support for qwen2_5_omni_thinker #24420
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request re-enables LoRA support for the qwen2_5_omni_thinker model, which is a prerequisite for bitsandbytes (BNB) quantization. The changes correctly add the SupportsLoRA interface, define the packed_modules_mapping for merged layers in both the language and vision components, and implement the get_mm_mapping method to identify the different parts of the multimodal model. These changes appear correct and necessary to fix the LoRA functionality that was removed in a previous refactoring. The implementation aligns well with vLLM's patterns for multimodal LoRA support. I have reviewed the code and found no issues.
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Purpose
get_mm_mappingandSupportLoRA, deleted by Migrate Qwen2 inputs to TensorSchema #23475Test Plan
python examplexamples/offline_inference/vision_language.py \ --modality image \ --model-type qwen2_5_omniQwen/Qwen2.5-Omni-3B, with additional parameters: quantization="bitsandbytes" and enable_lora=TrueTest Result
I can generated the reasonable result on my local device:
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.