Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Qwen2.5VL support #12402

Draft
wants to merge 6 commits into
base: master
Choose a base branch
from
Draft

Add Qwen2.5VL support #12402

wants to merge 6 commits into from

Conversation

HimariO
Copy link
Contributor

@HimariO HimariO commented Mar 15, 2025

Original issue: #11483

Changes

  • Add new gguf key for clip model to support
    • GLU MLP,
    • window attention,
    • RMS norm
  • Updated clip.cpp vision model to incorporate these new components.
  • Modified qwen2_vl_surgery.py and convert_hf_to_gguf.py to support the Qwen2.5VL model.

Model Conversion

The only change in the conversion process compared to Qwen2VL is the addition of the model_type parameter when creating the vision encoder GGUF file. (For the rest of the process and how to build llama-qwen2vl-cli, refer to #10361.)

PYTHONPATH=$PYTHONPATH:$(pwd)/gguf-py python3 examples/llava/qwen2_vl_surgery.py "/path/to/model" --data_type fp16 --model_type "qwen2.5vl"

@github-actions github-actions bot added examples python python script changes labels Mar 15, 2025
@HimariO HimariO mentioned this pull request Mar 16, 2025
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
examples python python script changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant