Add support for R-4B multimodal model #17840
Draft
+71
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds support for the R-4B model (YannQi/R-4B), a multimodal large language model with auto-thinking capabilities. Changes: - convert_hf_to_gguf.py: Added RVisionModel and RTextModel classes to handle the R model architecture (RForConditionalGeneration) - RVisionModel uses LFM2 projector type with scale_factor=1 (no patch merging) - RTextModel extends Qwen3Model for the language component - Proper tensor name mapping for the projector (pre_norm, linear_1, linear_2) - tools/mtmd/clip.cpp: Modified build_patch_merge_permute() to support scale_factor=1, which skips patch merging for models that don't need it - R model uses SigLIP vision encoder with 729 tokens (27x27 patches) - Projector: LayerNorm → Linear → GELU → Linear (no patch downsampling) Architecture: - Base text model: Qwen3-4B - Vision encoder: SigLIP (384x384, patch size 14) - Projector: 2-layer MLP with pre-normalization (no patch merging) - Feature selection: full (keeps all 729 vision tokens) Tested with llama-mtmd-cli and successfully generates English responses with Chinese internal reasoning (<think> tags).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit adds support for the R-4B model (YannQi/R-4B), a multimodal large language model with auto-thinking capabilities.
Changes:
convert_hf_to_gguf.py: Added RVisionModel and RTextModel classes to handle the R model architecture (RForConditionalGeneration)
tools/mtmd/clip.cpp: Modified build_patch_merge_permute() to support scale_factor=1, which skips patch merging for models that don't need it
Architecture:
Tested with llama-mtmd-cli and successfully generates English responses with Chinese internal reasoning ( tags).
Make sure to read the contributing guidelines before submitting a PR