Skip to content

Commit ee1404b

Browse files
[GGUF] Fix Gemma3 quantization support
This commit implements complete GGUF quantization support for Gemma3 models with true Q4_0 compression, addressing gibberish output and enabling 50% memory reduction. Changes: 1. gguf_loader.py: Add gemma3_text -> gemma3 model type mapping 2. gemma3.py: - Add Gemma3 RMSNorm weight correction (-1.0 offset) - Fix qweight_type tensor shape (scalar -> [1]) - Fix F16 embedding handling (no reshape needed) - Enable GGUF quantization in linear layers - Handle UninitializedParameter for GGUF layers Key fixes: - RMSNorm correction: Gemma3 uses (1+weight) convention but GGUF stores full values, requiring -1.0 subtraction - F16 embeddings: GGUF raw data is already in PyTorch layout, preventing data corruption from unnecessary reshape operations - qweight_type shape: GGUF layers expect shape [1] not scalar [] Tested on: - 8 Gemma3 variants (1B-27B parameters) - Both instruction-tuned and pretrained versions - Q4_0 quantization format - 100% success rate with coherent text generation Fixes #14753, #15480 Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
1 parent d76541a commit ee1404b

File tree

2 files changed

+246
-58
lines changed

2 files changed

+246
-58
lines changed

vllm/model_executor/model_loader/gguf_loader.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,10 @@ def _get_gguf_weights_map(self, model_config: ModelConfig):
6363
# hack: ggufs have a different name than transformers
6464
if model_type == "cohere":
6565
model_type = "command-r"
66+
if model_type == "gemma3_text":
67+
# Gemma3 models use "gemma3_text" in HuggingFace but
68+
# "gemma3" in GGUF architecture naming
69+
model_type = "gemma3"
6670
if model_type in ("deepseek_v3", "deepseek_v2"):
6771
model_type = "deepseek2"
6872
# GGUF layer map assumes that we will have a merged expert weights

0 commit comments

Comments
 (0)