[Feature]: Support gemma3 architecture

[manwithaplandy]

🚀 The feature, motivation and pitch

I am using vLLM for hosting of LLMs/SLMs and with the recent release of Gemma 3, I would love to have it supported in vLLM. Google has stated Gemma 3 has day 1 support from HF Transformers, so it should (hopefully) be relatively simple to integrate into vLLM.

Currently, when attempting to load google/gemma-3-12b-it, the following error is given:

ERROR 03-12 18:19:00 engine.py:400] ValueError: The checkpoint you are trying to load has model type `gemma3` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
ERROR 03-12 18:19:00 engine.py:400]
ERROR 03-12 18:19:00 engine.py:400] You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`

Alternatives

No response

Additional context

https://blog.google/technology/developers/gemma-3/

https://developers.googleblog.com/en/introducing-gemma3/

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

vLLM already supports it. You need to build from the main branch.

[mrfelipemartins]

Even using latest commit from main I was still getting this error:

ValueError: The checkpoint you are trying to load has model type `gemma3` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

I had to update transformers from latest commit as well:

pip install git+https://github.com/huggingface/transformers.git

[gemchen]

already installed "vllm-0.7.4.dev399+gce201246.precompiled" with source code.
but when trying to launch GGUF quantized model raise such error, can anyone help?

ValueError: GGUF model with architecture gemma3 is not supported yet

[DarkLight1337]

As mentioned in the docs, yes you have to install transformers from source as well.

[DarkLight1337]

already installed "vllm-0.7.4.dev399+gce201246.precompiled" with source code. but when trying to launch GGUF quantized model raise such error, can anyone help?

ValueError: GGUF model with architecture gemma3 is not supported yet

If you have installed both dev vLLM and dev transformers and still get this issue, please open a separate issue and mention GGUF in the title for better visibility.

[manwithaplandy]

Do we know when this will be packaged into a release?

[DarkLight1337]

Around end of this week.

[cwh1981]

pip install git+https://github.com/huggingface/transformers@v4.49.0-Gemma-3

[cwh1981]

git clone https://github.com/vllm-project/vllm.git
cd vllm
VLLM_USE_PRECOMPILED=1 pip install --editable .

[llajan]

I think there are still some issues with Gemma 3 support. I used the Gemma 3 version of transformers and the latest code from main (as described above by @cwh1981).
When I load the Gemma 3 27b it model, with the following command, the server itself gets launched, but in the logs vllm complains that some weights are not initialized from checkpoint - I suspect the model may not have loaded properly.
python -m vllm.entrypoints.openai.api_server --tensor-parallel-size 4 --model google/gemma-3-27b-it --port 8000

Some weights are not initialized from checkpoints: {'layers.48.post_feedforward_layernorm.weight', 'layers.28.post_feedforward_layernorm.weight', 'layers.28.input_layernorm.weight', 'layers.8.input_layernorm.weight', 'layers.9.self_attn.qkv_proj.weight', 'layers.32.self_attn.o_proj.weight', 'layers.40.pre_feedf
orward_layernorm.weight', 'layers.40.post_attention_layernorm.weight', 'layers.43.input_layernorm.weight', 'layers.13.self_attn.q_norm.weight', 'layers.38.self_attn.q_norm.weight', 'layers.7.self_attn.q_norm.weight', 'layers.39.self_attn.k_norm.weight', 'layers.41.post_attention_layernorm.weight', 'layers.57.self_attn.o_proj.weight', 'layers.55.mlp.down_proj.weight', 'layers.29.se
lf_attn.k_norm.weight', 'layers.47.post_attention_layernorm.weight', ....

[funes79]

pip install git+https://github.com/huggingface/transformers@v4.49.0-Gemma-3

llava 1.2.2.post1 requires timm==0.6.13, which is not installed.
llava 1.2.2.post1 requires accelerate==0.21.0, but you have accelerate 1.5.1 which is incompatible.
llava 1.2.2.post1 requires tokenizers==0.15.1, but you have tokenizers 0.21.1 which is incompatible.
llava 1.2.2.post1 requires transformers==4.37.2, but you have transformers 4.50.0.dev0 which is incompatible.

Tried to uninstall these newer versions, but then bumped into another dependency issue

[funes79]

git clone https://github.com/vllm-project/vllm.git cd vllm VLLM_USE_PRECOMPILED=1 pip install --editable .

Worth noting that this does not work on mac.

AssertionError: VLLM_USE_PRECOMPILED is only supported for CUDA builds
      /private/var/folders/xc/0dd3k5sx3v34xqw6ps7r4rd80000gn/T/pip-build-env-bpy0c6n1/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:1021: _DebuggingTips: Problem in editable installation.

[thiyagu-lily]

I think there are still some issues with Gemma 3 support. I used the Gemma 3 version of transformers and the latest code from main (as described above by @cwh1981). When I load the Gemma 3 27b it model, with the following command, the server itself gets launched, but in the logs vllm complains that some weights are not initialized from checkpoint - I suspect the model may not have loaded properly. python -m vllm.entrypoints.openai.api_server --tensor-parallel-size 4 --model google/gemma-3-27b-it --port 8000

Some weights are not initialized from checkpoints: {'layers.48.post_feedforward_layernorm.weight', 'layers.28.post_feedforward_layernorm.weight', 'layers.28.input_layernorm.weight', 'layers.8.input_layernorm.weight', 'layers.9.self_attn.qkv_proj.weight', 'layers.32.self_attn.o_proj.weight', 'layers.40.pre_feedf
orward_layernorm.weight', 'layers.40.post_attention_layernorm.weight', 'layers.43.input_layernorm.weight', 'layers.13.self_attn.q_norm.weight', 'layers.38.self_attn.q_norm.weight', 'layers.7.self_attn.q_norm.weight', 'layers.39.self_attn.k_norm.weight', 'layers.41.post_attention_layernorm.weight', 'layers.57.self_attn.o_proj.weight', 'layers.55.mlp.down_proj.weight', 'layers.29.se
lf_attn.k_norm.weight', 'layers.47.post_attention_layernorm.weight', ....

Facing the same issue.