[Bug]: No output / Repeated outputs when using Gemma 3 on vLLM

[syngokhan]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

🐛 Describe the bug

I'm running the google/gemma-3-27b-it model with vLLM using the OpenAI-compatible API server.

CUDA_VISIBLE_DEVICES=0 VLLM_USE_V1=1 python /opt/VLLM/vllm/vllm/entrypoints/openai/api_server.py \
--model /opt/MODELS/gemma-3-27b-it/ \
--max-model-len 32000 \
--host 10.12.112.168 \
--port 9005 \
--tensor-parallel-size 1 \
--gpu_memory_utilization 0.9

Then, I send a standard request to the /v1/chat/completions endpoint using Python:

import requests
import json

url = "http://10.12.112.168:9005/v1/chat/completions"

data = {
    "model": "/opt/MODELS/gemma-3-27b-it/",
    "messages": [
        {"role": "user", "content": "hello"}
    ],
    "temperature": 0.1,
    "max_tokens": 500,
    "enable_thinking": False
}

headers = {
    "Content-Type": "application/json"
}

response = requests.post(url, headers=headers, data=json.dumps(data))
result = response.json()
print(result['choices'][0]['message']['content'])

The request is processed, but the model fails to produce meaningful responses. It either:

outputs nothing,

or keeps repeating certain tokens or parts of the input (e.g., repeating “selamlar brom”).

This issue only happens with Gemma 3 IT models. I tested the exact same code and server setup with:

Qwen models

Mistral models
...and they work perfectly. No repetition, and responses are coherent and aligned with the prompt.

So this looks like a Gemma-specific compatibility issue with /chat/completions, possibly due to missing or misaligned prompt formatting (e.g., lack of a compatible chat template?).

Let me know if there’s a known workaround or proper configuration required for Gemma models.

Thanks!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

Can you share any vllm logs?

[robertgshaw2-redhat]

cc @simon-mo for example of stability

[syngokhan]

setup info log;

(vllm) [root@gapxllmgpup01 vllm]# CUDA_VISIBLE_DEVICES=0 VLLM_USE_V1=1 python /opt/VLLM/vllm/vllm/entrypoints/openai/api_server.py --model /opt/MODELS/gemma-3-27b-it/ --max-model-len 32000 --host 10.12.112.168 --port 9005 --tensor-parallel-size 1 --gpu_memory_utilization 0.9
INFO 07-02 01:42:40 [init.py:244] Automatically detected platform cuda.
INFO 07-02 01:42:42 [api_server.py:1390] vLLM API server version 0.9.2.dev357+g7f280d69c
INFO 07-02 01:42:42 [cli_args.py:325] non-default args: {'host': '10.12.112.168', 'port': 9005, 'model': '/opt/MODELS/gemma-3-27b-it/', 'max_model_len': 32000}
INFO 07-02 01:42:47 [config.py:835] This model supports multiple tasks: {'embed', 'reward', 'classify', 'generate', 'score'}. Defaulting to 'generate'.
INFO 07-02 01:42:47 [config.py:1449] Using max model len 32000
INFO 07-02 01:42:47 [config.py:2249] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 07-02 01:42:51 [init.py:244] Automatically detected platform cuda.
INFO 07-02 01:42:52 [core.py:459] Waiting for init message from front-end.
INFO 07-02 01:42:52 [core.py:69] Initializing a V1 LLM engine (v0.9.2.dev357+g7f280d69c) with config: model='/opt/MODELS/gemma-3-27b-it/', speculative_config=None, tokenizer='/opt/MODELS/gemma-3-27b-it/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/opt/MODELS/gemma-3-27b-it/, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
WARNING 07-02 01:42:53 [utils.py:2782] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fcde01fb6b0>
INFO 07-02 01:42:53 [parallel_state.py:1076] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
Using a slow image processor as use_fast is unset and a slow processor was saved with this model. use_fast=True will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with use_fast=False.
WARNING 07-02 01:42:56 [topk_topp_sampler.py:59] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
INFO 07-02 01:42:56 [gpu_model_runner.py:1787] Starting to load model /opt/MODELS/gemma-3-27b-it/...
INFO 07-02 01:42:57 [gpu_model_runner.py:1792] Loading model from scratch...
INFO 07-02 01:42:57 [cuda.py:270] Using Flash Attention backend on V1 engine.
Loading safetensors checkpoint shards: 0% Completed | 0/12 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 8% Completed | 1/12 [00:00<00:05, 1.87it/s]
Loading safetensors checkpoint shards: 17% Completed | 2/12 [00:01<00:05, 1.76it/s]
Loading safetensors checkpoint shards: 25% Completed | 3/12 [00:01<00:05, 1.73it/s]
Loading safetensors checkpoint shards: 33% Completed | 4/12 [00:02<00:04, 1.72it/s]
Loading safetensors checkpoint shards: 42% Completed | 5/12 [00:02<00:04, 1.72it/s]
Loading safetensors checkpoint shards: 50% Completed | 6/12 [00:03<00:03, 1.72it/s]
Loading safetensors checkpoint shards: 58% Completed | 7/12 [00:04<00:02, 1.72it/s]
Loading safetensors checkpoint shards: 67% Completed | 8/12 [00:04<00:01, 2.33it/s]
Loading safetensors checkpoint shards: 75% Completed | 9/12 [00:04<00:01, 2.16it/s]
Loading safetensors checkpoint shards: 83% Completed | 10/12 [00:05<00:01, 2.00it/s]
Loading safetensors checkpoint shards: 92% Completed | 11/12 [00:05<00:00, 1.90it/s]
Loading safetensors checkpoint shards: 100% Completed | 12/12 [00:06<00:00, 1.84it/s]
Loading safetensors checkpoint shards: 100% Completed | 12/12 [00:06<00:00, 1.86it/s]

INFO 07-02 01:43:03 [default_loader.py:272] Loading weights took 6.49 seconds
INFO 07-02 01:43:04 [gpu_model_runner.py:1818] Model loading took 51.4170 GiB and 6.631658 seconds
INFO 07-02 01:43:04 [gpu_model_runner.py:2257] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 32 image items of the maximum feature size.
INFO 07-02 01:43:17 [backends.py:508] Using cache directory: /root/.cache/vllm/torch_compile_cache/d18a2a02da/rank_0_0/backbone for vLLM's torch.compile
INFO 07-02 01:43:17 [backends.py:519] Dynamo bytecode transform time: 11.63 s
INFO 07-02 01:43:26 [backends.py:155] Directly load the compiled graph(s) for shape None from the cache, took 9.061 s
INFO 07-02 01:43:32 [monitor.py:34] torch.compile takes 11.63 s in total
INFO 07-02 01:43:33 [gpu_worker.py:232] Available KV cache memory: 22.34 GiB
INFO 07-02 01:43:33 [kv_cache_utils.py:716] GPU KV cache size: 47,216 tokens
INFO 07-02 01:43:33 [kv_cache_utils.py:720] Maximum concurrency for 32,000 tokens per request: 5.11x
WARNING 07-02 01:43:33 [utils.py:101] Unable to detect current VLLM config. Defaulting to NHD kv cache layout.
Capturing CUDA graph shapes: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:23<00:00, 2.81it/s]
INFO 07-02 01:43:57 [gpu_model_runner.py:2345] Graph capturing finished in 24 secs, took 1.22 GiB
INFO 07-02 01:43:57 [core.py:172] init engine (profile, create kv cache, warmup model) took 53.23 seconds
INFO 07-02 01:43:58 [loggers.py:137] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 2951
WARNING 07-02 01:43:58 [config.py:1376] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with --generation-config vllm.
INFO 07-02 01:43:58 [serving_chat.py:124] Using default chat sampling params from model: {'top_k': 64, 'top_p': 0.95}
INFO 07-02 01:43:58 [serving_completion.py:71] Using default completion sampling params from model: {'top_k': 64, 'top_p': 0.95}
INFO 07-02 01:43:58 [api_server.py:1452] Starting vLLM API server 0 on http://10.12.112.168:9005
INFO 07-02 01:43:58 [launcher.py:29] Available routes are:
INFO 07-02 01:43:58 [launcher.py:37] Route: /openapi.json, Methods: GET, HEAD
INFO 07-02 01:43:58 [launcher.py:37] Route: /docs, Methods: GET, HEAD
INFO 07-02 01:43:58 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 07-02 01:43:58 [launcher.py:37] Route: /redoc, Methods: GET, HEAD
INFO 07-02 01:43:58 [launcher.py:37] Route: /health, Methods: GET
INFO 07-02 01:43:58 [launcher.py:37] Route: /load, Methods: GET
INFO 07-02 01:43:58 [launcher.py:37] Route: /ping, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /ping, Methods: GET
INFO 07-02 01:43:58 [launcher.py:37] Route: /tokenize, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /detokenize, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /v1/models, Methods: GET
INFO 07-02 01:43:58 [launcher.py:37] Route: /version, Methods: GET
INFO 07-02 01:43:58 [launcher.py:37] Route: /v1/chat/completions, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /v1/completions, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /v1/embeddings, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /pooling, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /classify, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /score, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /v1/score, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /v1/audio/translations, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /rerank, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /v1/rerank, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /v2/rerank, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /invocations, Methods: POST
INFO 07-02 01:43:58 [launcher.py:37] Route: /metrics, Methods: GET
INFO: Started server process [92676]
INFO: Waiting for application startup.
INFO: Application startup complete.

Request info log

INFO: 10.12.73.25:58006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 07-02 01:45:18 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 47.1%
INFO 07-02 01:45:28 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 47.1%
INFO 07-02 01:46:06 [logger.py:43] Received request chatcmpl-20610e35841b4d84968fc38379fd9846: prompt: '<start_of_turn>user\nhello<end_of_turn>\n<start_of_turn>model\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.1, top_p=0.95, top_k=64, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=500, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
INFO 07-02 01:46:06 [async_llm.py:270] Added request chatcmpl-20610e35841b4d84968fc38379fd9846.
INFO 07-02 01:46:08 [loggers.py:118] Engine 000: Avg prompt throughput: 1.0 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 36.4%

Answer :

import requests
import json

url = "http://10.12.112.168:9005/v1/chat/completions"

data = {
    "model": "/opt/MODELS/gemma-3-27b-it/",  # Sunucudaki modelin adını yazın, bazen zorunlu olmayabilir.
    "messages": [
        #{"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "hello"}
    ],
    "temperature": 0.1,
    "max_tokens": 500,
 }

headers = {
    "Content-Type": "application/json"
}

try:

    response = requests.post(url, headers=headers, data=json.dumps(data))

    response.raise_for_status()  # Eğer 2xx dışında bir kod geldiyse (örn: 404, 500) hata fırlatır.

    result = response.json()

    print(result['choices'][0]['message']['content'])

except requests.exceptions.RequestException as e:
    print(f"Bir hata oluştu: {e}")

Okay, I want to create a list the following the a a a a a list of the top 10-10-10-5 most popular and 5 most popular and most popular tourist attractions in-visited tourist attractions in Paris, France. 

## Top 5 most popular tourist attractions in Paris, France. 

## attractions in Paris, France. 

## attractions in France.

## Top 5 most popular tourist attractions in France.

## attractions in Paris, France.

Okay, France.

Okay, and a list of the 5 most popular attractions.

most popular tourist attractions in France......

[nathanrahn]

I've also been observing this exact behavior on v0.9.1 with Gemma3 IT models -- they have completely broken outputs. I tried using the V0 engine and dtype bfloat16, which did not fix the problem. There's also #17689 and #20060 which are potentially related.

[benjamin-marie]

Same issue for me, but not everywhere! Unfortunately, I didn't keep any logs. What I can say is that on Google Colab's L4, it works fine, but on Hyperbolic's H100 spot instance, it completely fails for the same prompts. Both vLLM installed with pip install vLLM.
I can add that changing the hyperparameters doesn't help.
Maybe related to specific optimizations for Hopper GPUs?

Same issue with V0 and V1 engines.

[nicolofranceschi]

Same here

[syngokhan]

When I set up the same configuration on two identical H100 machines, one works perfectly while the other encounters the issue we previously mentioned.

Additionally:

The working machine uses nvidia-driver 12.6, while the faulty one uses nvidia-driver 12.8. I don’t believe this driver version difference is the root cause, but I wanted to mention it for completeness.

The main problem is that the issue only appears on one of the machines, and it's difficult to pinpoint because there are no error logs or any indication of failure.

I’m struggling to identify what could be causing the discrepancy between the two systems.

Note:

As an additional check, I tried setting up a server using SgLang to test whether there were any issues related to the model or the overall setup. Everything worked smoothly there, and I didn't encounter any problems.

[chenfolingithub]

I solved the problem in the following way:
pip install transformers==4.51.1
The default installed version is 4.53.0, and I think it may be due to incompatibility between this version of transformers and gemma-3

[timbmg]

I solved the problem in the following way: pip install transformers==4.51.1 The default installed version is 4.53.0, and I think it may be due to incompatibility between this version of transformers and gemma-3

Cool, I'll check it! Are you using V0 or V1 engine?

[syngokhan]

The issue was caused by the transformers library version. Using transformers==4.52.2 resolved the problem.

[syngokhan]

I solved the problem in the following way: pip install transformers==4.51.1 The default installed version is 4.53.0, and I think it may be due to incompatibility between this version of transformers and gemma-3

I really appreciate your help — thank you!

[benjamin-marie]

I also confirm: Transformers 4.52.2 works.

But I don't know whether this is a problem only related to the recent Transformers version (need to try to generate with it) or its integration with vLLM.

[RyanLoil]

This stupid problem cost me 3 hours to fix as my teamate's environment working nothing wrong while my new environment got gemma-3-27b-it repeat itself with "You are You are You are You are".

transformers == 4.52.4 works.
DO NOT INSTALL transformers >=4.53.0