[Bug]: Stuck request and empty streaming for gemma3 serving with ^v0.8.5

[PedroMiolaSilva]

Your current environment

ok

🐛 Describe the bug

I'm running VLLM with gemma3 and I've noticed with versions above (and including) v0.8.5, this model does not respond.

export MODEL_ID=ISTA-DASLab/gemma-3-27b-it-GPTQ-4b-128g
export MODEL_ID_PORT=8000
export MODEL_ID_GPU=0

docker run \
--runtime nvidia \
-e VLLM_USE_V1=0 \
--ipc=host \
-p "${MODEL_ID_PORT}:8000" \
--env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}" \
--env "CUDA_VISIBLE_DEVICES=${MODEL_ID_GPU}" \
-v "${HF_HOME}:/root/.cache/huggingface" \
-v "VLLM_LOGGING_LEVEL=${VLLM_LOGGING_LEVEL}" \
vllm/vllm-openai:v0.8.5 \
--model ${MODEL_ID} \
--tokenizer google/gemma-3-27b-it \
--gpu-memory-utilization 0.9 \
--max-model-len 32000 \
--max_num_seqs 8 \
--served-model-name ista-daslab-gemma-3-27b-it-gptq-4b-128g

This the example request:

curl http://localhost:8000/v1/chat/completions \
 -H "Content-Type: application/json" \
-d '{
    "stream": false,
    "model": "ista-daslab-gemma-3-27b-it-gptq-4b-128g",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": [
        {"type": "text", "text": "hello"}
        ]}
    ]
}'

And the response is never delivered. What is interesting is that the GPU KV Cache usage increases indefinetely:

INFO 05-05 07:16:33 [metrics.py:486] Avg prompt throughput: 1.9 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 05-05 07:16:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.6%, CPU KV cache usage: 0.0%.
INFO 05-05 07:16:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.1%, CPU KV cache usage: 0.0%.
INFO 05-05 07:16:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.6%, CPU KV cache usage: 0.0%.

If we set stream to true, these responses come with empty content:

data: {"id":"chatcmpl-b5b346954cdb4fe087d15231250fbee4","object":"chat.completion.chunk","created":1746454706,"model":"ista-daslab-gemma-3-27b-it-gptq-4b-128g","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-b5b346954cdb4fe087d15231250fbee4","object":"chat.completion.chunk","created":1746454706,"model":"ista-daslab-gemma-3-27b-it-gptq-4b-128g","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-b5b346954cdb4fe087d15231250fbee4","object":"chat.completion.chunk","created":1746454706,"model":"ista-daslab-gemma-3-27b-it-gptq-4b-128g","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

When I downgrade to v0.8.4, it works normally.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[mces89]

got the same issue in v0.8.5.

[gigit0000]

I was able to reproduce it on 2x RTX4090:

docker run --runtime nvidia -e VLLM_USE_V1=0 --ipc=host -p "${MODEL_ID_PORT}:8000" --env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}" --env "CUDA_VISIBLE_DEVICES=0,1" -v "${HF_HOME}:/root/.cache/huggingface" -v "VLLM_LOGGING_LEVEL=${VLLM_LOGGING_LEVEL}" vllm/vllm-openai:v0.8.5 --model ${MODEL_ID} --tokenizer google/gemma-3-27b-it --gpu-memory-utilization 0.9 --max-model-len 32000 --max_num_seqs 8 --served-model-name ista-daslab-gemma-3-27b-it-gptq-4b-128g --tensor_parallel_size 2

INFO 05-05 12:59:51 [logger.py:39] Received request chatcmpl-056276148d4045dda6e026b44dd16e18: prompt: '<start_of_turn>user\nYou are a helpful assistant.\n\nhello<end_of_turn>\n<start_of_turn>model\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=0.95, top_k=64, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=31983, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO 05-05 12:59:51 [engine.py:310] Added request chatcmpl-056276148d4045dda6e026b44dd16e18.
INFO 05-05 12:59:53 [metrics.py:486] Avg prompt throughput: 3.4 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
INFO 05-05 12:59:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 65.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.2%, CPU KV cache usage: 0.0%.
INFO 05-05 13:00:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 65.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 2.2%, CPU KV cache usage: 0.0%.
INFO 05-05 13:00:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 3.1%, CPU KV cache usage: 0.0%.
INFO 05-05 13:00:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 63.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 4.1%, CPU KV cache usage: 0.0%.
INFO 05-05 13:00:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 63.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 5.0%, CPU KV cache usage: 0.0%.
INFO 05-05 13:00:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.0%, CPU KV cache usage: 0.0%.
INFO 05-05 13:00:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.9%, CPU KV cache usage: 0.0%.

[lcw99]

temporarily add following to vision_config.
"torch_dtype": "bfloat16"

[chaunceyjiang]

It might be related to #17629.

[i-makashev]

I experienced same problem with 0.8.5, but now 0.9.1 is running without this issue

[PedroMiolaSilva]

Any news here? I've got the exact same behavior with devstral:

#!/bin/bash
source ../.env
export MODEL_ID=/root/.cache/huggingface/hub/models--mistralai--Devstral-Small-2505_gguf/snapshots/424014781ef9b398zc7f00a2f27cbc34d74e4d4a/devstralQ4_K_M.gguf
export MODEL_ID_PORT=8000
export MODEL_ID_GPU=0

docker run \
--runtime nvidia \
-e VLLM_USE_V1=1 \
--ipc=host \
-p "${MODEL_ID_PORT}:8000" \
--env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}" \
--env "CUDA_VISIBLE_DEVICES=${MODEL_ID_GPU}" \
-v "${HF_HOME}:/root/.cache/huggingface" \
-v "VLLM_LOGGING_LEVEL=${VLLM_LOGGING_LEVEL}" \
vllm/vllm-openai:v0.9.2 \
--model ${MODEL_ID} \
--gpu-memory-utilization 0.95 \
--max-model-len 32768 \
--served-model-name devstral

[hmellor]

@PedroMiolaSilva which vLLM version are you using when experiencing problems with devstral?

[PedroMiolaSilva]

@hmellor vllm/vllm-openai:v0.9.2

[hmellor]

Hmm I thought this might be related to the sliding attention issues people have been having, but Devstral is not a sliding attention model