[Bug]: Gemma-3 models output empty text.

[gotoolkits]

Your current environment

Using the latest main branch self-compiled LLM engine (v0.7.4.dev418+gd47807ba) to run the Gemma-3 model, only the Gemma-3-1b-it model outputs text content normally, while the 4/12/27B models output empty text.

### Running Environment
Ubuntu 20.04
Python 3.12
CUDA 12.0
CPU - 16 cores
Memory - 64GB

### Command executed:
vllm serve google/gemma-3-4b-it --tensor-parallel-size 2 --dtype=half --gpu-memory-utilization 0.9 --max-model-len=17370

🐛 Describe the bug

curl http://localhost:8000/v1/completions \
>     -H "Content-Type: application/json" \
>     -d '{
>         "model": "google/gemma-3-4b-it",
>         "prompt": "who are you?",
>         "stream": true
> }'
data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":null}

data: {"id":"cmpl-39c393fdbc284c8fbc31de7db7c50770","object":"text_completion","created":1741949943,"model":"google/gemma-3-4b-it","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":null}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

You should use Chat Completions API (not Completions API) as the instruction-tuned models expect a chat template.

[chadshao]

Me too, encountered the same problem and urgently need a solution.

[chadshao]

You should use Chat Completions API (not Completions API) as the instruction-tuned models expect a chat template.

How to operate specifically?

[DarkLight1337]

You can refer to the OpenAI API Reference for details

[gotoolkits]

Use： POST /v1/chat/completions

INFO:     Started server process [98698]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO 03-14 14:01:15 [chat_utils.py:345] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
INFO:     127.0.0.1:46632 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)

[DarkLight1337]

Can you show the request you posted?

[gotoolkits]

# vllm serve google/gemma-3-4b-it --tensor-parallel-size 2 --dtype=half --gpu-memory-utilization 0.9 --max-model-len=17370 --port 8001


(VllmWorkerProcess pid=4301) INFO 03-14 19:51:44 [model_runner.py:1442] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes:  97%|█████████████████████████████████████████████████████████████▏ | 34/35 [00:20<00:00,  1.74it/s](VllmWorkerProcess pid=4301) INFO 03-14 19:52:06 [custom_all_reduce.py:229] Registering 2415 cuda graph addresses
Capturing CUDA graph shapes: 100%|███████████████████████████████████████████████████████████████| 35/35 [00:22<00:00,  1.55it/s]
INFO 03-14 19:52:07 [custom_all_reduce.py:229] Registering 2415 cuda graph addresses
(VllmWorkerProcess pid=4301) INFO 03-14 19:52:07 [model_runner.py:1570] Graph capturing finished in 23 secs, took 1.38 GiB
INFO 03-14 19:52:07 [model_runner.py:1570] Graph capturing finished in 23 secs, took 1.38 GiB
INFO 03-14 19:52:07 [llm_engine.py:441] init engine (profile, create kv cache, warmup model) took 77.49 seconds
INFO 03-14 19:52:07 [api_server.py:958] Starting vLLM API server on http://0.0.0.0:8001
INFO 03-14 19:52:07 [launcher.py:26] Available routes are:
INFO 03-14 19:52:07 [launcher.py:34] Route: /openapi.json, Methods: HEAD, GET
INFO 03-14 19:52:07 [launcher.py:34] Route: /docs, Methods: HEAD, GET
INFO 03-14 19:52:07 [launcher.py:34] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 03-14 19:52:07 [launcher.py:34] Route: /redoc, Methods: HEAD, GET
INFO 03-14 19:52:07 [launcher.py:34] Route: /health, Methods: GET
INFO 03-14 19:52:07 [launcher.py:34] Route: /ping, Methods: POST, GET
INFO 03-14 19:52:07 [launcher.py:34] Route: /tokenize, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /detokenize, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /v1/models, Methods: GET
INFO 03-14 19:52:07 [launcher.py:34] Route: /version, Methods: GET
INFO 03-14 19:52:07 [launcher.py:34] Route: /v1/chat/completions, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /v1/completions, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /v1/embeddings, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /pooling, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /score, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /v1/score, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /v1/audio/transcriptions, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /rerank, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /v1/rerank, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /v2/rerank, Methods: POST
INFO 03-14 19:52:07 [launcher.py:34] Route: /invocations, Methods: POST
INFO:     Started server process [3856]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO 03-14 19:52:52 [chat_utils.py:345] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
INFO:     127.0.0.1:36494 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/vllm/vllm/entrypoints/utils.py", line 56, in wrapper
    return handler_task.result()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 410, in create_chat_completion
    generator = await handler.create_chat_completion(request, raw_request)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/vllm/vllm/entrypoints/openai/serving_chat.py", line 184, in create_chat_completion
    ) = await self._preprocess_chat(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/vllm/vllm/entrypoints/openai/serving_engine.py", line 411, in _preprocess_chat
    request_prompt = apply_hf_chat_template(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/vllm/vllm/entrypoints/chat_utils.py", line 1079, in apply_hf_chat_template
    return tokenizer.apply_chat_template(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1695, in apply_chat_template
    rendered_chat = compiled_template.render(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/jinja2/environment.py", line 1295, in render
    self.environment.handle_exception()
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/jinja2/environment.py", line 942, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 19, in top-level template code
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/jinja2/sandbox.py", line 401, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/transformers/utils/chat_template_utils.py", line 419, in raise_exception
    raise jinja2.exceptions.TemplateError(message)
jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...

# curl http://localhost:8001/v1/chat/completions \
>     -H "Content-Type: application/json" \
>     -d '{
>         "model": "google/gemma-3-4b-it",
>         "messages": [
>           {
>             "role": "developer",
>             "content": "You are a helpful assistant."
>           },
>           {
>             "role": "user",
>             "content": "1+1="
>           }
>         ],
>         "stream": true
> }'

[gotoolkits]

How to define a template file?
" --chat-template-content-format xxxx.jinja"

[DarkLight1337]

1. You should use system role, not developer
2. Gemma3 is trained on bfloat16, so changing the dtype may result in numeric instability. Try removing that argument so that bfloat16 is used automatically.

[DarkLight1337]

How to define a template file?
" --chat-template-content-format xxxx.jinja"

The model already defines a chat template so you don't have to do this.

[gotoolkits]

The system interface is set without errors, but the output content remains empty and cannot be stopped.

1. ould use system role, not developer

data: {"id":"chatcmpl-291406e98e764e7a8534dcfd22be7e5f","object":"chat.completion.chunk","created":1741954143,"model":"google/gemma-3-4b-it","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-291406e98e764e7a8534dcfd22be7e5f","object":"chat.completion.chunk","created":1741954143,"model":"google/gemma-3-4b-it","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-291406e98e764e7a8534dcfd22be7e5f","object":"chat.completion.chunk","created":1741954143,"model":"google/gemma-3-4b-it","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-291406e98e764e7a8534dcfd22be7e5f","object":"chat.completion.chunk","created":1741954143,"model":"google/gemma-3-4b-it","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-291406e98e764e7a8534dcfd22be7e5f","object":"chat.completion.chunk","created":1741954143,"model":"google/gemma-3-4b-it","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-291406e98e764e7a8534dcfd22be7e5f","object":"chat.completion.chunk","created":1741954143,"model":"google/gemma-3-4b-it","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-291406e98e764e7a8534dcfd22be7e5f","object":"chat.completion.chunk","created":1741954143,"model":"google/gemma-3-4b-it","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-291406e98e764e7a8534dcfd22be7e5f","object":"chat.completion.chunk","created":1741954143,"model":"google/gemma-3-4b-it","choices":[{"index":0,"delta":{"content":""},"logprobs":null,"finish_reason":null}]}

^C

[DarkLight1337]

You need to also fix the dtype problem as I mentioned

[gotoolkits]

# vllm serve google/gemma-3-4b-it --tensor-parallel-size 2 --gpu-memory-utilization 0
.9 --max-model-len=17370 --port 8001

ERROR 03-14 20:19:53 [engine.py:411] File "/home/ubuntu/vllm/vllm/worker/worker.py", line 526, in _check_if_gpu_supports_dtype
ERROR 03-14 20:19:53 [engine.py:411] raise ValueError(
ERROR 03-14 20:19:53 [engine.py:411] ValueError: Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your Tesla T4 GPU has compute capability 7.5. You can use float16 instead by explicitly setting the dtype flag in CLI, for example: --dtype=half.
Process SpawnProcess-1:
ERROR 03-14 20:19:53 [multiproc_worker_utils.py:124] Worker VllmWorkerProcess pid 17397 died, exit code: -15
INFO 03-14 20:19:53 [multiproc_worker_utils.py:128] Killing local vLLM worker processes
Traceback (most recent call last):
File "/root/miniconda3/envs/myenv/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/root/miniconda3/envs/myenv/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/vllm/vllm/engine/multiprocessing/engine.py", line 413, in run_mp_engine
raise e
File "/home/ubuntu/vllm/vllm/engine/multiprocessing/engine.py", line 402, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/engine/multiprocessing/engine.py", line 125, in from_engine_args
return cls(ipc_path=ipc_path,
^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/engine/multiprocessing/engine.py", line 77, in init
self.engine = LLMEngine(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/engine/llm_engine.py", line 274, in init
self.model_executor = executor_class(vllm_config=vllm_config, )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/executor/executor_base.py", line 271, in init
super().init(*args, **kwargs)
File "/home/ubuntu/vllm/vllm/executor/executor_base.py", line 52, in init
self._init_executor()
File "/home/ubuntu/vllm/vllm/executor/mp_distributed_executor.py", line 124, in _init_executor
self._run_workers("init_device")
File "/home/ubuntu/vllm/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
driver_worker_output = run_method(self.driver_worker, sent_method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/utils.py", line 2238, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/worker/worker_base.py", line 604, in init_device
self.worker.init_device() # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/worker/worker.py", line 157, in init_device
_check_if_gpu_supports_dtype(self.model_config.dtype)
File "/home/ubuntu/vllm/vllm/worker/worker.py", line 526, in _check_if_gpu_supports_dtype
raise ValueError(
ValueError: Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your Tesla T4 GPU has compute capability 7.5. You can use float16 instead by explicitly setting the dtype flag in CLI, for example: --dtype=half.
Traceback (most recent call last):
File "/root/miniconda3/envs/myenv/bin/vllm", line 8, in
sys.exit(main())
^^^^^^
File "/home/ubuntu/vllm/vllm/entrypoints/cli/main.py", line 75, in main
args.dispatch_function(args)
File "/home/ubuntu/vllm/vllm/entrypoints/cli/serve.py", line 33, in cmd
uvloop.run(run_server(args))
File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/uvloop/init.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/root/miniconda3/envs/myenv/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/myenv/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/root/miniconda3/envs/myenv/lib/python3.12/site-packages/uvloop/init.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 947, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/myenv/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 139, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/myenv/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/vllm/vllm/entrypoints/openai/api_server.py", line 233, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

[DarkLight1337]

Looks like your GPU is too old. As a last resort you can try using float32 but it will increase memory usage