[CLI] Use streaming in CLI chat and completion commands

[simon-mo]

Summary

• stream token outputs in vllm chat CLI
• stream token outputs in vllm complete CLI
• factor out streaming loops into reusable helper functions

Testing

• ruff check vllm/entrypoints/cli/openai.py
• python -m py_compile vllm/entrypoints/cli/openai.py
• pre-commit run --files vllm/entrypoints/cli/openai.py (fails: command not found)
• pytest tests/v1/entrypoints/openai/test_completion.py::test_completion_streaming -q (fails: ModuleNotFoundError: torch)

---

https://chatgpt.com/codex/tasks/task_e_68af54ff5ee88329b50c13bf46c0da0d

[gemini-code-assist]

Code Review

This pull request introduces streaming support for the chat and complete CLI commands, which is a great enhancement for user experience. The implementation is straightforward, using helper functions to handle the streaming logic. My review includes suggestions to improve robustness by adding error handling around the streaming API calls and to enhance code clarity by adding type hints to the new helper functions. These changes will make the CLI tool more resilient and the code easier to maintain.

[gemini-code-assist]

To improve type safety and code readability, please add a type hint for the stream parameter. The openai client returns a Stream of ChatCompletionChunk objects. Using a string forward reference for the type hint is a good practice here.

You'll need to ensure the necessary types are imported within a TYPE_CHECKING block:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from openai import Stream
    from openai.types.chat import ChatCompletionChunk

Suggested change

	def _print_chat_stream(stream) -> str:
	def _print_chat_stream(stream: "Stream[ChatCompletionChunk]") -> str:

[chaunceyjiang]

+1

[gemini-code-assist]

For consistency and to improve type safety, please add a type hint for the stream parameter. The openai client returns a Stream of Completion objects for completion requests.

You'll need to ensure the necessary types are imported within a TYPE_CHECKING block:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from openai import Stream
    from openai.types import Completion

Suggested change

	def _print_completion_stream(stream) -> str:
	def _print_completion_stream(stream: "Stream[Completion]") -> str:

[gemini-code-assist]

The streaming API call can raise exceptions (e.g., openai.APIError) if an issue occurs during generation. To prevent the CLI from crashing and to provide a better user experience, it's best to wrap the streaming logic in a try...except block to gracefully handle any potential errors.

        try:
            stream = client.chat.completions.create(
                model=model_name, messages=conversation, stream=True)
            output = _print_chat_stream(stream)
            conversation.append({"role": "assistant", "content": output})
        except Exception as e:
            print(f"\nAn error occurred: {e}")

[gemini-code-assist]

The streaming API call can raise exceptions. To make the CLI more robust, please wrap this call in a try...except block to handle potential errors gracefully.

            try:
                stream = client.chat.completions.create(
                    model=model_name, messages=conversation, stream=True)
                output = _print_chat_stream(stream)
                conversation.append({"role": "assistant", "content": output})
            except Exception as e:
                print(f"\nAn error occurred: {e}")

[gemini-code-assist]

The streaming API call can raise exceptions. To make the CLI more robust, please wrap this call in a try...except block to handle potential errors gracefully.

            try:
                stream = client.chat.completions.create(
                    model=model_name, messages=conversation, stream=True)
                output = _print_chat_stream(stream)
                conversation.append({"role": "assistant", "content": output})
            except Exception as e:
                print(f"\nAn error occurred: {e}")

[gemini-code-assist]

The streaming API call can raise exceptions. To make the CLI more robust, please wrap this call in a try...except block to handle potential errors gracefully.

            try:
                stream = client.completions.create(model=model_name,
                                                   prompt=args.quick,
                                                   stream=True)
                _print_completion_stream(stream)
            except Exception as e:
                print(f"\nAn error occurred: {e}")

[gemini-code-assist]

The streaming API call can raise exceptions. To make the CLI more robust, please wrap this call in a try...except block to handle potential errors gracefully.

            try:
                stream = client.completions.create(model=model_name,
                                                   prompt=input_prompt,
                                                   stream=True)
                _print_completion_stream(stream)
            except Exception as e:
                print(f"\nAn error occurred: {e}")

[simon-mo]

@chaunceyjiang sorry just saw the comments. i was mostly looking at the CLI file don't really have much type hint and is designed to be simple demo code.