[Serve.llm][Prototype][WIP] Simplify LLMServer and inherit OpenAIServingChat behavior

[kouroshHakha]

Addresses #53533

Made ready for review for copilot review not ready for human review yet.

Need to merge vllm changes as well:

• [frontend] Refactor CLI Args for a better modular integration vllm-project/vllm#20206
• [tech debt] Revisit lora request model checker vllm-project/vllm#20636

[Copilot]

Pull Request Overview

This PR refactors the Serve LLM stack to simplify the LLMServer by inheriting directly from vLLM’s OpenAI‐serving components, removes legacy processing layers, and unifies chat, completion, and embedding flows.

• Consolidates engine APIs to use vLLM’s OpenAIServingChat, Completion, and Embedding endpoints
• Refactors and shrinks LLMServer, removing ResponsePostprocessor and redundant code
• Updates mocks and tests to align with new request/response types and metadata fields

Reviewed Changes

Copilot reviewed 35 out of 36 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
release/llm_tests/serve/probes/test_basic.py	Adjusted max‐token length and updated invalid logprobs test cases
python/ray/llm/tests/serve/mocks/mock_vllm_engine.py	Overhauled mock engine to yield SSE strings and OpenAI‐style responses
python/ray/llm/_internal/serve/deployments/llm/llm_server.py	Simplified server logic; unified request→engine→batching pipeline
python/ray/llm/_internal/serve/deployments/llm/vllm/vllm_engine.py	Integrated vLLM’s OpenAIServingChat/Completion/Embedding APIs
python/ray/llm/tests/serve/utils/testing_utils.py	Added reusable response‐validation helpers

Comments suppressed due to low confidence (4)

release/llm_tests/serve/probes/test_basic.py:319

• The upper‐bound case for top_logprobs (> maximum allowed) was removed. To fully validate error handling, reintroduce a test value above the allowed maximum.

    invalid_num_logprobs = [-1]

python/ray/llm/tests/serve/mocks/mock_vllm_engine.py:5

• The mock implementation uses await asyncio.sleep(...) but asyncio is not imported. Add import asyncio at the top.

from typing import AsyncGenerator, Dict, Optional, Any, List, Union

python/ray/llm/tests/serve/conftest.py:17

• Fixture imports EmbeddingCompletionRequest, but the actual class is named EmbeddingRequest. Update the import and fixture accordingly.

from ray.llm._internal.serve.configs.openai_api_models import (

release/llm_tests/serve/probes/test_basic.py:163

• [nitpick] Using a raw literal 200000 can obscure its meaning. Consider replacing it with a named constant or adding a comment explaining its origin.

    length = 200000

[kouroshHakha]

Here is the tentative plan to break these changes up into smaller PRs:

1. vllm disable v0 completely

• make the default v1 [Serve.llm] Make llm serve endpoints compatible with vLLM serve frontend (1/N): Remove random v0 logic #54440
• make engine constructor take llm_config as input
• vllm_models.py
  • changes to def get_initialization_kwargs(self) [Serve.llm] Make llm serve endpoints compatible with vLLM serve frontend (2/N): Incremental refactor of LLMEngine  #54443

2. Upstream changes on node init code path [Serve.llm] Make llm serve endpoints compatible with vLLM serve frontend (3/N): Remove indirection layers of node initialization #54481

• serve_models
  • change to _model_architecture: str = PrivateAttr("UNSPECIFIED")
  • change to if hasattr(hf_config, "architectures") and hf_config.architectures:
• llm_server
  • changes to if self._llm_config.model_architecture:

3. Clean up llm_server abstraction [Serve.llm] Make llm serve endpoints compatible with vLLM serve frontend (4/N): Refactor LLMServer #54484

• remove llm_config from mandator init on server base
• define _init_multiplex_loader
• define _maybe_add_request_id_to_request
• define _maybe_resolve_lora_from_multiplex(request)
• define _batch_output_stream

4. Change the engine abstraction and llm server usage of that abstraction

• In here we should have everything working end to end
• Use vLLM OpenAI models instead of vendording our own copy

5. (optional) vllm 0.10.0 bump (to get the FrontendArgs and lora model checker changes)