[serve][llm] Unify and Extend Builder Configuration for LLM Deployments

[kouroshHakha]

Unify and Extend Builder Configuration for LLM Deployments

Overview

This PR transforms Ray Serve LLM builders by providing:

1. Unified configuration experience across standard LLM and Prefill/Decode deployments
2. Extensible plugin points for customizing components without writing custom builders

Previously, users who needed to customize ingress behavior or P/D proxy logic had to fork and maintain their own builder functions. Now, the builder configuration supports pluggable components out of the box.

Problem Statement

Inconsistent Configuration Patterns

Standard LLM and P/D builders used different configuration approaches, forcing users to learn multiple patterns.

Limited Extensibility

Users couldn't easily customize core components:

• Want a custom ingress with authentication? → Write your own builder
• Need modified P/D proxy behavior? → Copy and maintain builder code
• Want different routing logic? → Fork the builder setup

Solution

1. Unified Configuration Pattern

Both builders now share the same intuitive structure:

# Standard LLM Builder
build_openai_app({
    "llm_configs": [llm_config],
    "ingress_cls_config": {...},         # Configure ingress
    "ingress_deployment_config": {...}    # Configure deployment
})

# P/D Builder - Same patterns!
build_pd_openai_app({
    "prefill_config": prefill_config,
    "decode_config": decode_config,
    "proxy_cls_config": {...},            # Configure proxy (P/D-specific)
    "proxy_deployment_config": {...},     # Configure deployment
    "ingress_cls_config": {...},          # Configure ingress (same as above!)
    "ingress_deployment_config": {...}    # Configure deployment (same as above!)
})

2. Pluggable Component System

Inject custom components without custom builders:

# Use custom ingress with authentication
build_openai_app({
    "llm_configs": [llm_config],
    "ingress_cls_config": {
        "ingress_cls": "mycompany.auth.AuthenticatedIngress",
        "ingress_extra_kwargs": {
            "auth_provider": "okta",
            "required_scopes": ["llm:read"]
        }
    }
})

# Use custom P/D proxy with logging
build_pd_openai_app({
    "prefill_config": p_config,
    "decode_config": d_config,
    "proxy_cls_config": {
        "proxy_cls": "mycompany.observability.LoggingPDProxy",
        "proxy_extra_kwargs": {
            "log_level": "DEBUG",
            "trace_all_requests": True
        }
    }
})

Key Features

Component Extensibility

Users can now customize any component:

# Example: Custom ingress with rate limiting
"ingress_cls_config": {
    "ingress_cls": "my.custom.RateLimitedIngress",  # Your class
    "ingress_extra_kwargs": {                       # Your params
        "rate_limit": 100,
        "burst_size": 10
    }
}

# Example: Custom P/D proxy with caching
"proxy_cls_config": {
    "proxy_cls": "my.custom.CachingPDProxy",
    "proxy_extra_kwargs": {
        "cache_backend": "redis",
        "ttl_seconds": 300
    }
}

No custom builder needed! Just implement your component class with the expected signature.

Flexible Input Types

Accept dicts, objects, or file paths everywhere:

# Mix and match as needed
"llm_configs": [
    dict,                    # Inline dict
    LLMConfig(...),         # Pydantic object
    "configs/model.yaml"    # File path
]

"ingress_cls_config": {...} or IngressClsConfig(...)  # Dict or object

Sensible Defaults with Override Ability

# Minimal config - uses OpenAiIngress and PDProxyServer
build_pd_openai_app({
    "prefill_config": p,
    "decode_config": d
})

# Override only what you need
build_pd_openai_app({
    "prefill_config": p,
    "decode_config": d,
    "proxy_cls_config": {
        "proxy_cls": "my.custom.Proxy"  # Just change the class
        # proxy_extra_kwargs defaults to {}
    }
    # ingress_cls_config defaults to OpenAiIngress
})

Type-Safe with String Paths

Load custom classes from strings for YAML/config file compatibility:

# In Python code
"ingress_cls": MyCustomIngress

# In YAML config
ingress_cls: "mymodule.components:MyCustomIngress"  # Colon or dot notation

Use Cases Enabled

Before: Custom Builder Required

# User had to copy and modify the entire builder
def my_custom_pd_builder(prefill, decode):
    # 50+ lines of boilerplate copied from Ray
    prefill_deployment = build_llm_deployment(prefill)
    decode_deployment = build_llm_deployment(decode)
    
    # The one line they actually wanted to customize:
    proxy = MyCustomProxy(prefill_deployment, decode_deployment, my_custom_arg=True)
    
    # More boilerplate...
    ingress = serve.deployment(OpenAiIngress).bind([proxy])
    return ingress

After: Configuration Only

# Just configure it!
app = build_pd_openai_app({
    "prefill_config": prefill,
    "decode_config": decode,
    "proxy_cls_config": {
        "proxy_cls": MyCustomProxy,
        "proxy_extra_kwargs": {"my_custom_arg": True}
    }
})

Benefits

• No more custom builders for simple customizations
• Consistent patterns across deployment types
• Easy component swapping without code changes (e.g., via YAML configs)
• Type-safe configuration with validation

Testing

Release tests: https://buildkite.com/ray-project/release/builds/63841 ✅

Future work

• One thing we should do in a follow up PR is to make PDProxyServer inherit LLMServerProtocol than LLMServer. It will become confusing over time when developers look at PDProxyServer's implementation and use that as the example of how to implement the rest of the stuff.

[kouroshHakha]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a significant and well-executed revamp of the Prefill/Decode (P/D) and standard LLM deployment APIs. The move to a unified, Pydantic-based configuration model is a major improvement, making the APIs more intuitive, type-safe, and extensible. The ability to use pluggable components for ingress and proxy via configuration is a fantastic feature that will greatly simplify customizations for users. The code is cleaner and more maintainable as a result of this refactoring.

My review includes a couple of points for consideration. One is a behavioral change regarding the kv_transfer_config in P/D deployments, which is now mandatory. The other is a minor regression in parsing LLM configurations from raw YAML strings.

Overall, this is an excellent contribution that significantly enhances the usability and flexibility of Ray Serve's LLM capabilities.

[ruisearch42]

LGTM, just some nitpicks

[ruisearch42]

update this as well?

pd_serving_args: A dict that conforms to the PDServingArgs pydantic model.

[ruisearch42]

nit: The proxy class or class name

[ruisearch42]

nit: TypeError?

[ruisearch42]

can do this?

    @field_validator("prefill_config", "decode_config")
    @classmethod
    def _validate_llm_configs(cls, value: Any) -> LLMConfig:
        return cls._validate_llm_config(value)

[ruisearch42]

TypeError?

[kouroshHakha]

release tests failing here: https://buildkite.com/ray-project/release/builds/63841 ❌