Skip to content

Add platform api for request validation to reject requests that don't fit warmup shapes #77

@joerunde

Description

@joerunde

From talking to @njhill, there's already precedent for doing hardware-specific request validation with the supports_structured_output platform api. For spyre we need a way to reject requests that don't fit a warmup shape, so we should propose a more general api to validate requests up front. Maybe something like:

def validate_request(
        cls,
        prompt: PromptType,
        params: Union[SamplingParams, PoolingParams],
        lora_request: Optional[LoRARequest] = None,
) -> None:
        """Raises if this request is unsupported on this platform"""

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions