Unlock VideoUrl for vLLM Models

[brianalexander]

Description

https://ai.pydantic.dev/api/models/openai/#pydantic_ai.models.openai.OpenAIModel

                elif isinstance(item, VideoUrl):  # pragma: no cover
                    raise NotImplementedError('VideoUrl is not supported for OpenAI.')

I cannot use video reasoning with vLLM because of this.

Using a custom override for now

elif isinstance(item, VideoUrl):  # Updated to handle VideoUrl
                    # First, check if it's a local file path or a URL
                    import os
                    from urllib.parse import urlparse
                    
                    parsed = urlparse(item.url)
                    if parsed.scheme in ('http', 'https', 'file', ''):  # URL or local path
                        actual_file_path = item.url
                        if parsed.scheme == 'file':
                            actual_file_path = parsed.path
                        elif parsed.scheme in ('', 'file') and os.path.exists(item.url):
                            actual_file_path = item.url
                            
                        if os.path.exists(actual_file_path):
                            # It's a local file, read and encode as base64
                            with open(actual_file_path, 'rb') as video_file:
                                video_base64 = base64.b64encode(video_file.read()).decode('utf-8')
                            
                            # Determine the video format from the file extension
                            _, ext = os.path.splitext(actual_file_path)
                            mime_type = f'video/{ext[1:]}' if ext else 'video/mp4'
                            
                            # Create a data URI with the video content
                            video_data_uri = f'data:{mime_type};base64,{video_base64}'
                            
                            # Add to content as video_url part (the correct format for video)
                            content.append({
                                'type': 'video_url',
                                'video_url': {'url': video_data_uri}
                            })
                        else:
                            # It's a URL, try to download and handle it
                            downloaded_item = await download_item(item, data_format='base64_uri', type_format='extension')
                            content.append({
                                'type': 'video_url',
                                'video_url': {'url': item.url}
                            })
                    else:
                        # It's a URL, try to download and handle it
                        downloaded_item = await download_item(item, data_format='base64_uri', type_format='extension')
                        content.append({
                            'type': 'video_url',
                            'video_url': {'url': item.url}
                        })
                else:
                    assert_never(item)

vLLM command

python -m vllm.entrypoints.openai.api_server   --model cpatonn/Qwen3-VL-8B-Instruct-AWQ-4bit   --trust-remote-code   --max-model-len 16384

Relevant links:

• https://github.com/vllm-project/vllm/blob/main/examples/online_serving/openai_chat_completion_client_for_multimodal.py
• https://ai.pydantic.dev/api/models/openai/#pydantic_ai.models.openai.OpenAIResponsesModel
• https://ai.pydantic.dev/input/#user-side-download-vs-direct-file-url

Thanks!

References

No response

[DouweM]

@brianalexander Thanks for the report; this also just came up here: #3121 (comment)

Let me copy in what I responded there:

---

[Since this is a vLLM-only feature so far, it] should be a VLLMModel subclass of OpenAIChatModel (or OpenAIResponsesModel?), similar to the OpenRouterModel being introduced in #3089. That PR will likely introduce some hooks that make it easy to override just how a specific part like VideoUrl is handled. (cc @ajac-zero)

It would be good to introduce a VLLMProvider at the same time that can set specific model profile settings as I suggested in #3270 (comment):

So between this and openai_responses_requires_function_call_status_none (#3245), I think we need a new VLLMProvider that'll use harmony_model_profile and then always set openai_responses_requires_function_call_status_none=True, so that new users can just use that provider and won't have to figure out these 2 profile settings themselves.

---

In that issue @mtblanton indicated he may submit a PR to this effect, but if you @brianalexander can beat him to it you're more than welcome to :)

[mtblanton]

In that issue @mtblanton indicated he may submit a PR to this effect, but if you @brianalexander can beat him to it you're more than welcome to :)

We haven't started on this yet, so no worries about duplicating effort!