Unsupported binary content type: video/mp4

[ddgszc]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

mp4 video content is not supported

this is my code:

from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
from pydantic_ai import Agent,BinaryContent

def get_model():
    model = OpenAIChatModel(
        model_name="xxx",
        provider=OpenAIProvider(
            api_key="xxx",
            base_url="xxx",
        ),
    )
    return model
with open("./test.mp4", "rb") as f:
    image_data = f.read()

agent=Agent(
    model=get_model(),
    system_prompt="You are a helpful assistant.",
)

result=agent.run_sync(
    [
        "Tell me the content of the video",
        BinaryContent(data=image_data, media_type="video/mp4"),
    ]
)

print(result.output)

Error info:

Traceback (most recent call last):
  File "/home/liuy/test/main.py", line 22, in <module>
    result=agent.run_sync(
           ^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/agent/abstract.py", line 328, in run_sync
    return get_event_loop().run_until_complete(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/agent/abstract.py", line 229, in run
    async for node in agent_run:
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/run.py", line 150, in __anext__
    next_node = await self._graph_run.__anext__()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_graph/graph.py", line 758, in __anext__
    return await self.next(self._next_node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_graph/graph.py", line 731, in next
    self._next_node = await node.run(ctx)
                      ^^^^^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/_agent_graph.py", line 397, in run
    return await self._make_request(ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/_agent_graph.py", line 439, in _make_request
    model_response = await ctx.deps.model.request(message_history, model_settings, model_request_parameters)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 402, in request
    response = await self._completions_create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 465, in _completions_create
    openai_messages = await self._map_messages(messages)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 664, in _map_messages
    async for item in self._map_user_message(message):
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 747, in _map_user_message
    yield await self._map_user_prompt(part)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuy/test/.venv/lib/python3.12/site-packages/pydantic_ai/models/openai.py", line 813, in _map_user_prompt
    raise RuntimeError(f'Unsupported binary content type: {item.media_type}')
RuntimeError: Unsupported binary content type: video/mp4

Example Code

Python, Pydantic AI & LLM client version

requires-python = ">=3.12"
dependencies = [
    "pydantic-ai>=1.0.16",
]

[DouweM]

@ddgszc As far as I know, OpenAI does not support video input. Their cookbook recommends splitting the image up into frames manually, and then including them as images: https://cookbook.openai.com/examples/gpt_with_vision_for_video_understanding.

If you have an example of sending video data directly to the OpenAI SDK/API, let me know, and we can support it.

[mtblanton]

@DouweM my company wants to use a model hosted through vLLM's online serving that accepts video input. vLLM provides openai-compatible endpoints but accepts more types of input, including a video_url. https://docs.vllm.ai/en/v0.10.2/features/multimodal_inputs.html#video-inputs_1

example input:

        "content": [
            {
                "type": "text",
                "text": "What's in this video?"
            },
            {
                "type": "video_url",
                "video_url": {
                    "url": "<video_url>"
                },
                "uuid": "<video_url_uuid>"
            },
        ],

We were planning on making a custom model for this, but I was wondering if there would be any interest in getting that functionality into the main package. If so, what would be the preferred way to implement that?

[DouweM]

@mtblanton We'd definitely welcome support for that! It should be a VLLMModel subclass of OpenAIChatModel (or OpenAIResponsesModel?), similar to the OpenRouterModel being introduced in #3089. That PR will likely introduce some hooks that make it easy to override just how a specific part like VideoUrl is handled. (cc @ajac-zero)

It would be good to introduce a VLLMProvider at the same time that can set specific model profile settings as I suggested in #3270 (comment):

So between this and openai_responses_requires_function_call_status_none (#3245), I think we need a new VLLMProvider that'll use harmony_model_profile and then always set openai_responses_requires_function_call_status_none=True, so that new users can just use that provider and won't have to figure out these 2 profile settings themselves.