Missing Tokens from Beginning of any Streaming text content after 0.4.*

[newsbubbles]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

Meeting Notes from Troubleshooting Session

Meeting Purpose

Investigate and resolve streaming response issues with Kimi K2 and Claude models on staging environment.

Key Takeaways

• The issue of missing initial tokens in streaming responses was caused by upgrading Pydantic AI from v0.3.x to v0.4.x, not by Kimi K2 or Claude models as initially suspected
• Downgrading Pydantic AI to v0.3.7 resolved the issue
• The team will pin Pydantic AI to v0.3.7 temporarily until the problem is fixed in newer versions
• Detailed notes will be shared with Pydantic and Open Router communities to report the bug

Notes

Initial Problem Description

• Streaming responses on staging were missing initial tokens/words
• Issue wasn't present on production environment
• Initially suspected to be related to Kimi K2 model or Open Router
• The final message includes the missing tokens, while streaming does not

Debugging Process

• Compared pip packages between staging and production environments
• Found Pydantic AI version difference: staging on 0.4.9, production on 0.2.4
• Downgraded Pydantic AI on staging to 0.2.4, which resolved the issue
• Tested with Claude model, confirming the problem wasn't model-specific

Root Cause Identification

• Issue started occurring when Pydantic AI was upgraded from 0.3.x to 0.4.x
• Coincidentally upgraded on the same day as switching to Kimi K2, leading to initial misattribution
• Tested Pydantic AI v0.3.7, which worked correctly without missing tokens

Open Router Provider Investigation

• Briefly explored using Open Router provider instead of OpenAI provider
• Ultimately determined not to be the source of the problem (didn't affect/fix the bug)

GitHub Issue Search

• Searched Pydantic AI GitHub for related issues
• Found some mentions of streaming issues, but nothing exactly matching the current problem

Version Testing

• Tested Pydantic AI v0.5.0, issue still present
• Confirmed v0.3.7 works correctly

Our Individual Solution

We have no choice but to pin our environments to pydantic-ai 0.3.7 until this issue is resolved.

Example Code

Python, Pydantic AI & LLM client version

pydantic-ai 0.4.7 (initially on staging) ... 0.5.0 (tested, not working)... 0.3.7 (tested, working)... 0.2.4 (on production)

Any LLM, doesn't matter if it's kimi-k2, claude models, openai models, or even the new z.ai GLM 4.5, they all show the same problem which led us to believe initially it was a pydantic-ai issue, as openrouter only makes the route and this spans across multiple providers.

[majdalsado]

Seeing similar, just downgraded back to 0.35 to avoid this.

[DouweM]

@newsbubbles Thanks for the report, I've been doing some work on streaming lately but it looks like our (extensive) test coverage was not sufficient in this case.

This was likely introduced by one of these changes I made in v0.4.7:

• Ignore leading whitespace when streaming text, fixing run_stream + Ollama + Qwen3 #2294
• Ignore empty text alongside tool calls when streaming from Ollama #2286
• Parse '<think>' tags in streamed text as thinking parts #2290

Can you please confirm that v0.4.6 works fine and the issue starts with v0.4.7?

The first one sounds like the most likely culprit, but that should only affect blank (space, newline) initial tokens. Is that the issue you're seeing?

Either way a reproducible example (that behaves differently across 2 versions) would be much appreciated, there's not much we can do without one as all our streaming tests are happy so we're clearly missing some edge case.

[taaha]

I created a simple experiment
simple_mcp_server.py

from datetime import datetime
from mcp.server.fastmcp import FastMCP

mcp = FastMCP(name="Tool Example")


@mcp.tool()
def sum(a: int, b: int) -> int:
    """Add two numbers together."""
    return a + b


@mcp.tool()
def get_weather(city: str, unit: str = "celsius") -> str:
    """Get weather for a city."""
    # This would normally call a weather API
    return f"Weather in {city}: 22degrees{unit[0].upper()}"


@mcp.tool()
def get_current_time() -> str:
    """Get the current time."""
    return datetime.now().strftime("%Y-%m-%d %H:%M:%S")


def main():
    mcp.run()


if __name__ == "__main__":
    main()

and another file
streaming_test.py

# PydanticAI Agent with MCP

from pydantic_ai import Agent, RunContext
from pydantic_ai.mcp import MCPServerStdio
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

from dotenv import load_dotenv
import os

# import logfire

import asyncio
from dataclasses import dataclass
from datetime import date

from pydantic_ai import Agent
from pydantic_ai.messages import (
    FinalResultEvent,
    FunctionToolCallEvent,
    FunctionToolResultEvent,
    PartDeltaEvent,
    PartStartEvent,
    TextPartDelta,
    ToolCallPartDelta,
)
from pydantic_ai.tools import RunContext

load_dotenv()

# Set up OpenRouter based model
API_KEY = os.getenv("OPENROUTER_API_KEY")
model = OpenAIModel(
    "z-ai/glm-4.5",
    provider=OpenAIProvider(base_url="https://openrouter.ai/api/v1", api_key=API_KEY),
)


mcp_servers = [MCPServerStdio("python", ["simple_mcp_server.py"])]

# Set up Agent with Server
agent = Agent(
    model,
    mcp_servers=mcp_servers,
    system_prompt="""
You are a helpful assistant.
""",
)


async def main():
    async with agent.run_mcp_servers():
        output_messages: list[str] = []
        user_prompt = "What is the current time and weather in Tokyo?"
        async with agent.iter(user_prompt) as run:
            async for node in run:
                if Agent.is_user_prompt_node(node):
                    # A user prompt node => The user has provided input
                    print(f"=== UserPromptNode: {node.user_prompt} ===")
                    output_messages.append(
                        f"=== UserPromptNode: {node.user_prompt} ==="
                    )
                elif Agent.is_model_request_node(node):
                    # A model request node => We can stream tokens from the model's request
                    output_messages.append(
                        "=== ModelRequestNode: streaming partial request tokens ==="
                    )
                    print("=== ModelRequestNode: streaming partial request tokens ===")
                    async with node.stream(run.ctx) as request_stream:
                        async for event in request_stream:
                            if isinstance(event, PartStartEvent):
                                output_messages.append(
                                    f"[Request] Starting part {event.index}: {event.part!r}"
                                )
                                print(
                                    f"[Request] Starting part {event.index}: {event.part!r}"
                                )
                            elif isinstance(event, PartDeltaEvent):
                                if isinstance(event.delta, TextPartDelta):
                                    output_messages.append(
                                        f"[Request] Part {event.index} text delta: {event.delta.content_delta!r}"
                                    )
                                    print(
                                        f"[Request] Part {event.index} text delta: {event.delta.content_delta!r}"
                                    )
                                elif isinstance(event.delta, ToolCallPartDelta):
                                    output_messages.append(
                                        f"[Request] Part {event.index} args_delta={event.delta.args_delta}"
                                    )
                                    print(
                                        f"[Request] Part {event.index} args_delta={event.delta.args_delta}"
                                    )
                            elif isinstance(event, FinalResultEvent):
                                output_messages.append(
                                    f"[Result] The model produced a final result (tool_name={event.tool_name})"
                                )
                                print(
                                    f"[Result] The model produced a final result (tool_name={event.tool_name})"
                                )
                elif Agent.is_call_tools_node(node):
                    # A handle-response node => The model returned some data, potentially calls a tool
                    output_messages.append(
                        "=== CallToolsNode: streaming partial response & tool usage ==="
                    )
                    print(
                        "=== CallToolsNode: streaming partial response & tool usage ==="
                    )
                    async with node.stream(run.ctx) as handle_stream:
                        async for event in handle_stream:
                            if isinstance(event, FunctionToolCallEvent):
                                output_messages.append(
                                    f"[Tools] The LLM calls tool={event.part.tool_name!r} with args={event.part.args} (tool_call_id={event.part.tool_call_id!r})"
                                )
                                print(
                                    f"[Tools] The LLM calls tool={event.part.tool_name!r} with args={event.part.args} (tool_call_id={event.part.tool_call_id!r})"
                                )
                            elif isinstance(event, FunctionToolResultEvent):
                                output_messages.append(
                                    f"[Tools] Tool call {event.tool_call_id!r} returned => {event.result.content}"
                                )
                                print(
                                    f"[Tools] Tool call {event.tool_call_id!r} returned => {event.result.content}"
                                )
                elif Agent.is_end_node(node):
                    assert run.result.output == node.data.output
                    # Once an End node is reached, the agent run is complete
                    output_messages.append(
                        f"=== Final Agent Output: {run.result.output} ==="
                    )
                    print(f"=== Final Agent Output: {run.result.output} ===")


if __name__ == "__main__":
    import asyncio

    asyncio.run(main())

The console output for pydantic-ai==0.4.6

[08/07/25 16:36:25] INFO     Processing request of type ListToolsRequest                                                                                                                                                         server.py:625
=== UserPromptNode: What is the current time and weather in Tokyo? ===
=== ModelRequestNode: streaming partial request tokens ===
                    INFO     Processing request of type ListToolsRequest                                                                                                                                                         server.py:625
[Request] Starting part 0: TextPart(content='\n')
[Result] The model produced a final result (tool_name=None)
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: '\nI'
[Request] Part 0 text delta: "'ll get"
[Request] Part 0 text delta: ' the current'
[Request] Part 0 text delta: ' time and'
[Request] Part 0 text delta: ' weather in'
[Request] Part 0 text delta: ' Tokyo for'
[Request] Part 0 text delta: ' you.\n'
[Request] Starting part 1: ToolCallPart(tool_name='get_current_time', args='{}', tool_call_id='call_c06488fb347f4b80912fd327')
[Request] Part 0 text delta: '\n'
[Request] Starting part 2: ToolCallPart(tool_name='get_weather', args='{"city": "Tokyo"}', tool_call_id='call_5a71c72aa15d44e992fb6fa3')
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
=== CallToolsNode: streaming partial response & tool usage ===
[Tools] The LLM calls tool='get_current_time' with args={} (tool_call_id='call_c06488fb347f4b80912fd327')
[Tools] The LLM calls tool='get_weather' with args={"city": "Tokyo"} (tool_call_id='call_5a71c72aa15d44e992fb6fa3')
[08/07/25 16:36:27] INFO     Processing request of type CallToolRequest                                                                                                                                                          server.py:625
                    INFO     Processing request of type CallToolRequest                                                                                                                                                          server.py:625
[Tools] Tool call 'call_c06488fb347f4b80912fd327' returned => 2025-08-07 16:36:27
[Tools] Tool call 'call_5a71c72aa15d44e992fb6fa3' returned => Weather in Tokyo: 22degreesC
=== ModelRequestNode: streaming partial request tokens ===
                    INFO     Processing request of type ListToolsRequest                                                                                                                                                         server.py:625
[Request] Starting part 0: TextPart(content='\n')
[Result] The model produced a final result (tool_name=None)
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: '\nThe'
[Request] Part 0 text delta: ' current time'
[Request] Part 0 text delta: ' is **'
[Request] Part 0 text delta: '16'
[Request] Part 0 text delta: ':36'
[Request] Part 0 text delta: ':27'
[Request] Part 0 text delta: '**'
[Request] Part 0 text delta: ' on August'
[Request] Part 0 text delta: ' 7'
[Request] Part 0 text delta: ', '
[Request] Part 0 text delta: '2025'
[Request] Part 0 text delta: '.\n\n'
[Request] Part 0 text delta: 'The weather'
[Request] Part 0 text delta: ' in Tokyo'
[Request] Part 0 text delta: ' is **'
[Request] Part 0 text delta: '22°C'
[Request] Part 0 text delta: '** ('
[Request] Part 0 text delta: '22 degrees'
[Request] Part 0 text delta: ' Celsius).'
[Request] Part 0 text delta: ''
[Request] Part 0 text delta: ''
=== CallToolsNode: streaming partial response & tool usage ===
=== Final Agent Output: 

The current time is **16:36:27** on August 7, 2025.

The weather in Tokyo is **22°C** (22 degrees Celsius). ===

The console output for pydantic_ai==0.4.7

[08/07/25 16:35:59] INFO     Processing request of type ListToolsRequest                                                                                                                                                         server.py:625
=== UserPromptNode: What is the current time and weather in Tokyo? ===
=== ModelRequestNode: streaming partial request tokens ===
                    INFO     Processing request of type ListToolsRequest                                                                                                                                                         server.py:625
[Request] Starting part 0: TextPart(content="I'll")
[Result] The model produced a final result (tool_name=None)
[Request] Part 0 text delta: ' get the'
[Request] Part 0 text delta: ' current time'
[Request] Part 0 text delta: ' and weather'
[Request] Part 0 text delta: ' in'
[Request] Part 0 text delta: ' Tokyo for'
[Request] Part 0 text delta: ' you.\n'
[Request] Starting part 1: ToolCallPart(tool_name='get_current_time', args='{}', tool_call_id='call_64e7f1529d96498da9c1bfa8')
[Request] Part 0 text delta: '\n'
[Request] Starting part 2: ToolCallPart(tool_name='get_weather', args='{"city": "Tokyo"}', tool_call_id='call_28fb8d9bce2741c59ce1871b')
=== CallToolsNode: streaming partial response & tool usage ===
[Tools] The LLM calls tool='get_current_time' with args={} (tool_call_id='call_64e7f1529d96498da9c1bfa8')
[Tools] The LLM calls tool='get_weather' with args={"city": "Tokyo"} (tool_call_id='call_28fb8d9bce2741c59ce1871b')
[08/07/25 16:36:02] INFO     Processing request of type CallToolRequest                                                                                                                                                          server.py:625
                    INFO     Processing request of type CallToolRequest                                                                                                                                                          server.py:625
[Tools] Tool call 'call_64e7f1529d96498da9c1bfa8' returned => 2025-08-07 16:36:02
[Tools] Tool call 'call_28fb8d9bce2741c59ce1871b' returned => Weather in Tokyo: 22degreesC
=== ModelRequestNode: streaming partial request tokens ===
                    INFO     Processing request of type ListToolsRequest                                                                                                                                                         server.py:625
[Request] Starting part 0: TextPart(content='\nHere')
[Result] The model produced a final result (tool_name=None)
[Request] Part 0 text delta: "'s the"
[Request] Part 0 text delta: ' current information'
[Request] Part 0 text delta: ' for'
[Request] Part 0 text delta: ' Tokyo:\n\n'
[Request] Part 0 text delta: '**Current'
[Request] Part 0 text delta: ' Time:**'
[Request] Part 0 text delta: ' '
[Request] Part 0 text delta: '2025'
[Request] Part 0 text delta: '-08'
[Request] Part 0 text delta: '-07'
[Request] Part 0 text delta: ' 16'
[Request] Part 0 text delta: ':36'
[Request] Part 0 text delta: ':02'
[Request] Part 0 text delta: '\n\n**'
[Request] Part 0 text delta: 'Weather in'
[Request] Part 0 text delta: ' Tokyo:**'
[Request] Part 0 text delta: ' 22'
[Request] Part 0 text delta: '°C ('
[Request] Part 0 text delta: '22 degrees'
[Request] Part 0 text delta: ' Celsius)'
=== CallToolsNode: streaming partial response & tool usage ===
=== Final Agent Output: 
Here's the current information for Tokyo:

**Current Time:** 2025-08-07 16:36:02

**Weather in Tokyo:** 22°C (22 degrees Celsius) ===

Did multiple runs on both versions for sanity check. 0.4.6 outputs a lot of empty strings before output. 0.4.7 removes that. I think the first PR @DouweM mentioned may be the culprit. I think the issue may be here if you see console ouput

[Request] Starting part 0: TextPart(content='\nHere')
[Result] The model produced a final result (tool_name=None)
[Request] Part 0 text delta: "'s the"

In 0.4.6 we were getting a '' or \n before [Result] The model produced a final result (tool_name=None) but after you removed these character in 0.4.7, now we get the starting word before [Result] The model produced a final result (tool_name=None). hence the missing tokens. Is there a utility to [Result] The model produced a final result (tool_name=None) after first token stream?

[DouweM]

@taaha Thanks for looking into this. The blank chunks being dropped was the goal of #2294, so that seems to be working well, and the "Final Agent Output" at the end has the full (post-blanks) message so no tokens are actually being dropped.

Is your code where you're seeing dropped initial tokens only starting to stream PartDeltaEvents after the FinalResultEvent, ignoring the PartStartEvent that started the TextPart? That would've worked before if the PartStartEvent always had uninteresting text anyway, but wouldn't have worked with models that don't start with blanks like openai:gpt-4o used in the example at https://ai.pydantic.dev/agents/#streaming:

        "[Request] Starting part 0: TextPart(content='It will be ')",
        '[Result] The model produced a final output (tool_name=None)',
        "[Request] Part 0 text delta: 'warm and sunny '",
        "[Request] Part 0 text delta: 'in Paris on '",
        "[Request] Part 0 text delta: 'Tuesday.'",

It's a bit cumbersome, but the most "correct" way to handle these events was and is to keep track of the most recently seen event, and once the FinalResultEvent is seen, process (print/stream) that and then start streaming the deltas that follow.

I can totally see how one could've ended up with an implementation that didn't do that and worked for the models you were using though, so I want to make sure that still works. We could move the FinalResultEvent to be emitted before the PartStartEvent(TextPart), but your (and others') code likely doesn't read from that (only from PartDeltaEvent), and others' "correct" code that looks for the most recent PartStartEvent(TextPart) would break.

So a better solution that works for both of these implementations would be to have the PartStartEvent(TextPart) always have content='', and only include the real initial content in the next PartDeltaEvent.

Would that work for you?

[DouweM]

On second thought, I don't like that "better solution", and I will change #2294 to be opt-in and only enabled by default for models we know to exhibit the issue it was meant to address.

[majdalsado]

So this issue for us turned out to actually be related a new DeltaEvent called PartStartEvent which we weren't handling in our node.stream().

I don't believe this event existed in prior versions which is why we weren't handling it. Checking the docs here it seems like it was updated with that event so all is good :)