Properly handle Gemini thought summaries and export them via OpenTelemetry in Google LLM plugin

[giovaborgogno]

Bug Description

Gemini’s API provides thought summaries when include_thoughts=true is enabled. These appear in content.parts with the part.thought flag set and must be handled separately from normal answer text. https://ai.google.dev/gemini-api/docs/thinking#summaries

Right now, the Google LLM plugin in agents/livekit-plugins/livekit-plugins-google/livekit/plugins/google /llm.py ignores part.thought entirely. The parsing code only looks at part.function_call and part.text:

def _parse_part(self, id: str, part: types.Part) -> llm.ChatChunk | None:
    if part.function_call:
        chat_chunk = llm.ChatChunk(
            id=id,
            delta=llm.ChoiceDelta(
                role="assistant",
                tool_calls=[
                    llm.FunctionToolCall(
                        arguments=json.dumps(part.function_call.args),
                        name=part.function_call.name,
                        call_id=part.function_call.id or utils.shortuuid("function_call_"),
                    )
                ],
                content=part.text,
            ),
        )
        return chat_chunk

    return llm.ChatChunk(
        id=id,
        delta=llm.ChoiceDelta(content=part.text, role="assistant"),
    )

There’s no check for part.thought, so thought summaries are treated as regular assistant output.

Problems

1. Bug: Thoughts are spoken by TTS
   When include_thoughts=True, thought summaries are merged into the same content used for user-facing responses. The TTS layer receives them and reads the agent’s internal reasoning out loud, which is not what Gemini’s “thinking” feature is meant for.

2. Missing observability of thoughts in OTEL
   LiveKit already uses OpenTelemetry, but the Gemini thought summaries are not surfaced there at all. There is no way to inspect the model’s internal reasoning in traces/logs while keeping it hidden from the end user and TTS.

Expected Behavior

Separation of thoughts vs. answer
Parts with part.thought == True should not be included in the assistant’s user-visible content.
Thought parts should never be forwarded to TTS or any channel that is meant for end-user output.

Observability via existing OpenTelemetry
Thought summaries should be attached to the existing OTEL spans/traces for Gemini calls

Reproduction Steps

e.g.:


session = AgentSession(
        stt="assemblyai/universal-streaming-multilingual",
        llm=google.LLM(
            model="gemini-2.5-flash-preview-09-2025",
            temperature=0.8,
            thinking_config=types.ThinkingConfig(
                include_thoughts=True,
                thinking_budget=1500,
            ),
        ),
        tts=elevenlabs.TTS(
            voice_id=elevenlabs_voice_id,
            model=elevenlabs_model,
            language=tts_language,
        ),
    )

Operating System

macOS Tahoe

Models Used

assemblyai, google plugin, eleven labs

Package Versions

# Core LiveKit dependencies
livekit>=1.0.13
livekit-agents[images,elevenlabs]>=1.3.6
livekit-api>=1.0.5
livekit-protocol>=1.0.6

# LiveKit plugins
livekit-plugins-google>=1.3.6

# Google Gemini API
google-generativeai==0.8.3

# Telemetry (Langfuse/OpenTelemetry/Judgment Labs)
opentelemetry-api>=1.39.0
opentelemetry-sdk>=1.39.0
opentelemetry-exporter-otlp-proto-http>=1.39.0
judgeval>=0.1.0  # Judgment Labs tracing (OpenTelemetry compatible)

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response

[giovaborgogno]

Once the wrapping of Gemini thoughts (as proposed in #4191) or any equivalent solution is available, it becomes straightforward to keep thought summaries out of the user-facing / TTS output while still exporting them through OpenTelemetry.

As long as the model output reliably marks them (e.g., with ... or a similar mechanism), we can instrument them cleanly.

Here’s the kind of logic I plan to use once thoughts become detectable in the stream. It intercepts the chunks produced by the default llm_node, removes any thought content before TTS, and emits the captured thoughts as OTEL events:

from opentelemetry.trace import get_tracer

async def llm_node(self, chat_ctx, tools, model_settings):
    tracer = get_tracer(__name__)

    async def process_stream():
        inside_thoughts = False
        current_thought = ""

        with tracer.start_as_current_span("llm.thought_processing") as span:
            async for chunk in Agent.default.llm_node(self, chat_ctx, tools, model_settings):
                if chunk is None:
                    yield chunk
                    continue

                content = getattr(chunk.delta, "content", None)
                if not content:
                    yield chunk
                    continue

                has_open = "<think>" in content
                has_close = "</think>" in content

                # Both tags in a single chunk
                if has_open and has_close:
                    before = content.split("<think>", 1)[0]
                    thought = content.split("<think>", 1)[1].split("</think>", 1)[0]
                    after = content.split("</think>", 1)[1]

                    span.add_event("thought", {"content": thought})
                    content = before + after

                else:
                    # Opening tag
                    if has_open:
                        inside_thoughts = True
                        current_thought = ""
                        content = content.split("<think>", 1)[1]

                    # Closing tag
                    if has_close:
                        current_thought += content.split("</think>", 1)[0]
                        span.add_event("thought", {"content": current_thought})

                        inside_thoughts = False
                        current_thought = ""
                        content = content.split("</think>", 1)[1]

                # While inside a <think> block, accumulate but do not emit
                if inside_thoughts:
                    current_thought += content
                    continue

                # Strip leftover tags just in case
                content = content.replace("<think>", "").replace("</think>", "")

                if content:
                    chunk.delta.content = content
                    yield chunk

    async for chunk in process_stream():
        yield chunk

This gives full observability through OTEL and prevents thoughts from leaking into speech, once the underlying LLM output exposes them in a consistent way —regardless of whether the proposed PR is the final implementation.

[phanosanastasiou]

Is this now possible with the new gemini model update?

[Arkoder]

Is there a solution this? Want to try gemini 3 flash with livekit agents but getting the same thought signature issue with function calling

[sahalghaf]

+1 facing the same issue

[FrejusGdm]

+1 facing the same issue