fix: Agent Example and TURN detection

[d3xvn]

• Implement automatic LLM triggering in _on_transcript() for both modes:
  • Without turn detection: triggers immediately on transcript completion
  • With turn detection: accumulates transcripts and waits for TurnEndedEvent
• Add _pending_user_transcripts dict to track multi-chunk transcripts per user
• Implement turn detection LLM response in _on_turn_event()
• Add TTS interruption when user starts speaking (barge-in)
• Fix FAL turn detection event emission logic
• Fix double TTS triggering in OpenAI LLM plugin (was emitting LLMResponseCompletedEvent twice)

Summary by CodeRabbit

• New Features
  • Turn-based handling with per-speaker transcript accumulation and optional immediate responses.
  • Automatic interruption of TTS when another participant begins speaking.
• Bug Fixes
  • Avoid duplicate "response completed" events for streaming LLM outputs.
• Documentation
  • Example updated to demonstrate enabling turn detection.
• Chores
  • Renamed dependencies from stream-agents to vision-agents plugins; added fal-client and updated package sources.

[coderabbitai]

Walkthrough

Implements per-speaker transcript accumulation and turn-based LLM triggering, TTS interruption on non-agent speech, realtime-mode short-circuiting, and expanded partial-transcript handling. Fal turn detection now emits explicit TURN_ENDED/TURN_STARTED on speaker switches. OpenAI plugin avoids duplicate completion events for streaming. Example and deps updated to vision-agents and fal-client.

Changes

Cohort / File(s)	Summary
Agent turn-based handling agents-core/vision_agents/core/agents/agents.py	Adds self._pending_user_transcripts for per-speaker accumulation; handles TurnStartedEvent / TurnEndedEvent to interrupt TTS and trigger LLM responses when appropriate; respects realtime_mode to short-circuit LLM triggering; accumulates transcripts in _on_transcript and extends _on_partial_transcript behavior for streaming user messages.
Turn detection event semantics agents-core/vision_agents/core/turn_detection/fal_turn_detection.py	Reworks _process_turn_prediction to emit TURN_ENDED more deterministically (including on speaker switches), emit previous-speaker TURN_ENDED before new TURN_STARTED, and update/clear current_speaker around events.
OpenAI LLM completion events plugins/openai/vision_agents/plugins/openai/openai_llm.py	Emits LLMResponseCompletedEvent only for non-streaming OpenAIResponse paths to avoid duplicate completion events when streaming responses are handled elsewhere.
Example app enablement examples/01_simple_agent_example/simple_agent_example.py	Imports FalTurnDetection and passes an instance to the agent via the new turn_detection parameter to enable turn-detection during the example run.
Example project dependencies examples/01_simple_agent_example/pyproject.toml	Replaces stream-agents* dependencies with vision-agents*, adds fal-client>=0.5.3, updates [tool.uv.sources] to vision-agents plugin sources, and removes krisp-audio sources/export mapping.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor User
  participant Mic as FalTurnDetection
  participant Agent
  participant LLM
  participant TTS

  User->>Mic: audio
  Mic->>Agent: TURN_STARTED(user_id)
  Note over Agent,TTS: If TTS playing and speaker != agent → interrupt TTS

  par Speech streaming
    Mic-->>Agent: PARTIAL_TRANSCRIPT / TRANSCRIPT chunks
    Agent->>Agent: Accumulate per-speaker transcripts
  end

  Mic->>Agent: TURN_ENDED(user_id)
  alt realtime_mode == true
    Agent->>Agent: Short-circuit (no LLM trigger)
  else turn_detection enabled
    Agent->>Agent: Fetch accumulated transcript for user
    alt transcript non-empty
      Agent->>LLM: simple_response(text, participant)
      alt streaming response
        LLM-->>Agent: stream events/tokens
        Agent-->>TTS: stream speak (optional)
        Note right of Agent: Completion emitted by streaming path
      else non-streaming response
        LLM-->>Agent: final response
        Agent-->>Agent: emit LLMResponseCompletedEvent
        Agent-->>TTS: speak response (optional)
      end
      Agent->>Agent: clear pending transcript for user
    else
      Agent->>Agent: no-op
    end
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• GetStream/agents#66 — Modifies Agent turn-detection integration and fal turn event handling; overlaps with changes to agents.py and fal_turn_detection.py.
• GetStream/agents#68 — Renames/exports turn-detection base and relates to the new turn_detection parameter and turn-detector usage in examples.

Poem

I listen—glass mouth, waiting to split—
and catalog the small departures of air.
Each voice a cold room where I stitch an ending,
clamp the sudden syllable, press it into a lamp.
The machine learns the shape of goodbye and says it back.

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Title Check	⚠️ Warning	The title “fix: Agent Example and TURN detection” is too generic and splits focus between an example and turn detection without capturing the primary change of introducing turn-based LLM triggering, transcript accumulation, TTS interruption, and related bug fixes.	Consider renaming the PR to clearly summarize the main feature and fixes, for example “Add turn-based LLM triggering with transcript accumulation and fix turn detection logic” to provide a concise, accurate overview of the scope of changes.
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/agent-example

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 589f6da and 8c01c31.

⛔ Files ignored due to path filters (1)

• examples/01_simple_agent_example/uv.lock is excluded by !**/*.lock

📒 Files selected for processing (5)

• agents-core/vision_agents/core/agents/agents.py (3 hunks)
• agents-core/vision_agents/core/turn_detection/fal_turn_detection.py (1 hunks)
• examples/01_simple_agent_example/pyproject.toml (1 hunks)
• examples/01_simple_agent_example/simple_agent_example.py (2 hunks)
• plugins/openai/vision_agents/plugins/openai/openai_llm.py (1 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*.py

📄 CodeRabbit inference engine (.cursor/rules/python.mdc)

**/*.py: Never adjust sys.path (e.g., sys.path.append/insert/assignment)
Docstrings must follow the Google style guide

Files:

• plugins/openai/vision_agents/plugins/openai/openai_llm.py
• agents-core/vision_agents/core/agents/agents.py
• agents-core/vision_agents/core/turn_detection/fal_turn_detection.py
• examples/01_simple_agent_example/simple_agent_example.py

🧬 Code graph analysis (3)

agents-core/vision_agents/core/agents/agents.py (2)

agents-core/vision_agents/core/turn_detection/events.py (1)

• TurnEndedEvent (29-44)

plugins/openai/vision_agents/plugins/openai/openai_llm.py (1)

• simple_response (67-91)

agents-core/vision_agents/core/turn_detection/fal_turn_detection.py (1)

agents-core/vision_agents/core/turn_detection/turn_detection.py (2)

• _emit_turn_event (99-126)
• TurnEvent (12-16)

examples/01_simple_agent_example/simple_agent_example.py (1)

agents-core/vision_agents/core/turn_detection/fal_turn_detection.py (1)

• FalTurnDetection (31-377)

🪛 GitHub Actions: CI (unit)

agents-core/vision_agents/core/agents/agents.py

[error] 807-807: F541: f-string without any placeholders. Remove extraneous f prefix. Found 1 error; 1 fixable with the --fix option.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8c01c31 and 2eacdfb.

📒 Files selected for processing (1)

• agents-core/vision_agents/core/agents/agents.py (3 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*.py

📄 CodeRabbit inference engine (.cursor/rules/python.mdc)

**/*.py: Never adjust sys.path (e.g., sys.path.append/insert/assignment)
Docstrings must follow the Google style guide

Files:

• agents-core/vision_agents/core/agents/agents.py

🧬 Code graph analysis (1)

agents-core/vision_agents/core/agents/agents.py (3)

agents-core/vision_agents/core/turn_detection/events.py (2)

• TurnStartedEvent (10-25)
• TurnEndedEvent (29-44)

agents-core/vision_agents/core/edge/types.py (1)

• duration (73-109)

plugins/openai/vision_agents/plugins/openai/openai_llm.py (1)

• simple_response (67-91)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)

• GitHub Check: unit / Ruff & mypy
• GitHub Check: unit / Test "not integration"
• GitHub Check: unit / Ruff & mypy
• GitHub Check: unit / Test "not integration"

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Don't assume transcripts arrive before TurnEnded.

Turn detection emits TurnEndedEvent as soon as silence is detected, but STT final transcripts can land a beat later (we call turn_detection.process_audio before stt.process_audio). When that happens, Line 701 sees an empty transcript, we skip simple_response, and nothing ever re-triggers once the transcript finally shows up in _on_transcript. Result: user turns are silently dropped whenever STT lags behind turn detection—major functional regression.

Please track “turn ended but transcript pending” state. One way:

@@
-        self._pending_user_transcripts: Dict[str, str] = {}
+        self._pending_user_transcripts: Dict[str, str] = {}
+        self._pending_turn_completions: set[str] = set()
@@ def _on_turn_event(...):
-            if event.speaker_id and event.speaker_id != self.agent_user.id:
+            if event.speaker_id and event.speaker_id != self.agent_user.id:
                 transcript = self._pending_user_transcripts.get(event.speaker_id, "")
 
-                if transcript and transcript.strip():
+                if transcript and transcript.strip():
                     ...
                     self._pending_user_transcripts[event.speaker_id] = ""
+                    self._pending_turn_completions.discard(event.speaker_id)
+                else:
+                    self._pending_turn_completions.add(event.speaker_id)
@@ def _on_transcript(...):
-            if user_id not in self._pending_user_transcripts:
+            if user_id not in self._pending_user_transcripts:
                 ...
             else:
                 ...
 
+            if user_id in getattr(self, "_pending_turn_completions", set()):
+                participant = getattr(event, "user_metadata", None)
+                await self.simple_response(self._pending_user_transcripts[user_id], participant)
+                self._pending_user_transcripts[user_id] = ""
+                self._pending_turn_completions.discard(user_id)

Any equivalent solution that ensures a late-arriving transcript still fires the LLM response works.

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents

In agents-core/vision_agents/core/agents/agents.py around lines 699–718, the
TurnEnded handling assumes the final STT transcript is already available and
skips triggering the LLM when the transcript arrives late; fix by recording that
a turn ended for this speaker when TurnEnded is received (e.g., add speaker_id
to a pending_turns set or mark a flag alongside the empty transcript) and then,
in the transcript arrival path (_on_transcript or wherever transcripts are
written to _pending_user_transcripts), check for that pending-turn-ended marker
and if present call simple_response(transcript, participant) and clear both the
pending marker and the stored transcript; ensure you still call simple_response
immediately when TurnEnded sees a non-empty transcript and avoid double-calling
by clearing the marker after handling.