Bug: Duplicate ObservationEvent with same tool_call_id causes LLM API error on conversation resume

[jpshackelford]

Bug Description

⚠️ This bug puts the conversation in an unrecoverable state. Once triggered, the agent cannot proceed and the conversation is permanently stuck. Every subsequent attempt to run the agent fails with the same error.

When a conversation is resumed after being paused/finished, a duplicate ObservationEvent can be created with the same tool_call_id as an existing observation. This causes the Anthropic API to reject the request with:

litellm.BadRequestError: Error code: 400 - {
  "error": {
    "message": "litellm.BadRequestError: AnthropicException - {
      \"type\": \"error\",
      \"error\": {
        \"type\": \"invalid_request_error\",
        \"message\": \"messages.59: `tool_use` ids were found without `tool_result` blocks immediately after: toolu_01CGASf7KnafqkuQMuLstHi4. Each `tool_use` block must have a corresponding `tool_result` block in the next message.\"
      },
      \"request_id\": \"req_011CXMLWpMmp7TBFbCVyLXY3\"
    }
    No fallback model group found for original model_group=prod/claude-opus-4-5-20251101.
    Fallbacks=[{"qwen3-coder-480b": ["qwen3-coder-480b-or"]}, {"glm-4.5": ["glm-4.5-or"]}].
    Received Model Group=prod/claude-opus-4-5-20251101
    Available Model Group Fallbacks=None",
    "type": null,
    "param": null,
    "code": "400"
  }
}

Since the duplicate observation is persisted in the event stream, there is no way to recover - every LLM call will include the malformed message history and fail.

Steps to Reproduce

Observed in conversation af22d964cb9f464e957320d647c7471e:

1. Start a conversation and let the agent run actions
2. Let the conversation finish normally (agent calls finish tool)
3. Wait some time (in this case ~1.5 hours)
4. Resume the conversation by sending a new user message
5. The conversation fails with the tool_use/tool_result mismatch error
6. All subsequent attempts to continue the conversation fail with the same error

Root Cause Analysis

Analyzing the event stream via /api/v1/conversation/{id}/events/search:

Event	Timestamp	Type	Details
94	21:04:03	ActionEvent	tool_call_id=toolu_01CGASf7KnafqkuQMuLstHi4
95	21:04:24	ObservationEvent	First observation for action 94
110	21:06:10	ObservationEvent	Finish action - conversation completed
111	21:06:10	StateUpdate	execution_status=finished
(1.5 hour gap)
114	22:45:53	MessageEvent	User sent new message
116	22:45:54	ObservationEvent	DUPLICATE - same tool_call_id=toolu_01CGASf7KnafqkuQMuLstHi4

The result is:

• 1 tool_use (ActionEvent) with ID toolu_01CGASf7KnafqkuQMuLstHi4
• 2 tool_result (ObservationEvents) with the same ID

Claude's API requires exactly one tool_result per tool_use.

Likely Causes

1. Event sync issue during resume: When the conversation was restarted, events may not have been fully loaded from persistence, causing get_unmatched_actions() to incorrectly identify the old action as unmatched and re-execute it.

2. Missing deduplication in filter_unmatched_tool_calls(): The current implementation in View.filter_unmatched_tool_calls() uses sets to track tool_call_ids. If there are multiple observations with the same tool_call_id, they're all kept because the ID exists in the action set.

Suggested Fixes

Defensive Fix (view.py)

Modify filter_unmatched_tool_calls() to track counts and only keep one observation per tool_call_id:

@staticmethod
def filter_unmatched_tool_calls(
    events: list[LLMConvertibleEvent],
) -> list[LLMConvertibleEvent]:
    # ... existing code ...
    
    # Track which tool_call_ids have already been seen for observations
    seen_observation_tool_call_ids: set[ToolCallID] = set()
    
    result = []
    for event in events:
        if event.id in removed_event_ids:
            continue
        if isinstance(event, ObservationBaseEvent):
            if event.tool_call_id in tool_call_ids_to_remove:
                continue
            # NEW: Skip duplicate observations
            if event.tool_call_id in seen_observation_tool_call_ids:
                logger.warning(
                    f"Skipping duplicate observation for tool_call_id: {event.tool_call_id}"
                )
                continue
            if event.tool_call_id is not None:
                seen_observation_tool_call_ids.add(event.tool_call_id)
        result.append(event)
    return result

This defensive fix would also recover existing stuck conversations by filtering out the duplicate observations at LLM message construction time.

Root Cause Fix

Investigate why duplicate observations are created during conversation resume. Key locations:

• openhands-agent-server/openhands/agent_server/event_service.py - the start() method
• openhands-sdk/openhands/sdk/conversation/state.py - get_unmatched_actions()
• Event persistence/loading logic during remote sandbox resume

Environment

• Platform: OpenHands Cloud (app.all-hands.dev)
• Model: prod/claude-opus-4-5-20251101
• Conversation ID: af22d964cb9f464e957320d647c7471e

Related Files

• openhands-sdk/openhands/sdk/context/view.py - filter_unmatched_tool_calls()
• openhands-sdk/openhands/sdk/conversation/state.py - get_unmatched_actions()
• openhands-agent-server/openhands/agent_server/event_service.py - start() method

[enyst]

Has there been a condensation event between 94 and 116 ?

I would love the raw trajectory or telemetry logs, if that was possible by any chance. It's OK if not, it's weird because we have seen this before and I'm pretty sure we have related it to condensation. 🤔

[enyst]

Oh, I just got it too. Unfortunately on the cloud as well:

litellm.BadRequestError: Error code: 400 - {'error': {'message': 'litellm.BadRequestError: AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"messages.116.content.1: each tool_use must have a single result. Found multiple `tool_result` blocks with id: toolu_016UK3iccP6uNkbyfazKMQXV"},"request_id":"req_011CXMP4ZnoTv7eUEDEMhibt"}. Received Model Group=claude-opus-4-5\nAvailable Model Group Fallbacks=None', 'type': None, 'param': None, 'code': '400'}}

[jpshackelford]

Trajectory Analysis for Conversation af22d964cb9f464e957320d647c7471e

I retrieved the full event stream via the V1 API. Here's what I found:

✅ Yes, there was a duplicate ObservationEvent

The trajectory confirms a duplicate observation for tool_call_id=toolu_01CGASf7KnafqkuQMuLstHi4:

Event	Timestamp	Type	tool_call_id	Notes
94	21:04:03.209	ActionEvent	toolu_01CGASf7KnafqkuQMuLstHi4	Terminal command (pytest)
95	21:04:24.579	ObservationEvent	toolu_01CGASf7KnafqkuQMuLstHi4	✅ First observation - test passed
96-111	...	Various	...	Conversation continues normally
110	21:06:10.669	ObservationEvent	(finish tool)	Agent finished
(1h40m gap - conversation resumed on new runtime)
114	22:45:53.712	MessageEvent	-	User sent new message
116	22:45:54.384	ObservationEvent	toolu_01CGASf7KnafqkuQMuLstHi4	⚠️ DUPLICATE observation
118	22:45:58.275	ConversationErrorEvent	-	LLMBadRequestError

Key Evidence: Different Runtime Hosts

The two observations were executed on different Kubernetes pods, confirming the command was re-executed when the conversation resumed on a new runtime:

Event 95 (original):

{
  "hostname": "runtime-mjhjdtteuwsifhaq-794b86b65f-qdj27",
  "exit_code": 0,
  "output": "1 passed, 3 warnings"
}

Event 116 (duplicate):

{
  "hostname": "runtime-mjhjdtteuwsifhaq-794b86b65f-cqlv6",
  "exit_code": 1,
  "output": "/usr/local/bin/python: No module named pytest"
}

Timeline Reconstruction

1. 21:04:03 - Agent executed pytest command (event 94)
2. 21:04:24 - Observation received, test passed (event 95)
3. 21:06:10 - Agent called finish tool, conversation completed
4. ~1h40m gap - Runtime was likely terminated
5. 22:45:53 - User sent new message, conversation resumed on NEW runtime
6. 22:45:54 - System re-executed the old terminal command on the new runtime (which failed because pytest wasn't installed)
7. 22:45:54 - Duplicate observation emitted with same tool_call_id
8. 22:45:58 - LLM API rejected the malformed message history

Was this caused by condensation?

Based on this analysis, the duplicate observation doesn't appear to be caused by condensation per se. The issue seems to be in the conversation resume logic when a new runtime is spun up:

1. When the conversation resumed, get_unmatched_actions() appears to have incorrectly identified action 94 as "unmatched" (possibly because events weren't fully loaded from persistence yet)
2. The system then re-executed the command on the new runtime
3. A new observation was created with the same tool_call_id as the original

This would explain why:

• The two observations have different hostnames (different runtimes)
• The duplicate has a different result (pytest not installed on fresh runtime)
• The issue occurs specifically on conversation resume after a gap

Raw Trajectory

Full event timeline (click to expand)

Events marked with ** are in the problematic range (94-118):

   0: [21:00:04.902] SystemPromptEvent
   1: [21:00:04.911] ConversationStateUpdateEvent
...
  93: [21:04:03.208] ConversationStateUpdateEvent
**94: [21:04:03.209] ActionEvent          tool=terminal  tcid=toolu_01CGASf7KnafqkuQMuLstHi4
**95: [21:04:24.579] ObservationEvent     tool=terminal  tcid=toolu_01CGASf7KnafqkuQMuLstHi4  (FIRST)
...
**109: [21:06:10.667] ActionEvent         tool=finish
**110: [21:06:10.669] ObservationEvent    tool=finish
**111: [21:06:10.670] ConversationStateUpdateEvent
**112: [22:44:04.514] ConversationStateUpdateEvent  <-- ~1h40m gap
**113: [22:45:53.711] ConversationStateUpdateEvent
**114: [22:45:53.712] MessageEvent (user)
**115: [22:45:53.714] ConversationStateUpdateEvent
**116: [22:45:54.384] ObservationEvent     tool=terminal  tcid=toolu_01CGASf7KnafqkuQMuLstHi4  (DUPLICATE!)
**117: [22:45:58.274] ConversationStateUpdateEvent
**118: [22:45:58.275] ConversationErrorEvent  ERROR=LLMBadRequestError

The full JSON trajectory (135 events) is available if needed for further debugging.

[csmith49]

As a sanity check, I grabbed the LLM completions and counted the number of messages per completion call sorted by timestamp. They're monotonically increasing, which also helps confirm that there's no condensation that happened in the middle.

[csmith49]

On the defensive fix: we could try to resolve this in the View, but it already assumes the event stream is well-formed (at least before condensations are applied) and I worry handling all the scrambled event cases will just complicate the already brittle logic.

What if we just ensure that we never add an event with a duplicate ID to the event stream? That way it never gets persisted in the first place.

[jpshackelford]

Just happened to me again.

Conversation:
da67eaa7652947959ce029f511641f72

litellm.BadRequestError: Error code: 400 - {'error': {'message': 'litellm.BadRequestError: AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"messages.134.content.2: unexpected `tool_use_id` found in `tool_result` blocks: toolu_01JwVuYxmP2aC2YakVunJPVv. Each `tool_result` block must have a corresponding `tool_use` block in the previous message."},"request_id":"req_011CXMXnbnTR1BDjcFyuV88b"}No fallback model group found for original model_group=prod/claude-opus-4-5-20251101. Fallbacks=[{\'qwen3-coder-480b\': [\'qwen3-coder-480b-or\']}, {\'glm-4.5\': [\'glm-4.5-or\']}]. Received Model Group=prod/claude-opus-4-5-20251101\nAvailable Model Group Fallbacks=None\nError doing the fallback: litellm.BadRequestError: AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"messages.134.content.2: unexpected `tool_use_id` found in `tool_result` blocks: toolu_01JwVuYxmP2aC2YakVunJPVv. Each `tool_result` block must have a corresponding `tool_use` block in the previous message."},"request_id":"req_011CXMXnbnTR1BDjcFyuV88b"}No fallback model group found for original model_group=prod/claude-opus-4-5-20251101. Fallbacks=[{\'qwen3-coder-480b\': [\'qwen3-coder-480b-or\']}, {\'glm-4.5\': [\'glm-4.5-or\']}] LiteLLM Retried: 1 times, LiteLLM Max Retries: 2', 'type': None, 'param': None, 'code': '400'}}

[jpshackelford]

Full trajectory JSON (135 events): from af22d964cb9f464e957320d647c7471e (first conversation mentioned)

REDACTED

[jpshackelford]

@csmith49 As I understand it. the view fix would address this for past conversations that were already in a bad state, whereas the ensure that we never add an event with a duplicate ID to the event stream would prevent it from happening in the first place. Seems like we would want both.

While redundant, the impact of stuck agent is a very poor outcome for a user that isn't aware of recovery possibilities via api calls that allow retrieval of lost data. Given the severity of the loss as experienced by a new user--"Oh no, that was an hour of uncommitted fixes/features and I just now got it all working"--and the level of trust and good will lost in the experience, I would say anything we can do to allow the agent to recover from a bad state is important to do, even if it didn't address the root cause.

My inclination is to open a PR that includes both fixes and also adds logging to see if we can't spot what has gone wrong with the recovery logic.

[enyst]

According to the agent (and I think I see in the log), there were other actions / observations between the two duplicate tool_call_id observations. That seems very weird!

• Between the first ObservationEvent (2026-01-21T21:04:24.579150) and the second ObservationEvent (2026-01-21T22:45:54.384580) there are 20 events total, including 5 ActionEvent/ObservationEvent pairs (10 events) with their own
tool_call_ids.

Here’s the full in-between list (timestamp | kind | tool_call_id if any):

• 2026-01-21T21:04:52.507759 | ConversationStateUpdateEvent | —
• 2026-01-21T21:04:52.508735 | ActionEvent | toolu_01RcpYEZtGMooWk71mwGfW8e
• 2026-01-21T21:04:52.511451 | ObservationEvent | toolu_01RcpYEZtGMooWk71mwGfW8e
• 2026-01-21T21:04:56.104911 | ConversationStateUpdateEvent | —
• 2026-01-21T21:04:56.105905 | ActionEvent | toolu_013VM88xboNJzkL9CjeXGSjq
• 2026-01-21T21:05:17.047599 | ObservationEvent | toolu_013VM88xboNJzkL9CjeXGSjq
• 2026-01-21T21:05:49.991880 | ConversationStateUpdateEvent | —
• 2026-01-21T21:05:49.992878 | ActionEvent | toolu_01UTeFn2XxbkdyPELprfKjHa
• 2026-01-21T21:05:49.994780 | ObservationEvent | toolu_01UTeFn2XxbkdyPELprfKjHa
• 2026-01-21T21:05:54.676210 | ConversationStateUpdateEvent | —
• 2026-01-21T21:05:54.677178 | ActionEvent | toolu_01UXMJpFg6gUmAdzYnDuzLVn
• 2026-01-21T21:05:55.310316 | ObservationEvent | toolu_01UXMJpFg6gUmAdzYnDuzLVn
• 2026-01-21T21:06:10.666524 | ConversationStateUpdateEvent | —
• 2026-01-21T21:06:10.667529 | ActionEvent | toolu_01VHMDaueVqwNbeYsJTK4eGA
• 2026-01-21T21:06:10.669374 | ObservationEvent | toolu_01VHMDaueVqwNbeYsJTK4eGA
• 2026-01-21T21:06:10.670773 | ConversationStateUpdateEvent | —
• 2026-01-21T22:44:04.514685 | ConversationStateUpdateEvent | —
• 2026-01-21T22:45:53.711646 | ConversationStateUpdateEvent | —
• 2026-01-21T22:45:53.712892 | MessageEvent | —
• 2026-01-21T22:45:53.714865 | ConversationStateUpdateEvent | —

[enyst]

Between the two ObservationEvents for the duplicated tool_call_id (toolu_01CGASf7KnafqkuQMuLstHi4), the in-between events are:

• 2026-01-21T21:04:52.507759 | ConversationStateUpdateEvent | tool: - | tool_call_id: -
• 2026-01-21T21:04:52.508735 | ActionEvent | tool: file_editor | tool_call_id: toolu_01RcpYEZtGMooWk71mwGfW8e
• 2026-01-21T21:04:52.511451 | ObservationEvent | tool: file_editor | tool_call_id: toolu_01RcpYEZtGMooWk71mwGfW8e
• 2026-01-21T21:04:56.104911 | ConversationStateUpdateEvent | tool: - | tool_call_id: -
• 2026-01-21T21:04:56.105905 | ActionEvent | tool: terminal | tool_call_id: toolu_013VM88xboNJzkL9CjeXGSjq
• 2026-01-21T21:05:17.047599 | ObservationEvent | tool: terminal | tool_call_id: toolu_013VM88xboNJzkL9CjeXGSjq
• 2026-01-21T21:05:49.991880 | ConversationStateUpdateEvent | tool: - | tool_call_id: -
• 2026-01-21T21:05:49.992878 | ActionEvent | tool: think | tool_call_id: toolu_01UTeFn2XxbkdyPELprfKjHa
• 2026-01-21T21:05:49.994780 | ObservationEvent | tool: think | tool_call_id: toolu_01UTeFn2XxbkdyPELprfKjHa
• 2026-01-21T21:05:54.676210 | ConversationStateUpdateEvent | tool: - | tool_call_id: -
• 2026-01-21T21:05:54.677178 | ActionEvent | tool: terminal | tool_call_id: toolu_01UXMJpFg6gUmAdzYnDuzLVn
• 2026-01-21T21:05:55.310316 | ObservationEvent | tool: terminal | tool_call_id: toolu_01UXMJpFg6gUmAdzYnDuzLVn
• 2026-01-21T21:06:10.666524 | ConversationStateUpdateEvent | tool: - | tool_call_id: -
• 2026-01-21T21:06:10.667529 | ActionEvent | tool: finish | tool_call_id: toolu_01VHMDaueVqwNbeYsJTK4eGA
• 2026-01-21T21:06:10.669374 | ObservationEvent | tool: finish | tool_call_id: toolu_01VHMDaueVqwNbeYsJTK4eGA
• 2026-01-21T21:06:10.670773 | ConversationStateUpdateEvent | tool: - | tool_call_id: -
• 2026-01-21T22:44:04.514685 | ConversationStateUpdateEvent | tool: - | tool_call_id: -
• 2026-01-21T22:45:53.711646 | ConversationStateUpdateEvent | tool: - | tool_call_id: -
• 2026-01-21T22:45:53.712892 | MessageEvent | tool: - | tool_call_id: -
• 2026-01-21T22:45:53.714865 | ConversationStateUpdateEvent | tool: - | tool_call_id: -