ask-user freeform snapshot format mismatch causes test failures #315
Description
Summary
The should_handle_freeform_user_input_response test snapshot in test/snapshots/ask-user/ uses a tool result format that doesn't match the current CLI v0.0.400 output, causing the test to timeout across all SDK implementations.
Affected SDKs
- ✅ Node.js SDK -
test/e2e/ask-user.test.ts→should handle freeform user input responseFAILS - ✅ .NET SDK -
AskUserTests.cs→Should_Handle_Freeform_User_Input_Response(likely fails) - ✅ Python SDK - (likely fails)
Problem Details
Snapshot Content (current)
The snapshot file test/snapshots/ask-user/should_handle_freeform_user_input_response.yaml contains:
- role: tool
tool_call_id: toolcall_0
content: "User response (freeform): This is my custom freeform answer that was not in the choices"Actual CLI v0.0.400 Output
When the SDK responds to an ask_user tool request with wasFreeform: true, the CLI produces:
{
"type": "tool.execution_complete",
"data": {
"toolCallId": "toolcall_0",
"result": {
"content": "User responded: This is my custom freeform answer that was not in the choices"
}
}
}Mismatch
| Expected (snapshot) | Actual (CLI v0.0.400) |
|---|---|
User response (freeform): ... |
User responded: ... |
The ReplayingCapiProxy cannot find a matching cached response for the second API call because the tool result content doesn't match the snapshot.
Steps to Reproduce
- Clone the repository
- Run the Node.js SDK tests:
cd nodejs npm install npm test
- Observe the failure:
❯ test/e2e/ask-user.test.ts (3 tests | 1 failed) 33067ms ✓ should invoke user input handler when model uses ask_user tool 2847ms ✓ should receive choices in user input request 196ms × should handle freeform user input response 30014ms FAIL test/e2e/ask-user.test.ts > User input (ask_user) > should handle freeform user input response Error: Test timed out in 30000ms.
Root Cause Analysis
The replay proxy uses prefix matching to find cached responses. When the CLI makes the second API call (after receiving the tool result), it sends messages including:
- role: tool
tool_call_id: toolcall_0
content: "User responded: This is my custom freeform answer..."But the snapshot expects:
- role: tool
tool_call_id: toolcall_0
content: "User response (freeform): This is my custom freeform answer..."Since these don't match, findAssistantIndexAfterPrefix() in replayingCapiProxy.ts returns undefined, and the proxy falls back to making a real API call (or times out waiting).
Suggested Fix
Update the snapshot to match the current CLI v0.0.400 output format:
models:
- claude-sonnet-4.5
conversations:
- messages:
- role: system
content: ${system}
- role: user
content: Ask me a question using ask_user and then include my answer in your response. The question should be 'What is your favorite color?'
- role: assistant
tool_calls:
- id: toolcall_0
type: function
function:
name: ask_user
arguments: '{"question":"What is your favorite color?","allow_freeform":true}'
- role: tool
tool_call_id: toolcall_0
content: "User responded: This is my custom freeform answer that was not in the choices"
- role: assistant
content: 'Your answer was: "This is my custom freeform answer that was not in the choices"'Environment
- CLI version:
@github/copilotv0.0.400 - Node.js: v20+
- OS: macOS (also likely reproducible on Linux/Windows)
Additional Context
The other two ask-user tests pass successfully because they use the non-freeform path which has the format "User selected: Option A" - this format appears to be consistent between the snapshot and CLI output.