fix(acp): fixtures now raise content mismatch errors#6912
fix(acp): fixtures now raise content mismatch errors#6912codefromthecrypt merged 1 commit intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes a bug in the ACP test fixture error reporting. Previously, when request body matching failed, the fixture would pop the expected value from the queue before checking if it matched, causing error messages to show the wrong expected value (or empty string if queue was exhausted). This made debugging test failures very difficult.
Changes:
- Fixed queue handling to peek before popping, ensuring error messages show the correct expected value
- Replaced JSON diff error messages with simple, readable expected/actual format
- Improved panic propagation in test runner to show actual failure messages
- Made test assertions more precise with
assert_eq!instead ofcontainschecks
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| crates/goose-acp/tests/fixtures/mod.rs | Fixed queue peek-before-pop logic, removed assert_json_diff dependency, improved error message formatting, and fixed panic propagation in test runner |
| crates/goose-acp/tests/common_tests/mod.rs | Changed test assertions from contains to assert_eq! for more precise validation, added error checking for run_builtin_and_mcp test |
| let message = if expected_body.is_empty() { | ||
| format!("Unexpected request:\n {}", body) | ||
| } else { | ||
| format!( |
There was a problem hiding this comment.
Did we lose something here with actual diff of the bodies together? Seems like we're only checking against empty.
There was a problem hiding this comment.
yeah the logic was wrong before as it was a substring search. So, what was happening is we were doing a diff of a substring vs an entire json body.
so it was doing a json diff of format!(r#"</info-msg>\n{prompt}""#) vs the actual json body.
This was a relic of an old version which did complete request matching, and is a bug in the current design. that's why json diff isn't useful here.
Make sense?
There was a problem hiding this comment.
Got it, thanks for clarifying!
Signed-off-by: Adrian Cole <adrian@tetrate.io>
7eab60d to
0827141
Compare
* origin/main: fix: detect context length errors in GCP Vertex AI provider (#6976) Added the ability to escape Jinja variables in recipes to include them in prompts (#6975) Bug Fix: bump pctx (#6967) fix(acp): fixtures now raise content mismatch errors (#6912) custom provider form minor improvements (#6966) Fix 'Edit In Place' feature appending instead of editing messages (#6955) docs: change RPI slash commands (#6963)
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Summary
Fix ACP test fixture to show clear error messages on content mismatch. Before, it returned empty string which defeated the logic involved.
Type of Change
AI Assistance
Testing
Introduced a typo and fixed the before and after issue
Related Issues
Relates to #6765
Diff
I introduced a typo (space) in the request match expression to trigger a failure.
Before
After: