-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Description
Describe the bug
The ADK's TrajectoryEvaluator performs exact comparison of tool arguments including authentication tokens that change on every run. When using OpenAPIToolset with OAuth2/Bearer authentication, the auth token is included in tool arguments with the prefix _auth_prefix_vaf_Authorization. Since these tokens are dynamic, evaluation always fails even when the tool calls are functionally identical.
To Reproduce
Steps to reproduce the behavior:
- Create an agent using
OpenAPIToolsetwith OAuth2 authentication - Record an evalset with a tool call that includes auth parameters
- Run
adk evalon the agent - See evaluation failure due to mismatched auth tokens
Example evalset snippet:
"tool_uses": [
{
"id": "adk-437d690f-5a5e-46ed-a47e-0dd374cdf7b5",
"args": {
"org_id": "my-org.com",
"_auth_prefix_vaf_Authorization": "Bearer eyJraWQ..."
},
"name": "list_conditions"
}
]Expected behavior
The evaluator should ignore authentication parameters (prefixed with _auth_prefix_) when comparing tool trajectories, as these are implementation details and not part of the actual tool logic.
Desktop (please complete the following information):
- OS: macOS
- Python version: 3.13
- ADK version: 1.7.0
Model Information:
Using gemini-2.5-pro
related related #1591