Skip to content

TrajectoryEvaluator fails with dynamic authentication tokens in OpenAPIToolset #2083

@sedghi

Description

@sedghi

Describe the bug
The ADK's TrajectoryEvaluator performs exact comparison of tool arguments including authentication tokens that change on every run. When using OpenAPIToolset with OAuth2/Bearer authentication, the auth token is included in tool arguments with the prefix _auth_prefix_vaf_Authorization. Since these tokens are dynamic, evaluation always fails even when the tool calls are functionally identical.

To Reproduce
Steps to reproduce the behavior:

  1. Create an agent using OpenAPIToolset with OAuth2 authentication
  2. Record an evalset with a tool call that includes auth parameters
  3. Run adk eval on the agent
  4. See evaluation failure due to mismatched auth tokens

Example evalset snippet:

"tool_uses": [
  {
    "id": "adk-437d690f-5a5e-46ed-a47e-0dd374cdf7b5",
    "args": {
      "org_id": "my-org.com",
      "_auth_prefix_vaf_Authorization": "Bearer eyJraWQ..."
    },
    "name": "list_conditions"
  }
]

Expected behavior
The evaluator should ignore authentication parameters (prefixed with _auth_prefix_) when comparing tool trajectories, as these are implementation details and not part of the actual tool logic.

Desktop (please complete the following information):

  • OS: macOS
  • Python version: 3.13
  • ADK version: 1.7.0

Model Information:
Using gemini-2.5-pro

Image

related related #1591

Metadata

Metadata

Labels

answered[Status] This issue has been answered by the maintainereval[Component] This issue is related to evaluationrequest clarification[Status] The maintainer need clarification or more information from the author

Type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions