Tool Call Accuracy Evaluator

[singankit]

Description

Adding ToolCallAccuracyEvaluator Evaluator

• Single Tool Call Example

tool_call_accuracy_evaluator = ToolCallAccuracyEvaluator(model_config=model_config)
response = tool_call_accuracy_evaluator(
            query="How is the weather in New York?",
            response="The weather in New York is sunny.",
            tool_calls={
                "type": "tool_call",
                "tool_call": {
                    "id": "call_eYtq7fMyHxDWIgeG2s26h0lJ",
                    "type": "function",
                        "function": {
                            "name": "fetch_weather",
                            "arguments": {
                                "location": "New York"
                            }
                        }
                }
            },
            tool_definitions={
                "id": "fetch_weather",
                "name": "fetch_weather",
                "description": "Fetches the weather information for the specified location.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                        "type": "string",
                        "description": "The location to fetch weather for."
                        }
                    }
                }
            }
        )

Output

{
    "tool_call_accuracy": 1.0,
    "per_tool_call_details": [
        {
            "tool_call_accurate": true,
            "tool_call_accurate_reason": "The input Data should get a Score of 1 because the TOOL CALL is directly relevant to the user's query about the weather in New York, has the correct parameters, and the values are accurately extracted from the CONVERSATION.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        }
    ]
}

• Multiple Function Tool Call

tool_call_accuracy_evaluator = ToolCallAccuracyEvaluator(model_config=model_config)
response = tool_call_accuracy_evaluator(
            query="How is the weather in New York?",
            tool_calls=[{
                "type": "tool_call",
                "tool_call": {
                    "id": "call_eYtq7fMyHxDWIgeG2s26h0lJ",
                    "type": "function",
                        "function": {
                            "name": "fetch_weather",
                            "arguments": {
                                "location": "Seattle"
                            }
                        }
                    }
                },
                {
                "type": "tool_call",
                "tool_call": {
                    "id": "call_eYtq7fMyHxDWIgeG2s26h0lJ",
                    "type": "function",
                        "function": {
                            "name": "fetch_weather",
                            "arguments": {
                                "location": "New York"
                            }
                        }
                    }
                }
                ],
            tool_definitions={
                "id": "fetch_weather",
                "name": "fetch_weather",
                "description": "Fetches the weather information for the specified location.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                        "type": "string",
                        "description": "The location to fetch weather for."
                        }
                    }
                }
            }
        )

Output

{
    "tool_call_accuracy": 0.5,
    "per_tool_call_details": [
        {
            "tool_call_accurate": false,
            "tool_call_accurate_reason": "The TOOL CALL is irrelevant to the user's inquiry about the weather in New York, as it attempts to fetch weather information for Seattle instead. The parameters used do not match the user's request, and the values are not appropriate or correct. Thus, the score is 0.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        },
        {
            "tool_call_accurate": true,
            "tool_call_accurate_reason": "The TOOL CALL is directly relevant to the user's inquiry about the weather in New York, has the correct parameters and values, and is likely to provide useful information. Thus, it deserves a score of 1 for being relevant and useful.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        }
    ]
}

If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[stevepon]

Is this supposed to be a common util? What if the score is a continuos float 0-1?

[singankit]

It can be overridden for each evaluator if needed. I made this configurable here to make it reusable if possible. For all existing evaluators we have score which are whole number and do not have decimals. But can be untrue for new evaluators.

[stevepon]

I guess the only thing I'd recommend then is a different variable name -- "valid_score_regex" or something. But this is not a big deal, feel free to ignore.

[stevepon]

Are we sure we want this to be an exception? As a user, I could see just wanting to call this evaluator after every agent run. I don't want to have to pre-filter on the client side, and adding try-catch also seems a bit cumbersome. Maybe instead there is a benign "no tools were called" response we could give?

[singankit]

Either this has to be done before or after the response to understand if the evaluator gave a response that is valid. I would prefer to enforce in on input side.

[stevepon]

This line is confusing to me. Has it been used successfully elsewhere?

[singankit]

Existing evaluators use it. I built this on top of the prompt from existing evaluators.

[changliu2]

Description

Adding ToolCallAccuracyEvaluator Evaluator

tool_call_accuracy_evaluator = ToolCallAccuracyEvaluator(model_config=model_config)
response = tool_call_accuracy_evaluator(
            query="How is the weather in New York?",
            response="The weather in New York is sunny.",
            tool_calls={
                "type": "tool_call",
                "tool_call": {
                    "id": "call_eYtq7fMyHxDWIgeG2s26h0lJ",
                    "type": "function",
                        "function": {
                            "name": "fetch_weather",
                            "arguments": {
                                "location": "New York"
                            }
                        }
                }
            },
            tool_definitions={
                "id": "fetch_weather",
                "name": "fetch_weather",
                "description": "Fetches the weather information for the specified location.",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                        "type": "string",
                        "description": "The location to fetch weather for."
                        }
                    }
                }
            }
        )

Output

{
    "tool_call_accuracy": 1.0,
    "tool_call_accuracy_reason": "The input Data should get a Score of 1 because the TOOL CALL is directly relevant to the user's inquiry about the weather in New York, uses the correct parameters, and the parameter values are accurate and contextually appropriate."
}

If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

can you also provide an example of 2 function calls evaluated?

[stevepon]

Do you want to say that the "TOOL CALL parameters include information that is not presented in the CONVERSAION"?

[singankit]

Yes and also if a tool was called with a parameter that is not in the tool definition

[stevepon]

I made the same comment on Jose's... that if we want to move toward json mode or structured output we might need to revise this approach, right? (And also that a potential optimization is to remove the chain of thought piece in since the explanation can essentially play the same role, and then just have the LLM write the explanation field in the json response first.)

[singankit]

While I was writing the prompt the chain of thought was helpful for me to debug. But agree if we do not expose it to customer should we ask LLM to generate it. Once we move to structured output yes we will need to revise it.

[singankit]

@changliu2 To help close on the keys for the output of this evaluator

Current

{
    "tool_call_accuracy": 0.5,
    "per_tool_call_details": [
        {
            "tool_call_accurate": false,
            "tool_call_accurate_reason": "The TOOL CALL is irrelevant to the user's inquiry about the weather in New York, as it attempts to fetch weather information for Seattle instead. The parameters used do not match the user's request, and the values are not appropriate or correct. Thus, the score is 0.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        },
        {
            "tool_call_accurate": true,
            "tool_call_accurate_reason": "The TOOL CALL is directly relevant to the user's inquiry about the weather in New York, has the correct parameters and values, and is likely to provide useful information. Thus, it deserves a score of 1 for being relevant and useful.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        }
    ]
}

Proposed

{
    "percentage_tool_call_accurate": 0.5,
    "tool_call_correctness_label": True  # Chang to help close of this name 
    "per_tool_call_details": [
        {
            "tool_call_accurate": false,
            "tool_call_accurate_reason": "The TOOL CALL is irrelevant to the user's inquiry about the weather in New York, as it attempts to fetch weather information for Seattle instead. The parameters used do not match the user's request, and the values are not appropriate or correct. Thus, the score is 0.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        },
        {
            "tool_call_accurate": true,
            "tool_call_accurate_reason": "The TOOL CALL is directly relevant to the user's inquiry about the weather in New York, has the correct parameters and values, and is likely to provide useful information. Thus, it deserves a score of 1 for being relevant and useful.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        }
    ]
}

[changliu2]

@changliu2 To help close on the keys for the output of this evaluator

Current

{
    "tool_call_accuracy": 0.5,
    "per_tool_call_details": [
        {
            "tool_call_accurate": false,
            "tool_call_accurate_reason": "The TOOL CALL is irrelevant to the user's inquiry about the weather in New York, as it attempts to fetch weather information for Seattle instead. The parameters used do not match the user's request, and the values are not appropriate or correct. Thus, the score is 0.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        },
        {
            "tool_call_accurate": true,
            "tool_call_accurate_reason": "The TOOL CALL is directly relevant to the user's inquiry about the weather in New York, has the correct parameters and values, and is likely to provide useful information. Thus, it deserves a score of 1 for being relevant and useful.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        }
    ]
}

Proposed

{
    "percentage_tool_call_accurate": 0.5,
    "tool_call_correctness_label": True  # Chang to help close of this name 
    "per_tool_call_details": [
        {
            "tool_call_accurate": false,
            "tool_call_accurate_reason": "The TOOL CALL is irrelevant to the user's inquiry about the weather in New York, as it attempts to fetch weather information for Seattle instead. The parameters used do not match the user's request, and the values are not appropriate or correct. Thus, the score is 0.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        },
        {
            "tool_call_accurate": true,
            "tool_call_accurate_reason": "The TOOL CALL is directly relevant to the user's inquiry about the weather in New York, has the correct parameters and values, and is likely to provide useful information. Thus, it deserves a score of 1 for being relevant and useful.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        }
    ]
}

@singankit, let's stick to the current convention pattern (key+ "_label"):

{
    "tool_call_accuracy": 0.5,
    "tool_call_accuracy_label": True,
    "per_tool_call_details": [
        {
            "tool_call_accurate": false,
            "tool_call_accurate_reason": "The TOOL CALL is irrelevant to the user's inquiry about the weather in New York, as it attempts to fetch weather information for Seattle instead. The parameters used do not match the user's request, and the values are not appropriate or correct. Thus, the score is 0.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        },
        {
            "tool_call_accurate": true,
            "tool_call_accurate_reason": "The TOOL CALL is directly relevant to the user's inquiry about the weather in New York, has the correct parameters and values, and is likely to provide useful information. Thus, it deserves a score of 1 for being relevant and useful.",
            "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ"
        }
    ]
}