Expose tool calling primitives

[tibbe]

Description

While working on human-in-the-loop (HITL) tool calling (see #1817), I've been implementing my own agent on top of the lower-level pydantic-ai modules (and thus not using the agent module). This means dealing with tool calling more directly.

In a HITL scenario you often want to send the human a representation of what actions the agent would take. A "diff" of a sort. So given a tool function of type:

ToolParams = ParamSpec('ToolParams', default=...)
"""Retrieval function param spec."""

ToolFuncContext = Callable[Concatenate[RunContext[AgentDepsT], ToolParams], Any]
"""A tool function that takes `RunContext` as the first argument.

We want another function with the same arguments but a different return type:

class Diff:
    """A human-readable diff showing what the tool would do."""

ToolFuncDiffContext = Callable[Concatenate[RunContext[AgentDepsT], ToolParams], Diff]

So when the agent wants to call a tool that needs a HITL, we take the arguments we would have passed to the actual tool and instead pass them to this diff function, producing a diff. We then send the diff to the human and, if they approve it, call the actual tool.

To do this well (i.e. without duplicating much of what pydantic-ai does) we need access to a function that, given a tool call (e.g. ToolCallPart) returned by the model, transform it into Python function parameters, so we can use them to call functions other than the tool itself.

In general, I see pydantic-ai as providing (at least) 3 core piece of functionality for tools:

• Given a Python function, create a schema that can be given to the model to describe the available tool call.
• Given a tool call from the model, convert the model response into function arguments that can be passed to the function.
• Given the function's return value, convert it into a format that can be given back to the model.

All these pieces already exist in pydantic-ai, but they aren't (completely) exposed. I think exposing them would make pydantic-ai more usable as a library, which we can build our own agents upon (if agent doesn't work for our use case).

I think this is in fact a strength of the whole pydantic ecosystem: to be able to use pieces to build your own things without having to use a all-or-nothing framework.

References

No response

[DouweM]

@tibbe Does the Toolsets feature that just got merged help? It's not been released yet, but it lets you define your own toolsets and wrapper toolsets that can modify tool call execution: https://github.com/pydantic/pydantic-ai/blob/main/docs/toolsets.md#changing-tool-execution

Deferred Tools are also possibly relevant: https://github.com/pydantic/pydantic-ai/blob/main/docs/toolsets.md#deferred-toolset

[tibbe]

@tibbe Does the Toolsets feature that just got merged help? It's not been released yet, but it lets you define your own toolsets and wrapper toolsets that can modify tool call execution: https://github.com/pydantic/pydantic-ai/blob/main/docs/toolsets.md#changing-tool-execution

Deferred Tools are also possibly relevant: https://github.com/pydantic/pydantic-ai/blob/main/docs/toolsets.md#deferred-toolset

@DouweM thanks for working on this!

In the context of this particular feature request (which is more narrow than what's discussed in e.g. #1995), I'm not sure if Toolset addresses this issue. Perhaps ToolSet can be used to implement what I want but it's not obvious to me how that would work. Let me try to clarify with some code what kind of "low-level" features I'm asking for here. Quoting myself above:

• Given a Python function, create a schema that can be given to the model to describe the available tool call.

I want a function which type is along the lines of:

def get_tool_definition(func: Callable[..., Any]) -> ToolDefinition:
    ...

This would use the existing pydantic-ai code for extracting the ToolDefinition schema by looking at the functions type and docstring.

• Given a tool call from the model, convert the model response into function arguments that can be passed to the function.

I want a function which type is along the lines of:

def get_tool_call_args(part: ToolCallPart) -> tuple[tuple[Any, ...], dict[str, Any]]:
    ...

This isn't quite e.g. ToolCallPart.args_as_dict() as there's additional logic for e.g. separating args from kwargs (which probably requires looking at e.g. the ToolDefinition.

• Given the function's return value, convert it into a format that can be given back to the model.

Relatively similar to the above, I want a function which type is along the lines of:

def get_tool_call_result(func_res: Any) -> ToolReturn:
    ...

Summary:

This is all about breaking pydantic-ai into more reusable pieces that can be used to build both the existing Agent but also any other agent the user wants to.

The three functions above are all lower-level than Agent, meaning that they could be used to write your own Agent. If Toolset is tied to Agent then it's probably not the right fit for what I'm asking for here.

[DouweM]

@tibbe Thanks, I was a bit too quick with my original comment as the deferred tool stuff turned out to be useful to others with related HITL tool approval needs and I wasn't sure if you "really" needed these primitives or if that had been in search of a workaround in place of a native solution (that deferred tools may provide).

Reading more closely now, and with your specific examples, it sounds like most of what you're looking for is implemented in https://github.com/pydantic/pydantic-ai/blob/main/pydantic_ai_slim/pydantic_ai/_function_schema.py.

Given a Python function, create a schema that can be given to the model to describe the available tool call.

from pydantic_ai._function_schema import function_schema
function_schema(function, GenerateToolJsonSchema).json_schema

Given a tool call from the model, convert the model response into function arguments that can be passed to the function.

from pydantic_ai._function_schema import function_schema
function_schema(function, GenerateToolJsonSchema)._call_args(tool_call_part.args_as_dict(), ctx)

Given the function's return value, convert it into a format that can be given back to the model.

from pydantic_ai.messages import tool_return_ta
tool_return_ta.dump_python(self.content, mode='json')

Obviously these are private and not very ergonomic right now, but I agree it makes sense to expose these abilities publicly. You'll know better what you'll want them to look like to fit into your own agent loop implementation, so I encourage you to play around with them and submit a PR to expose them publicly!