Pipecat and multi agent structure question

[tropxy]

Question

Hi, I come with a question on integrating pipecat into my project.
I have the need to create an agent, let's call it Planner, which is the agent that coordinates what to do and which agents to call.
If I want to integrate pipecat and once we have a user text input to send a summary of what is being said to the Planner, so the planner can decide what flow to trigger, can I do that with pipecat?
If I use the function call option will the LLM of pipecat framework properly wait for the answer from the other agent?

Basically what I am talking about here is a multi agent project where the pipecat with daily would be in, but not necessarily the other agents are built in the same framework (I could use pipecat and autogen for instance).
any help is appreciated, thank you!

[chadbailey59]

This is definitely possible with Pipecat. There are a few ways you could build it, but they will all use function calling with the Pipecat LLM.

The simplest version, off the top of my head: When you create your Pipecat LLM service, you'd include a special tool call definition in your system prompt, named something like call_planner. This allows the Pipecat LLM to interact with the user as much as necessary to eventually decide it has enough info to 'ask the planner' what to do next. You'd create a function handler for that call_planner function that sent a request to the planner through whatever API it's using. Your function handler would take the response from the planner and use it to determine what to say to the user as a result.

For more complex interactions, you may want to look at Pipecat Flows. It's a really powerful way to define conversation flows. You can even build flows dynamically, so your Planner could return Pipecat Flow nodes instead of just LLM responses.

[tropxy]

This is definitely possible with Pipecat. There are a few ways you could build it, but they will all use function calling with the Pipecat LLM.

The simplest version, off the top of my head: When you create your Pipecat LLM service, you'd include a special tool call definition in your system prompt, named something like call_planner. This allows the Pipecat LLM to interact with the user as much as necessary to eventually decide it has enough info to 'ask the planner' what to do next. You'd create a function handler for that call_planner function that sent a request to the planner through whatever API it's using. Your function handler would take the response from the planner and use it to determine what to say to the user as a result.

For more complex interactions, you may want to look at Pipecat Flows. It's a really powerful way to define conversation flows. You can even build flows dynamically, so your Planner could return Pipecat Flow nodes instead of just LLM responses.

Thanks for that, in fact I found not too soon after publishing my question pipecat-flows and I am using it now and fits pretty well to my needs, thanks a lot! 😎