.NET: How is the need for user interaction supposed to be recognized in workflows?

[stephentoub]

An AIAgent can return back various kinds of content that represent the need for the caller to do something in order to make forward progress. This could be FunctionCallContent without a paired FunctionResultContent, or UserInputRequestContent without a corresponding UserInputResponseContent, or any other custom content type a user may construct.

How is a workflow that involves AIAgents supposed to handle this? e.g. if I have a workflow that connects agent A to agent B, and the response from A includes a UserInputRequestContent that needs reaction, what is it supposed to do? And how does an arbitrary workflow (such as one from a reusable workflow builder that provides patterns like sequential, handoff, et.c) supposed to recognize / differentiate such things?

Should we special-case UserInputRequestContent and only that, and have FunctionCallContent changed to derive from UserInputRequestContent?

And in the case where user input is required, invariably the response will need to be handled by the same sub-agent that created the request. How will the user's response be handled so as to get it back to that node in the graph?

cc: @westey-m

[ekzhu]

I think in the case of "multi-agent workflows", as in nodes are agents, the workflow should suspend and request for input from caller if an agent emits a user input request.

The orchestration patterns (sequential, concurrent, magentic, etc) with agents in them should wire this up for the user.

[stephentoub]

the workflow should suspend and request for input from caller if an agent emits a user input request.

Sure. But:
a) how? Does the Python implementation currently handle this? What handles checkpointing the state, how is that state made available to the caller, how does the caller resume the workflow in the right place and with the additional response input? etc.
b) what constitutes a "user input request"? That needs to be formally defined, if all such packaged compositions are going to respect it.

[ekzhu]

a) how? Does the Python implementation currently handle this?

It is not currently implemented -- but we should definitely do it as the recent addition of agent middleware will introduce the real possibilities of getting user input request content. Here is the issue to track the implementation: #862

What handles checkpointing the state, how is that state made available to the caller, how does the caller resume the workflow in the right place and with the additional response input?

In Python, we will use the workflow's mechanism for requesting user input. See examples for normal workflows (not multi-agent workflows) here: https://github.com/microsoft/agent-framework/blob/main/python/samples/getting_started/workflow/human-in-the-loop/guessing_game_with_human_input.py

For .NET see this example: https://github.com/microsoft/agent-framework/blob/main/dotnet/samples/GettingStarted/Workflows/HumanInTheLoop/HumanInTheLoopBasic/WorkflowHelper.cs

b) what constitutes a "user input request"?

It should be subclasses of BaseUserInputRequest, e.g., FunctionApprovalUserInputRequest.

[stephentoub]

It should be subclasses of BaseUserInputRequest, e.g., FunctionApprovalUserInputRequest.

Yes, that's the option I was referring to with "Should we special-case UserInputRequestContent and only that, and have FunctionCallContent changed to derive from UserInputRequestContent?"

Right now .NET's FunctionCallContent does not derive from UserInputRequestContent (dotnet/extensions#6831). I'm suggesting we need a single, extensible signal that indicates to consumers (in this case, agent workflow implementations) that there's content that requires input external to the workflow in order to continue. If an FCC passes through the function invocation loop (for whatever reason, the loop couldn't handle it, such as if the user requested it not), then it's also requiring of external input.

Here is the issue to track the implementation: #862

@lokitoth, @alliscode, I assume we're tracking this and implementing the .NET side of things in lock-step?

we will use the workflow's mechanism for requesting user input

@lokitoth, @alliscode, does this exist for .NET as well and I'm just missing it?

[westey-m]

I do think agents that return FunctionCallContent is very much an edge case. With an agent-agnostic workflow especially, the workflow wouldn't necessarily know anything about the functions that a provided agent uses, so calling them is problematic. Also, per-invocation functions are not supported by all Agent types and therefore requires breaking out of the abstraction, which again is problematic with agent-agnostic workflows.

That's said, I do support the idea of having InputRequestContent instead of UserInputRequestContent and having FunctionCallContent also inherit from InputRequestContent. For any cases where we do require input, regardless of where it is in the stack, having a common way of identifying all input request types is very useful.

[markwallace-microsoft]

Expect this to be a P0 for v1.0

[TaoChenOSU]

This is how Python currently does it: #2022