[Feature Request]: Using registered functions in code execution, useful for non-openai models.

[ekzhu]

Is your feature request related to a problem? Please describe.

Many non-openai models do not support messages with function_call or tool_call field so registered functions in a UserProxyAgent cannot pick them up when the code block in the message contains a function call. For example, below:

user_proxy (to assistant):

What name does machine 511V0001 have? 

assistant (to user_proxy):

 To get the description of machine 511V0001, you can use the `get_description_of_erp_machine` function as follows:

```python
machine_description = get_description_of_erp_machine("511V0001")
print(f"The description of machine 511V0001 is: {machine_description}")
"""

 EXECUTING CODE BLOCK 0 (inferred language is python)...
SIGALRM is not supported on Windows. No timeout will be enforced.
user_proxy (to assistant):

exitcode: 1 (execution failed)
Code output: 
Traceback (most recent call last):
  File "", line 1, in <module>
    machine_description = get_description_of_erp_machine("511V0001")
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'get_description_of_erp_machine' is not defined

Describe the solution you'd like

Solution 1: a new registered reply function for code execution that includes the registered functions and always include them in the code file itself. This requires parsing the function objects and collect relevant imported modules the functions need, as the script must be self-sustained.

Solution 2: a structural response generator on the producer side of the message, which always try to find registered functions in code blocks and extract the arguments to create a JSON object that is compliant with OpenAI's function_call and tool_call schema. #1309

Solution 3: append the full function definition in the system message of the AssistantAgent, and ask them to always include the definition in code blocks they produce. This should be an optional setting for non-openai models only.

Additional context

#1202
#1150
#1323
#1309

[qingyun-wu]

This is a super appealing feature to have! Both solution 1 and solution 2 sound doable! Solution 3 might be tricky as it requires one to twist the system prompt. Effective prompt to use could be model-dependent.

[ekzhu]

It seems Solution 1 is probably most similar to the Skills in AutoGen Studio. I think perhaps we can find a way to unify both.

[yoadsn]

I stumbled on a similar thought process while inspecting the current implementation - Perhaps sharing how I approached it from a design perspective can contribute.

I believe that producers of messages should be responsible to produce a common model-agnostic "tool call request" structure which agents who support execution of such functions can fulfill.
This request structure can be base on the OAI one but does not have to.
Of course, the other side is a common structure for "too call response" which should ideally be routed to request origin or group manager.

Encapsulation should be enforced on both sides - producer (asks to execute tool) must not know the means a consumer (executes the tool) fulfills. (But it can ask for a specific agent to execute if this is known from context)

This should ideally cover:

• function calls of OAI - legacy structure
• tool calls of OAI
• skills in the sense AutoGen studio defines it
• code execution - assuming a function called "execute_code" and params (code, lang, others..) or similar

For clarity, one step towards that design would be to have the "generate_reply" of an agent that uses an LLM to be responsible to extract the "tool call" parts from the LLM completion. for an OAI model, that would be as simple as looking at either "function_call" or "tool_calls" within the "choices.message".
For other models, this could mean parsing code blocks, or asking the help of OAI model to translate a code block to a "tool usage" and so on.

This will improve some existing shortcoming for current state:

• the duality of function/tool request/response which followed OAI evolution of their API structure (move to one common structure)
• Ensure non OAI model can compensate and adapt to that structure without needing to adjust the execution side - the execution is compatible with any request origin
• Code execution which today is a "special" case would all under the same "tooling" concept allowing extension such as the "install required packages" tool which was was suggested in other discussion in the repo.
• Common routing of "tool request" and "tool response" between agents would apply regardless of the models/agesnt involved.

I even started going that path on my own local experiments but nothing that comes close to a PR around that.
Also note that while introducing middelware such as planned in #1240 this would reduce the code needed since the common structure would apply to a single middleware of producer/consumer of tools (in theory).

[ekzhu]

@yoadsn thanks for the design idea.

In the long run I think many models will match GPT-4 capability and should easily generate tool calls or some structured messages that can be parsed and executed. If all agents are GPT-4 then there is no problem.

In the short-term though I think we do need to do some "patching" on the producer side to make it aware of the contract that it should use to interact with the consumer, i.e., some custom instruction in the system prompt, given the type of consumer: vanila code executor, Jupyter kernel, etc. In some cases, we need to wrap the producer behind some structured response generator like Guidance to make sure it follows the contract.

@olimoz has this interesting notebook that demonstrates agents working together on a Jupyter Notebook. All the agents are customized with the contract about how to interact with the notebook. https://github.com/olimoz/AI_Teams_AutoGen/blob/main/JupyterNotebooksForAutoGen.ipynb

[yoadsn]

This is indeed interesting.
IMHO If that was built under the common tool execution paradigm it would have been implemented with tools like "add_cell, update_cell, exec_cell" etc. And the notebook execution kernel would be abstracted behind this.
In the implementation here, the LLM is "taught" a custom semantic to convey intent like "add cell" which is usage-specific.
An LLM that knows to "execute a tool" could use existing semantics and still interact with the NB while at the same time use other tools which are external to the NB and are not "cell based" such as "save_nb_to_file" or even "restart_kernel" without new semantics.
Hope this helps clarify the power in the above design direction.

[ekzhu]

IMHO If that was built under the common tool execution paradigm it would have been implemented with tools like "add_cell, update_cell, exec_cell" etc.

I agree. I think the "patching" of the producer agent would include such tools and their instruction. This can be done with add_to_agent method if some binder class (could be the consumer agent itself) implements the AgentCapability interface.

[ekzhu]

This is very interesting discussion. Thanks @yoadsn! Would you be interested in making a PR with an example of your design idea? The PR doesn't need to make library change, a Jupyter Notebook demonstrating the design idea can already help us move the discussion to the next level. @olimoz is interested in contributing to code execution system design as well.

[yoadsn]

@ekzhu Would love to contribute if I can.
I have moved away from AutoGen for a bit to complete some other pending project, I will attempt to: 1) stay on top of related conversations on this repo 2) Work on a PR or even visual diagrams/contracts as soon as I can.

One caveat, I am not coming from strong Python background so might not meet the coding standards - we'll see about that.

[ekzhu]

We all have something to learn from each other :D

I will put out a PR that allows better extension of code execution (e.g., switching different code execution backend).

[ekzhu]

@yoadsn I opened a new PR that may be able to create customized code executor that can pre-load python modules. From there we can add user-defined functions through this way. Would you like to take a look? #1405