GPTAssistantAgent cannot find the function needed for Function Calling

[atcelen]

Hello everybody,

I am having some issues with the GPTAssistantAgents in the way that they cannot find the function stated in their JSON schema.
I get the following error:

INFO:autogen.agentchat.contrib.gpt_assistant_agent:Intermediate executing(check_for_collisions, Sucess: False) : Error: Function check_for_collisions not found.
INFO:autogen.agentchat.contrib.gpt_assistant_agent:Intermediate executing(check_for_collisions, Sucess: False) : Error: Function check_for_collisions not found.

Here is a link to my code:
https://colab.research.google.com/drive/1SgibCLvf0HLnTxfWMzaPbEPLYRfZw4RU?usp=sharing

Thanks in advance,
Ata

[rickyloynd-microsoft]

@gagb fyi
@atcelen, this could be related to a recent openai release that broke autogen's support for function calling. We are working on #791 to fix that.

[sonichi]

@IANTHEREAL @kevin666aa

[IANTHEREAL]

It's wired, openai response a wrong function name. Driven by curiosity, your script runned well both on openai 1.2.3 and 1.3.7. 🤔

@atcelen if you can reproduce it consistently, can you provide a raw complete openai response while calling check_for_collisions?

--------------------------------------------------------------------------------

>>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is python)...
execute_code was called without specifying a value for use_docker. Since the python docker package is not available, code will be run natively. Note: this fallback behavior is subject to change
Executor (to chat_manager):

exitcode: 0 (execution succeeded)
Code output: 


--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION check_function...
Collision_agent (to chat_manager):

APPROVED

The placement of "desk_1" does not cause any collisions with other objects in the room.


--------------------------------------------------------------------------------

>>>>>>>> EXECUTING FUNCTION check_function...
Checking for discrepancies between desk_1 and north_wall...
Preposition : in front, is_touching : False
Discrepancy_agent (to chat_manager):

NOT APPROVED

I am unable to verify the placement of "desk_1" in relation to the north wall as there seems to be a file access error, preventing me from confirming whether the desk's location description matches its actual position in the room. The error suggests that the specified file for "desk_1" could not be found in the directory. Therefore, until the verification can be completed, the placement cannot be approved.


--------------------------------------------------------------------------------
Engineer (to chat_manager):

[atcelen]

Thank you very much for the fast response! I have tried using openai 1.3.7 as well but I am still having the issue. You can find the complete agent conversation below. Please let me know if you need any more information from my side:

Admin (to chat_manager):


        The room is 4m x 4m x 2.5m. The length, width and height in 'size_in_meters' represents x, y, z dimensions respectively.
        The lower southwest corner of the room is the origin ('x' : 0.0, 'y' : 0.0, 'z' : 0.0). The position of each object is
        its center point!

        Objects already in the room:
        []

        Position this JSON object in the room:
        ```json
        {'item_id': 'desk_1', 'itemType': 'desk', 'location_description': 'centered on the floor, near the north wall', 'size_in_meters': {'length': 1.0, 'width': 0.5, 'height': 0.75}}
        ```
        

--------------------------------------------------------------------------------
Interior_designer (to chat_manager):

To position the desk described in the JSON object according to the location description "centered on the floor, near the north wall," we need to consider the dimensions of the room and the size of the desk.

The room is 4m x 4m, and the desk dimensions are 1.0m in length, 0.5m in width, and 0.75m in height. Since the desk needs to be centered on the floor, we will place it in the middle of the room along the east-west axis (x-axis). The room's width is 4m, so the center along the x-axis is at 2.0m. The desk's length is 1.0m, so its center point will be at 2.0m on the x-axis.

The desk should be near the north wall, which is along the y-axis at 4.0m. To avoid placing the desk directly against the wall, we can assume a small buffer, let's say 0.1m from the wall. Since the desk's width is 0.5m, its center point will be at 4.0m (north wall) - 0.1m (buffer) - 0.25m (half the width of the desk) = 3.65m on the y-axis.

The desk is on the floor, so its center point on the z-axis will be half its height, which is 0.75m / 2 = 0.375m.

Therefore, the position of the desk in the room will be as follows:
- x: 2.0m (centered along the room's width)
- y: 3.65m (near the north wall with a small buffer)
- z: 0.375m (centered in height)

The JSON object with the position included would look like this:

```json
{
  "item_id": "desk_1",
  "itemType": "desk",
  "location_description": "centered on the floor, near the north wall",
  "size_in_meters": {
    "length": 1.0,
    "width": 0.5,
    "height": 0.75
  },
  "position_in_meters": {
    "x": 2.0,
    "y": 3.65,
    "z": 0.375
  }
}

---

Engineer (to chat_manager):

import json

# Define the desk object with the calculated position
desk_object = {
    "item_id": "desk_1",
    "itemType": "desk",
    "location_description": "centered on the floor, near the north wall",
    "size_in_meters": {
        "length": 1.0,
        "width": 0.5,
        "height": 0.75
    },
    "position": {
        "x": 2.0,
        "y": 3.65,
        "z": 0.375
    },
    "rotation": {
        "z_angle": 0.0  # Assuming no rotation is needed as it was not specified
    }
}

# Save the JSON object under the "item_id" name
with open(f"{desk_object['item_id']}.json", 'w') as json_file:
    json.dump(desk_object, json_file)

---

EXECUTING CODE BLOCK 0 (inferred language is python)...
WARNING:autogen.code_utils:execute_code was called without specifying a value for use_docker. Since the python docker package is not available, code will be run natively. Note: this fallback behavior is subject to change
Executor (to chat_manager):

exitcode: 0 (execution succeeded)
Code output:

---

INFO:autogen.agentchat.contrib.gpt_assistant_agent:Intermediate executing(check_for_collisions, Sucess: False) : Error: Function check_for_collisions not found.
INFO:autogen.agentchat.contrib.gpt_assistant_agent:Intermediate executing(check_for_collisions, Sucess: False) : Error: Function check_for_collisions not found.
INFO:autogen.agentchat.contrib.gpt_assistant_agent:Intermediate executing(check_for_collisions, Sucess: False) : Error: Function check_for_collisions not found.
Collision_agent (to chat_manager):

It appears that a system error occurred when attempting to check for collisions. As my available functions indicate, there should be a function named check_for_collisions within the functions namespace that assesses if there are any collisions between objects. Unfortunately, I received an error stating that this function could not be found.

I will attempt to correct the issue and run the check again. If the issue persists, it may be necessary to report this problem for further technical support. Let's try once more.

It seems there is a persistent issue with accessing the check_for_collisions function within the functions namespace. The function does not seem to be recognized by the system, resulting in repeated errors. This problem may need to be addressed by technical support to ensure that the available tools can function as intended.

At present, it is not possible for me to verify whether the placed object causes any collisions in the room. The standard procedure to return "APPROVED" or "NOT APPROVED" cannot be completed without the ability to check for collisions using the designated function.

---

INFO:autogen.agentchat.contrib.gpt_assistant_agent:Intermediate executing(check_for_discrepancies, Sucess: False) : Error: Function check_for_discrepancies not found.
Discrepancy_agent (to chat_manager):

There appears to be a misunderstanding or a technical error in accessing and utilizing the required function. The function named check_for_discrepancies within the functions namespace that is supposed to assess the correctness of the object's location description and its position in the room cannot be located, resulting in error messages.

It seems there might be an issue with the system's function library or a misunderstanding regarding the available functions. It may be necessary for technical support or administration to review and resolve this problem, as I am currently unable to perform the check for discrepancies using the function as intended.

Due to this technical limitation, I am unable to verify the appropriateness of the object's placement based on the description provided without the designated function. The procedure to confirm the object's position is "APPROVED" or "NOT APPROVED" cannot be fully executed at this time.

---

INFO:autogen.agentchat.contrib.gpt_assistant_agent:Intermediate executing(check_for_collisions, Sucess: False) : Error: Function check_for_collisions not found.
Collision_agent (to chat_manager):

There is a technical issue accessing the check_for_collisions function, and this error prevents me from performing the necessary check for collisions. Regrettably, without this function operating correctly, I cannot determine whether the placed object causes any collisions with other objects in the room.

It is advisable to have this issue reported and resolved by technical support to proceed with the collision check as required.

---

INFO:autogen.agentchat.contrib.gpt_assistant_agent:Intermediate executing(check_for_discrepancies, Sucess: False) : Error: Function check_for_discrepancies not found.
Discrepancy_agent (to chat_manager):

I apologize for the confusion earlier. I will now reassess the situation correctly.

Given the provided description, the desk placement of "centered on the floor, near the north wall" implies that desk_1 should be located at the central x-coordinate of the room and near the maximum y-coordinate without touching the north wall. Let’s use the provided function to verify if there’s any discrepancy between the location description and the position of the desk.

I will call the check_for_discrepancies function for you.

It appears that there has been an error in attempting to use the check_for_discrepancies function to assess the correctness of the desk's location description compared to its position in the room. Since the function could not be found, I am unable to provide a verification through the expected means.

While I am unable to use the designated function to check for discrepancies, based on the standard room description guidelines, the provided location description of "centered on the floor, near the north wall" for the desk seems to be reasonable given the room's dimensions and assuming no other objects are in the room to conflict with this placement. However, without running the function, I cannot provide an official "APPROVED" or "NOT APPROVED" response.

Given the technical difficulty in accessing the required function, further steps need to be taken to resolve the issue so that the proper verification can be completed.

---

Provide feedback to chat_manager. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: exit

[IANTHEREAL]

Please check your Functions configuration of your assistant in https://platform.openai.com/playground

[atcelen]

I think I have an idea of where the error stems from. Every time when I create a GPTAssistantAgent, OpenAI seems to cache an instance of this agent under "Assistants" (under the name collision_agent for example). However, whenever you make adjustments to this agent in your code, the program doesn't make the necessary adjustments to the agent and instead uses the cached version under this name.

The first time I created the collision_agent, I assigned it the function called "check_for_collision". Afterwards, I changed this function name to "check_function" but since the cached version of the agent was being used, the agent was still searching for "check_for_collision". The reason why you couldn't replicate this error was that you weren't able to replicate this error was that you were running the agent for the first time probably.

On the other hand, I have realized that previously I was able to create multiple agents under the same name but at some time between 24 Nov. - 5 Dec. this behavior seems to have changed.

In this snapshot, you can see that this agent was created at time 1701781314 (Tue Dec 05 2023 14:01:54 GMT+0100 (Central European Standard Time)), which was three days ago.

In this snapshot, you can see all the cached agents in my OpenAI account:

[atcelen]

The lines 58-64 in "gpt_assistant_agent.py" seems to be the root of my problem. The lines search for a cached assistant if no assistant_id is given.

[IANTHEREAL]

@atcelen Yes, That was my reasoning last night because there was no other possibility.. I will add warning message

[atcelen]

Also, is it possible to update the cached assistants without creating new ones somehow?

[IANTHEREAL]

The name of the assistant is not unique, meaning OpenAI allows two assistants with the same name, but they have different assistant IDs. Since we still can't support a selection mechanism, it's not quite suitable for updates.