An expensive bug GPT4. no matter what I do

[twobob]

I set this up on a new machine today.
I realised that in 2 games I had suddenly spent 7 dollars instantly. Pretty steep considering it was supposed to be using 3.5-turbo-16k

Checked over all the setting. Then parsed the entire source code for ANY reference to default models. anything. Changed every single reference. No matter what I set it is using the GPT4 API endpoint.

I have tried; the usual inline declarations; using the built in preferred template; just jamming it in as the hardcoded value to the method that checks it. So.. unless anyone has any other ideas. I'm all out.

7 bucks is no big shakes but I run systems in parallel and no doubt others batch too. Wouldn't want anyone to get an AMAZON AWS level surprise and not have reported. so. Enjoy. Reported.

EDIT: It's a linux box but I did install it under windows on there as well and test it (that test is probably the top 50c of the green bar ) It was identical so "not OS". and "yes, I also tried that."

[twobob]

Green 2 games. The blue bit, about 10 others....

[afourney]

Thanks for reporting and sorry about the spend. It sounds like you did due diligence on debugging, so I’m not sure how it was charged to GPT-4. We’ve not seen it call the wrong models before, so I am very curious to try and replicate this.

[victordibia]

To help reproduce, would be great to see

• Your code
• Emphasis on how you specify your OAI_CONFIG_LIST (the data structure used to define which LLM is used).

[LittleLittleCloud]

Maybe here? Looks like the default model is set to GPT-4

autogen/autogen/code_utils.py

Line 18 in 20d77a1

DEFAULT_MODEL = "gpt-4"

[afourney]

@LittleLittleCloud No I don't think that's it. That default is only accessed from the improve_code, improve_function, and implement calls -- and they aren't part of the main control path.

I am eyeing this:

autogen/autogen/oai/openai_utils.py

Line 331 in 20d77a1

# Ensure default models are always considered

   # Ensure default models are always considered
    default_models = ["gpt-4", "gpt-3.5-turbo"]

    for model in default_models:
        # Only assign default API key if the model is not present in the map.
        # If model is present but set to invalid/empty, do not overwrite.
        if model not in model_api_key_map:
            model_api_key_map[model] = "OPENAI_API_KEY"

@twobob How are you loading the config list? Are you using .env?

[twobob]

[
{
"model": "gpt-3.5-turbo-16k-0613",
"api_key": "ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA"
},
{
"model": "gpt-3.5-turbo-16k",
"api_key": "ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA"
},
{
"model": "gpt-3.5-turbo-0613",
"api_key": "ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA"
},
{
"model": "gpt-3.5-turbo",
"api_key": "ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA"
},
{
"model": "gpt-4", <--- -the single remaining reference - and its for azure so ¯\(ツ)/¯
"api_key": "",
"api_base": "",
"api_type": "azure",
"api_version": "2023-07-01-preview"
},
{
"model": "gpt-3.5-turbo",
"api_key": "",
"api_base": "",
"api_type": "azure",
"api_version": "2023-07-01-preview"
}
]

[twobob]

thanks for all the feedback. Yes there is zero reference to the word gpt-4 in my code now so
DEFAULT_MODEL = "gpt-4"
would have already been changed. I literally gutted it

[twobob]

@LittleLittleCloud No I don't think that's it. That default is only accessed from the improve_code, improve_function, and implement calls -- and they aren't part of the main control path.

I am eyeing this:

autogen/autogen/oai/openai_utils.py

Line 331 in 20d77a1

# Ensure default models are always considered

   # Ensure default models are always considered
    default_models = ["gpt-4", "gpt-3.5-turbo"]

    for model in default_models:
        # Only assign default API key if the model is not present in the map.
        # If model is present but set to invalid/empty, do not overwrite.
        if model not in model_api_key_map:
            model_api_key_map[model] = "OPENAI_API_KEY"

@twobob How are you loading the config list? Are you using .env?

me@mebuntu:~/repo/autogen$ echo $OPENAI_API_KEY
ssdsdsdsdsdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA

my env is indeed setup as well as the config file

[twobob]

To help reproduce, would be great to see

* Your code

* Emphasis on how you specify your OAI_CONFIG_LIST (the data structure used to define which LLM is used).

hi , which code. its the default code gutted to not contain any references to gpt_4 pretty much?

[sonichi]

Could you show the code that contains llm_config? If it contains "config_list" in it, could you also show the code which sets config_list?

[twobob]

I forgot to mention but I DID do this first on a windows box and wa using bash as my interpreter because "inline commands working for windows would be cool to do for everyone" I thought. That was when I noticed the bug.

Thought it was /maaaaybe/ the funky bash hack that was causing the error
Hence it being a vanilla ubuntu - like fresh for this job - and a very unhacked autogen other than the gutting.

Like this is the dream support box. literally nothing has happened on it other than this. since I have installed chatdev on the same box but meh. these tests were before that and nothing has changed in terms of the autogen anyway.

So there. the full sordid details. Honestly not sure where to go from there

It's not account since chatdev can use 3.5 just fine from the ubunutu install (why I installed chatdev after)

[twobob]

Could you show the code that contains llm_config? If it contains "config_list" in it, could you also show the code which sets config_list?

hi. it is the defaults straight from the readme or from a notebook, does it write logs anywhere? Ive mounted onto the windows drive from ubuntu so I can access anything...

[twobob]

i ran agentchat_groupchat_research.ipynb from the notebook folder in jupyter notebook

in the beginning I was running it

{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import autogen\n",
"\n",
"config_list_gpt4 = autogen.config_list_from_json(\n",
" "OAI_CONFIG_LIST",\n",
" filter_dict={\n",
" "model": ["gpt-4-32k", "gpt-4-32k-0314", "gpt-4-32k-v0314"],\n",
" },\n",
")\n",
"\n",
"config_list_gpt3 = autogen.config_list_from_json(\n",
" "OAI_CONFIG_LIST",\n",
" filter_dict={\n",
" "model": ["gpt-3.5-turbo-16k-0613", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-0613"],\n",
" },\n",
")"
]
},

and only the engineer was actually given gpt 4 access

"gpt4_config = {\n",
" "seed": 42, # change the seed for different trials\n",
" "temperature": 0,\n",
" "config_list": config_list_gpt4,\n",
" "request_timeout": 120,\n",
"}\n",
"gpt3_config = {\n",
" "seed": 42, # change the seed for different trials\n",
" "temperature": 0,\n",
" "config_list": config_list_gpt3,\n",
" "request_timeout": 90,\n",
"}\n",
"user_proxy = autogen.UserProxyAgent(\n",
" name="Admin",\n",
" system_message="A human admin. Interact with the planner to discuss the plan. Plan execution needs to be approved by this admin.",\n",
" code_execution_config=False,\n",
")\n",
"engineer = autogen.AssistantAgent(\n",
" name="Engineer",\n",
" llm_config=gpt3_config,\n",

but as shown there by the end all actors had been reduced to gpt_3config

[twobob]

@LittleLittleCloud No I don't think that's it. That default is only accessed from the improve_code, improve_function, and implement calls -- and they aren't part of the main control path.

I am eyeing this:

autogen/autogen/oai/openai_utils.py

Line 331 in 20d77a1

# Ensure default models are always considered

   # Ensure default models are always considered
    default_models = ["gpt-4", "gpt-3.5-turbo"]

    for model in default_models:
        # Only assign default API key if the model is not present in the map.
        # If model is present but set to invalid/empty, do not overwrite.
        if model not in model_api_key_map:
            model_api_key_map[model] = "OPENAI_API_KEY"

@twobob How are you loading the config list? Are you using .env?

Hmmm. you know I am certain I did NOT remove that one (I errantly excluded that from the replace)...
Hmm.. Time for new test then

[twobob]

So. i reran it just to make sure it wasnt anything really stupid. BEFORE I made the above change

Blue bar is me making 30 games yesterday with chatDev, green bar is just setting up the requirements (basically 4 round of chat) in autogen on "supposed 3.5" before I stopped it again and waited for the OpenAi usage to catch up.

so. I wasn't going mad it really was doing that.

I made the above change. and only added 3.5 models in the list and ran it again/

Still no change.

I attached my actual notebook.
agentchat_groupchat_research_That_WAS_ACTUTALLY_USED.zip