Added support for the new function calling capability in Chat Completions API

[rzubek]

Hi there!

Here's a PR that adds support for new function calling capability in GPT
https://openai.com/blog/function-calling-and-other-api-updates

Includes

• Data model for function definitions in the request, and parsing out function calls in the reply
• Test function that illustrates usage
• Model definitions for 0613 updates which are the min version for this API

Does not include changes to documentation (like the readme file), not sure what's the best place to document this.

[rzubek]

It's worth noting that this functionality is not compatible with streaming completions API. EDIT: see comment below.

If a chunked response contains a function call, the call gets split up just as if it were a message (on whitespace, from the looks of it), and the function name and arguments are returned as separate chunks. And then CreateCompletionAsStream() gets those multiple chunks, but is unable to put that back together because it's not the message being split.

It's not clear that splitting up function calls was intended behavior on their end - it splits JSON responses up into separate chunks, that aren't exactly valid by themselves, and it looks difficult to put the result back together. Maybe we put up a warning if we detect that the user is trying to use a streaming function, and they have the Functions list populated?

[oferavnery]

I would hate to lose streaming support. With streaming functions the pieces of the JSON are streamed as if they were normal text:

{
"id": "chatcmpl-7RbxtzQqEv5coBpV1C6UKygmXaECp",
"object": "chat.completion.chunk",
"created": 1686815149,
"model": "gpt-3.5-turbo-0613",
"choices": [
{
"index": 0,
"delta": {
"role": "assistant",
"content": null,
"function_call": {
"name": "function_name",
"arguments": ""
}
},
"finish_reason": null
}
]
}
{
"id": "chatcmpl-7RbxtzQqEv5coBpV1C6UKygmXaECp",
"object": "chat.completion.chunk",
"created": 1686815149,
"model": "gpt-3.5-turbo-0613",
"choices": [
{
"index": 0,
"delta": {
"function_call": {
"arguments": "{\n"
}
},
"finish_reason": null
}
]
}
And so on. As the stream does include the function output - just need to parse it differently. Imagine in streaming mode the backend identifies that a function is needed, streams an "Using Function" message to the client, and continues to work in the backend until function call is completed, then streaming of the api standard output is resumed.

[rzubek]

It was an unexpected bit of work, but I added support for streaming.

During stream processing we detect if a function call is being returned, and if so, a separate handler starts accumulating function call data, and then returns it as one packed when it's done.

I don't think there's a way around this kind of an accumulator, because the whole function call JSON needs to be collected before it can be passed anywhere else (e.g. to a JSON parser).

Feedback appreciated!

[rzubek]

By the way, for anyone interested, this is what a streamed function call looks like:

Unstreamed:

{..."choices":[{"index":0,"delta":{"role":"assistant","content":null,"function_call":{"name":"get_n_day_weather_forecast","arguments":"{\n  \"location\": \"Chicago, USA\",\n  \"format\": \"F\",\n  \"num_days\": 5\n}"}},"finish_reason":"function_call"}]}

Streamed:

{..."choices":[{"index":0,"delta":{"role":"assistant","content":null,"function_call":{"name":"get_n_day_weather_forecast","arguments":""}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"{\n"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"location"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"\":"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"Chicago"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":","}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" IL"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"\",\n"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"format"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"\":"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"C"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"elsius"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"\",\n"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"num"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"_days"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"\":"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"5"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"\n"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{"function_call":{"arguments":"}"}},"finish_reason":null}]}
{..."choices":[{"index":0,"delta":{},"finish_reason":"function_call"}]}