-
Notifications
You must be signed in to change notification settings - Fork 835
Add ToChatCompletion{Async} methods for combining StreamingChatCompletionUpdates #5605
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
dd9e6d5
to
bdf387f
Compare
src/Libraries/Microsoft.Extensions.AI/ChatCompletion/StreamingChatCompletionUpdateExtensions.cs
Outdated
Show resolved
Hide resolved
I hadn't realized that |
Neither of them update the chat history with the resulting message. |
src/Libraries/Microsoft.Extensions.AI/ChatCompletion/StreamingChatCompletionUpdateExtensions.cs
Outdated
Show resolved
Hide resolved
...Microsoft.Extensions.AI.Tests/ChatCompletion/StreamingChatCompletionUpdateExtensionsTests.cs
Outdated
Show resolved
Hide resolved
The naming of Do any of the following feel right to you? await foreach (var update in client.CompleteStreamingAsync(messages).AppendMessageTo(messages))
await foreach (var update in client.CompleteStreamingAsync(messages).AppendResultTo(messages))
await foreach (var update in client.CompleteStreamingAsync(messages).AppendResponseTo(messages))
await foreach (var update in client.CompleteStreamingAsync(messages).WithAppendTo(messages))
await foreach (var update in client.CompleteStreamingAsync(messages).ThenAppendTo(messages)) I mildly lean towards |
I tend to associate the |
src/Libraries/Microsoft.Extensions.AI/ChatCompletion/StreamingChatCompletionUpdateExtensions.cs
Outdated
Show resolved
Hide resolved
The hard part here is the ToChatCompletion. With that, someone can do the equivalent of List<StreamingChatCompletionUpdate> updates = [];
await foreach (var update in client.CompleteStreamingAsync(messages))
{
Console.Write(update);
updates.Add(update);
}
messages.Add(updates.ToChatCompletion().Message); Given that all of the raised concerns are about |
This is now just: public static ChatCompletion ToChatCompletion(
this IEnumerable<StreamingChatCompletionUpdate> updates, bool coalesceContent = true) |
(though now that I write that out, I'm wondering if I should add one for IAsyncEnumerable as well) |
0e845f5
to
dda3072
Compare
Refactored to have ToChatCompletion{Async} and to have them in the Abstractions library. Having them there means a leaf client can choose to use e.g. ToChatCompletionAsync to implement its CompleteAsync method around its CompleteStreamingAsync, if desired. |
...crosoft.Extensions.AI.Abstractions/ChatCompletion/StreamingChatCompletionUpdateExtensions.cs
Show resolved
Hide resolved
...crosoft.Extensions.AI.Abstractions/ChatCompletion/StreamingChatCompletionUpdateExtensions.cs
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As Microsoft.Extensions.AI.Abstractions
is accumulating more and more helper methods, I'm growing somewhat uncomfortable with the "Abstractions" suffix in this project. Would it make sense to rename this to be just Microsoft.Extensions.AI
and then have the existing Microsoft.Extensions.AI
called something along the lines of Microsoft.Extensions.AI.Middlewares
?
I think it's fine. These helpers are to aid in the development of implementations of the abstractions. It's very much in line with other M.E libs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good.
In a previous iteration of this PR, I think you were able to remove the equivalent logic from CachingChatClient
and use your new helper instead. But in the current version of the PR that's no longer the case. Is it no longer possible to avoid having two versions of this logic?
I put these into m.e.ai.abstractions, and the caching stuff is up in m.e.ai. We could reconsolidate if we wanted to expose the coalescing publicly from abstractions. |
Went another way: #5616 |
WithMessageAddedAsync
enables writing code like:and upon completion of the loop,
messages
will contain aChatMessage
merged from all of the updates.ToChatCompletion
is what's used to achieve that, but is also exposed on its own so that code which has an enumerable ofStreamingChatCompletionUpdate
s can merge them into aChatCompletion
. This enables a non-streaming implementation to easily be authored in terms of a streaming one.Microsoft Reviewers: Open in CodeFlow