Implementing CreateCompletionStream

[qbm5]

I have implemented a dedicated streaming function for the CreateCompletion function

public async IAsyncEnumerable<CompletionCreateResponse> CreateCompletionStream(CompletionCreateRequest createCompletionRequest, string? engineId = null)
    {
        createCompletionRequest.Stream = true;
        createCompletionRequest.LogitBias = new object();
        var content = new StringContent(System.Text.Json.JsonSerializer.Serialize(createCompletionRequest), Encoding.UTF8, "application/json");
   
        var request =  await _httpClient.PostAsync(_endpointProvider.CompletionCreate(ProcessEngineId(engineId)), content);

        using (var stream = await request.Content.ReadAsStreamAsync())
        {
            using(var reader = new StreamReader(stream))
            {
                while (!reader.EndOfStream)
                {
                    var line = await reader.ReadLineAsync();
                    if (!string.IsNullOrEmpty(line))
                    {
                        line = line.Replace("data: ", string.Empty);
                        if (line.Contains("[DONE]")) break;
                        var block = System.Text.Json.JsonSerializer.Deserialize<CompletionCreateResponse>(line);
                        if (null != block) yield return  block;
                    }
                }
            }
        }
    }

**Change requires updates to the class attributes to ignore null fields, example

  [JsonPropertyName("suffix"), JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
        public string? Suffix { get; set; }

**And the test looks like this

public static async Task RunSimpleCompletionStreamTest(IOpenAIService sdk)
       {
           ConsoleExtensions.WriteLine("Completion Testing is starting:", ConsoleColor.Cyan);

           try
           {
               ConsoleExtensions.WriteLine("Completion Test:", ConsoleColor.DarkCyan);
               var completionResult = sdk.Completions.CreateCompletionStream(new CompletionCreateRequest()
               {
                   Prompt = "Once upon a time",
                   //    PromptAsList = new []{"Once upon a time"},
                   MaxTokens = 100,
                   LogProbs = 1,
               }, Models.Davinci);

               await foreach (var completion in completionResult)
               {
                   if (completion.Successful)
                   {
                       Console.WriteLine(completion.Choices.FirstOrDefault());
                   }
                   else
                   {
                       if (completion.Error == null)
                       {
                           throw new Exception("Unknown Error");
                       }

                       Console.WriteLine($"{completion.Error.Code}: {completion.Error.Message}");
                   }
               }

               
           }
           catch (Exception e)
           {
               Console.WriteLine(e);
               throw;
           }
       }

I have the code working locally, but I do not have permissions to branch and PR, but I think this would be a nice addition.

[kayhantolga]

Oh, that's great. I think you can fork the library and then create PR back. Would you like to try that or do you want me to update the codes?

[qbm5]

I will try that. I haven't done opensource work before, so I am not sure if there are special steps to make it acceptable, but I will fork and PR later today

[andrepires]

Hi.
Any chances to get an example on how to use this in a Web API endpoint? Trying to have an Angular app receiving the GPT response in chuncks similaar to what ChatGPT does.
ChatGPT doesn't have that answer, as this is way too new :)
Please help.
Thanks

[qbm5]

openai/OpenAI.Playground/TestHelpers/CompletionTestHelper.cs

Line 126 in cf58906

public static async Task RunSimpleCompletionStreamTest(IOpenAIService sdk)

[andrepires]

Thank you. It looks like it is the case for SignalR or some other socket solution, right?

[qbm5]

Can stream a http response like this too

await foreach (var completion in result)
               {
               if (completion.Successful)
               {
                   completions.Add(completion);
                   await using var sw = new StreamWriter(response.Body);
                   if (string.IsNullOrEmpty(completion.Choices.FirstOrDefault()?.FinishReason))
                   {
                       var text = completion.Choices.FirstOrDefault()?.Text;
                       output += text;
                       await sw.WriteAsync(text);
                   }
                   else
                   {
                       await sw.WriteAsync("[DONE]");
                       await sw.FlushAsync();
                   }
               }