Missing Chunks in Streamed Responses When Using the SDK

[frankjony17]

Description:

I conducted several tests and can confirm that the issue arises only when using the SDK.

1. Direct API Calls via Postman:

• All requests were successful.

2. SDK Calls with Debugging: During SDK calls, errors were identified:

• Occasionally, all chunks are returned correctly.
• A significant percentage of the time, the last 3–4 chunks are not returned, and no error is displayed.
• The latest version of the SDK was tested, but the issue remains unresolved.

Direct calls were made to the Mistral model deployed on Azure, following the official SDK documentation; however, the issue persists.

Azure Test Code:

import { MistralAzure } from "@mistralai/mistralai-azure";

const mistral = new MistralAzure({
  apiKey: "some-key",
  endpoint: "https://mistral-small-2409-some-url.ai.azure.com",
});

export async function runMistralSmall() {
  const result = await mistral.chat.stream({
    model: "mistral-small-2409",
    stream: true,
    messages: [
      {
        content:
          "Who is the best French painter? Answer in one short sentence.",
        role: "user",
      },
    ],
  });

  for await (const event of result) {
    // Handle the event
    console.log(event.data.choices[0]);
  }
}

Responses:

• Missing Chunks (the last chunks are not returned):

{
  index: 0,
  delta: { role: 'assistant', content: '' },
  finishReason: null
}
{ index: 0, delta: { content: 'Claude' }, finishReason: null }
{ index: 0, delta: { content: ' Mon' }, finishReason: null }
{ index: 0, delta: { content: 'et' }, finishReason: null }
{ index: 0, delta: { content: ' is' }, finishReason: null }
{ index: 0, delta: { content: ' often' }, finishReason: null }
{ index: 0, delta: { content: ' considered' }, finishReason: null }
{ index: 0, delta: { content: ' the' }, finishReason: null }
{ index: 0, delta: { content: ' best' }, finishReason: null }
{ index: 0, delta: { content: ' French' }, finishReason: null }
{ index: 0, delta: { content: ' painter' }, finishReason: null }
{ index: 0, delta: { content: ' due' }, finishReason: null }
{ index: 0, delta: { content: ' to' }, finishReason: null }
{ index: 0, delta: { content: ' his' }, finishReason: null }
{ index: 0, delta: { content: ' significant' }, finishReason: null }
{ index: 0, delta: { content: ' contributions' }, finishReason: null }
{ index: 0, delta: { content: ' to' }, finishReason: null }
{ index: 0, delta: { content: ' the' }, finishReason: null }
{ index: 0, delta: { content: ' Im' }, finishReason: null }
{ index: 0, delta: { content: 'pression' }, finishReason: null }
{ index: 0, delta: { content: 'ist' }, finishReason: null }
{ index: 0, delta: { content: ' movement' }, finishReason: null }

All Chunks Returned Correctly (occasionally, all chunks are returned):

{
  index: 0,
  delta: { role: 'assistant', content: '' },
  finishReason: null
}
{ index: 0, delta: { content: 'Claude' }, finishReason: null }
{ index: 0, delta: { content: ' Mon' }, finishReason: null }
{ index: 0, delta: { content: 'et' }, finishReason: null }
{ index: 0, delta: { content: ' is' }, finishReason: null }
{ index: 0, delta: { content: ' often' }, finishReason: null }
{ index: 0, delta: { content: ' considered' }, finishReason: null }
{ index: 0, delta: { content: ' one' }, finishReason: null }
{ index: 0, delta: { content: ' of' }, finishReason: null }
{ index: 0, delta: { content: ' the' }, finishReason: null }
{ index: 0, delta: { content: ' best' }, finishReason: null }
{ index: 0, delta: { content: ' French' }, finishReason: null }
{ index: 0, delta: { content: ' pain' }, finishReason: null }
{ index: 0, delta: { content: 'ters' }, finishReason: null }
{ index: 0, delta: { content: ' due' }, finishReason: null }
{ index: 0, delta: { content: ' to' }, finishReason: null }
{ index: 0, delta: { content: ' his' }, finishReason: null }
{ index: 0, delta: { content: ' significant' }, finishReason: null }
{ index: 0, delta: { content: ' contributions' }, finishReason: null }
{ index: 0, delta: { content: ' to' }, finishReason: null }
{ index: 0, delta: { content: ' the' }, finishReason: null }
{ index: 0, delta: { content: ' Im' }, finishReason: null }
{ index: 0, delta: { content: 'pression' }, finishReason: null }
{ index: 0, delta: { content: 'ist' }, finishReason: null }
{ index: 0, delta: { content: ' movement' }, finishReason: null }
{ index: 0, delta: { content: '.' }, finishReason: null }
{ index: 0, delta: { content: '' }, finishReason: 'stop' }

Please investigate this issue as it significantly impacts the reliability of SDK-based integrations.

[petod108]

I got the same random error. To get stream not finished.

[jean-malo]

Hi,
Thanks for opening this issue. I will look into it and get back to you when I have more information.
To speed up the investigation on our side, did you observe this consistently for specific prompts or is it mostly random ?

Thanks!

[jean-malo]

@petod108 are you using Azure too ?

[frankjony17]

Hi, Thanks for opening this issue. I will look into it and get back to you when I have more information. To speed up the investigation on our side, did you observe this consistently for specific prompts or is it mostly random ?

Thanks!

I tested with more than 5 random prompts, small and large ones and the behavior is the same.