🐛 Bug Report: Generation crashes because finish_reason is not always present in the async delta

[clemlesne]

Which component is this bug for?

OpenAI Instrumentation

📜 Description

The async chat completion crashes because the finish_reason is absent from a delta.

It sometimes happens with the Azure OpenAI backend.

👟 Reproduction steps

Extract of the code I used:

# Instrument OpenAI
environ["TRACELOOP_TRACE_CONTENT"] = str(
    True
)  # Instrumentation logs prompts, completions, and embeddings to span attributes, set to False to lower monitoring costs or to avoid logging PII
OpenAIInstrumentor(enrich_token_usage=True).instrument()

[...]

stream: AsyncStream[ChatCompletionChunk] = await client.chat.completions.create(
    **chat_kwargs,
    stream=True,
)
async for chunck in stream:
    choices = chunck.choices
    if not choices:  # Skip empty choices, happens sometimes with GPT-4 Turbo
        continue
    delta = choices[0].delta
    yield delta

👍 Expected behavior

Even if finish_reason is absent, generation must not crash because of the instrumentation.

👎 Actual Behavior with Screenshots

The generation crashes because finish_reason is absent in a stream delta.

Short log:

File "/xxx/lib/python3.11/site-packages/opentelemetry/instrumentation/openai/shared/chat_wrappers.py", line 299, in _set_choice_counter_metrics
     "llm.response.finish_reason": choice["finish_reason"],
                                   ~~~~~~^^^^^^^^^^^^^^^^^
 KeyError: 'finish_reason'

Full log:

Traceback (most recent call last):
  File "/xxx/lib/python3.11/site-packages/opentelemetry/instrumentation/openai/shared/chat_wrappers.py", line 514, in __anext__
    chunk = await self.__wrapped__.__anext__()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/xxx/lib/python3.11/site-packages/openai/_streaming.py", line 144, in __anext__
    return await self._iterator.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopAsyncIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/xxx/helpers/call_llm.py", line 228, in load_llm_chat
    chat_task.result()
  File "/xxx/helpers/call_llm.py", line 400, in _execute_llm_chat
    async for delta in completion_stream(
  File "/xxx/helpers/llm_worker.py", line 113, in completion_stream
     raise e
   File "/xxx/helpers/llm_worker.py", line 102, in completion_stream
     async for chunck in _completion_stream_worker(
 Loading LLM chat
   File "/xxx/helpers/llm_worker.py", line 188, in _completion_stream_worker
     async for chunck in stream:
   File "/xxx/lib/python3.11/site-packages/opentelemetry/instrumentation/openai/shared/chat_wrappers.py", line 517, in __anext__
     self._close_span()
   File "/xxx/lib/python3.11/site-packages/opentelemetry/instrumentation/openai/shared/chat_wrappers.py", line 578, in _close_span
     _set_choice_counter_metrics(
   File "/xxx/lib/python3.11/site-packages/opentelemetry/instrumentation/openai/shared/chat_wrappers.py", line 299, in _set_choice_counter_metrics
     "llm.response.finish_reason": choice["finish_reason"],
                                   ~~~~~~^^^^^^^^^^^^^^^^^
 KeyError: 'finish_reason'

🤖 Python Version

3.11.9

📃 Provide any additional context for the Bug.

Versions:

• openai v1.30.3
• opentelemetry-instrumentation-openai v0.21.4

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

[nirga]

Thanks for reporting @clemlesne! This definitely shouldn't throw an error, so I've added a guard against it (we have it across the repo but apparently this was one spot that was missing), and fixed the bug.