Skip to content

Antonpirker/OpenAI add async #4603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

antonpirker
Copy link
Member

This adds async instrumentation for /responses API (and also makes sure the span ops and some common attributes are set)

@antonpirker antonpirker changed the base branch from antonpirker/openai-overhaul to antonpirker/openai-responses-api July 21, 2025 14:44
@antonpirker antonpirker marked this pull request as ready for review July 21, 2025 14:44
@antonpirker antonpirker requested a review from a team as a code owner July 21, 2025 14:44
@antonpirker antonpirker merged commit 1ef2a21 into antonpirker/openai-responses-api Jul 21, 2025
117 of 124 checks passed
@antonpirker antonpirker deleted the antonpirker/openai-add-async branch July 21, 2025 14:44
Copy link

codecov bot commented Jul 21, 2025

❌ 6 Tests Failed:

Tests completed Failed Passed Skipped
24095 6 24089 2354
View the top 3 failed test(s) by shortest run time
tests.integrations.openai.test_openai::test_ai_client_span_responses_api
Stack Traces | 0.097s run time
.../integrations/openai/test_openai.py:1028: in test_ai_client_span_responses_api
    assert spans[0]["data"] == {
E   assert {'thread.id': '139840740129664', 'thread.name': 'MainThread', 'gen_ai.system': 'openai', 'gen_ai.request.model': 'gpt-4o', 'gen_ai.operation.name': 'responses', 'gen_ai.request.messages': 'How do I check if a Python object is an instance of a class?', 'gen_ai.response.text': '[{"id": "message-id", "content": [{"annotations": [], "text": "the model response", "type": "output_text"}], "role": "assistant", "status": "completed", "type": "message"}]', 'gen_ai.usage.input_tokens': 20, 'gen_ai.usage.input_tokens.cached': 5, 'gen_ai.usage.output_tokens': 10, 'gen_ai.usage.output_tokens.reasoning': 8, 'gen_ai.usage.total_tokens': 30} == {'gen_ai.request.messages': 'How do I check if a Python object is an instance of a class?', 'gen_ai.request.model': 'gpt-4o', 'gen_ai.usage.input_tokens': 20, 'gen_ai.usage.input_tokens.cached': 5, 'gen_ai.usage.output_tokens': 10, 'gen_ai.usage.output_tokens.reasoning': 8, 'gen_ai.usage.total_tokens': 30, 'gen_ai.response.text': '[{"id": "message-id", "content": [{"annotations": [], "text": "the model response", "type": "output_text"}], "role": "assistant", "status": "completed", "type": "message"}]', 'thread.id': <ANY>, 'thread.name': <ANY>}
E     
E     Common items:
E     {'gen_ai.request.messages': 'How do I check if a Python object is an instance '
E                                 'of a class?',
E      'gen_ai.request.model': 'gpt-4o',
E      'gen_ai.response.text': '[{"id": "message-id", "content": [{"annotations": '
E                              '[], "text": "the model response", "type": '
E                              '"output_text"}], "role": "assistant", "status": '
E                              '"completed", "type": "message"}]',
E      'gen_ai.usage.input_tokens': 20,
E      'gen_ai.usage.input_tokens.cached': 5,
E      'gen_ai.usage.output_tokens': 10,
E      'gen_ai.usage.output_tokens.reasoning': 8,
E      'gen_ai.usage.total_tokens': 30,
E      'thread.id': '139840740129664',
E      'thread.name': 'MainThread'}
E     Left contains 2 more items:
E     {'gen_ai.operation.name': 'responses', 'gen_ai.system': 'openai'}
E     
E     Full diff:
E       {
E     +     'gen_ai.operation.name': 'responses',
E           'gen_ai.request.messages': 'How do I check if a Python object is an instance of a class?',
E           'gen_ai.request.model': 'gpt-4o',
E           'gen_ai.response.text': '[{"id": "message-id", "content": [{"annotations": [], "text": "the model '
E           'response", "type": "output_text"}], "role": "assistant", "status": '
E           '"completed", "type": "message"}]',
E     +     'gen_ai.system': 'openai',
E           'gen_ai.usage.input_tokens': 20,
E           'gen_ai.usage.input_tokens.cached': 5,
E           'gen_ai.usage.output_tokens': 10,
E           'gen_ai.usage.output_tokens.reasoning': 8,
E           'gen_ai.usage.total_tokens': 30,
E     -     'thread.id': <ANY>,
E     -     'thread.name': <ANY>,
E     +     'thread.id': '139840740129664',
E     +     'thread.name': 'MainThread',
E       }
tests.integrations.openai.test_openai::test_ai_client_span_responses_api_no_pii
Stack Traces | 0.098s run time
.../integrations/openai/test_openai.py:988: in test_ai_client_span_responses_api_no_pii
    assert spans[0]["data"] == {
E   AssertionError: assert {'thread.id': '139840740129664', 'thread.name': 'MainThread', 'gen_ai.system': 'openai', 'gen_ai.request.model': 'gpt-4o', 'gen_ai.operation.name': 'responses', 'gen_ai.usage.input_tokens': 20, 'gen_ai.usage.input_tokens.cached': 5, 'gen_ai.usage.output_tokens': 10, 'gen_ai.usage.output_tokens.reasoning': 8, 'gen_ai.usage.total_tokens': 30} == {'gen_ai.request.model': 'gpt-4o', 'gen_ai.usage.input_tokens': 20, 'gen_ai.usage.input_tokens.cached': 5, 'gen_ai.usage.output_tokens': 10, 'gen_ai.usage.output_tokens.reasoning': 8, 'gen_ai.usage.total_tokens': 30, 'thread.id': <ANY>, 'thread.name': <ANY>}
E     
E     Common items:
E     {'gen_ai.request.model': 'gpt-4o',
E      'gen_ai.usage.input_tokens': 20,
E      'gen_ai.usage.input_tokens.cached': 5,
E      'gen_ai.usage.output_tokens': 10,
E      'gen_ai.usage.output_tokens.reasoning': 8,
E      'gen_ai.usage.total_tokens': 30,
E      'thread.id': '139840740129664',
E      'thread.name': 'MainThread'}
E     Left contains 2 more items:
E     {'gen_ai.operation.name': 'responses', 'gen_ai.system': 'openai'}
E     
E     Full diff:
E       {
E     +     'gen_ai.operation.name': 'responses',
E           'gen_ai.request.model': 'gpt-4o',
E     +     'gen_ai.system': 'openai',
E           'gen_ai.usage.input_tokens': 20,
E           'gen_ai.usage.input_tokens.cached': 5,
E           'gen_ai.usage.output_tokens': 10,
E           'gen_ai.usage.output_tokens.reasoning': 8,
E           'gen_ai.usage.total_tokens': 30,
E     -     'thread.id': <ANY>,
E     -     'thread.name': <ANY>,
E     +     'thread.id': '139840740129664',
E     +     'thread.name': 'MainThread',
E       }
tests.integrations.openai.test_openai::test_ai_client_span_responses_api
Stack Traces | 0.103s run time
.../integrations/openai/test_openai.py:1028: in test_ai_client_span_responses_api
    assert spans[0]["data"] == {
E   assert {'thread.id': '140035318127488', 'thread.name': 'MainThread', 'gen_ai.system': 'openai', 'gen_ai.request.model': 'gpt-4o', 'gen_ai.operation.name': 'responses', 'gen_ai.request.messages': 'How do I check if a Python object is an instance of a class?', 'gen_ai.response.text': '[{"id": "message-id", "content": [{"annotations": [], "text": "the model response", "type": "output_text"}], "role": "assistant", "status": "completed", "type": "message"}]', 'gen_ai.usage.input_tokens': 20, 'gen_ai.usage.input_tokens.cached': 5, 'gen_ai.usage.output_tokens': 10, 'gen_ai.usage.output_tokens.reasoning': 8, 'gen_ai.usage.total_tokens': 30} == {'gen_ai.request.messages': 'How do I check if a Python object is an instance of a class?', 'gen_ai.request.model': 'gpt-4o', 'gen_ai.usage.input_tokens': 20, 'gen_ai.usage.input_tokens.cached': 5, 'gen_ai.usage.output_tokens': 10, 'gen_ai.usage.output_tokens.reasoning': 8, 'gen_ai.usage.total_tokens': 30, 'gen_ai.response.text': '[{"id": "message-id", "content": [{"annotations": [], "text": "the model response", "type": "output_text"}], "role": "assistant", "status": "completed", "type": "message"}]', 'thread.id': <ANY>, 'thread.name': <ANY>}
E     
E     Common items:
E     {'gen_ai.request.messages': 'How do I check if a Python object is an instance '
E                                 'of a class?',
E      'gen_ai.request.model': 'gpt-4o',
E      'gen_ai.response.text': '[{"id": "message-id", "content": [{"annotations": '
E                              '[], "text": "the model response", "type": '
E                              '"output_text"}], "role": "assistant", "status": '
E                              '"completed", "type": "message"}]',
E      'gen_ai.usage.input_tokens': 20,
E      'gen_ai.usage.input_tokens.cached': 5,
E      'gen_ai.usage.output_tokens': 10,
E      'gen_ai.usage.output_tokens.reasoning': 8,
E      'gen_ai.usage.total_tokens': 30,
E      'thread.id': '140035318127488',
E      'thread.name': 'MainThread'}
E     Left contains 2 more items:
E     {'gen_ai.operation.name': 'responses', 'gen_ai.system': 'openai'}
E     
E     Full diff:
E       {
E     +     'gen_ai.operation.name': 'responses',
E           'gen_ai.request.messages': 'How do I check if a Python object is an instance of a class?',
E           'gen_ai.request.model': 'gpt-4o',
E           'gen_ai.response.text': '[{"id": "message-id", "content": [{"annotations": [], "text": "the model '
E           'response", "type": "output_text"}], "role": "assistant", "status": '
E           '"completed", "type": "message"}]',
E     +     'gen_ai.system': 'openai',
E           'gen_ai.usage.input_tokens': 20,
E           'gen_ai.usage.input_tokens.cached': 5,
E           'gen_ai.usage.output_tokens': 10,
E           'gen_ai.usage.output_tokens.reasoning': 8,
E           'gen_ai.usage.total_tokens': 30,
E     -     'thread.id': <ANY>,
E     -     'thread.name': <ANY>,
E     +     'thread.id': '140035318127488',
E     +     'thread.name': 'MainThread',
E       }

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant