fix(ai): add mapping for gen_ai message roles #4884

shellmayr · 2025-10-03T12:42:00Z

Add a constant that contains the allowed message roles according to OTEL and a mapping
Apply that mapping to all gen_ai integrations
We will track input roles that do not conform to expectations via a Sentry issue in agent monitoring to make sure we continually update the mappings

Closes TET-1077

linear · 2025-10-03T12:42:52Z

TET-1077 Map message roles from different SDKs in Agent Monitoring

codecov · 2025-10-03T12:45:05Z

❌ 4 Tests Failed:

Tests completed	Failed	Passed	Skipped
23705	4	23701	2317

View the top 3 failed test(s) by shortest run time

tests.integrations.openai_agents.test_openai_agents::test_tool_execution_span

Stack Traces | 0.079s run time

.../integrations/openai_agents/test_openai_agents.py:562: in test_tool_execution_span
    assert re.sub(
E   assert '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\\"message\\": \\"hello\\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool_call", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]' == '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\\"message\\": \\"hello\\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]'
E     
E     - [{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\"message\": \"hello\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]
E     ?                                                                                                                                                                                                                                                                                                                                                                                                                    ^^^^^^^^^^^^^
E     + [{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\"message\": \"hello\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool_call", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]
E     ?                                                                                                                                                                                                                                                                                                                                                                                                                    ^^^^^^^^^^^^^^^^^^

tests.integrations.openai_agents.test_openai_agents::test_tool_execution_span

Stack Traces | 0.095s run time

.../integrations/openai_agents/test_openai_agents.py:562: in test_tool_execution_span
    assert re.sub(
E   assert '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\\"message\\": \\"hello\\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool_call", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]' == '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\\"message\\": \\"hello\\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]'
E     
E     - [{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\"message\": \"hello\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]
E     ?                                                                                                                                                                                                                                                                                                                                                                                                                    ^^^^^^^^^^^^^
E     + [{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\"message\": \"hello\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool_call", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]
E     ?                                                                                                                                                                                                                                                                                                                                                                                                                    ^^^^^^^^^^^^^^^^^^

tests.integrations.openai_agents.test_openai_agents::test_tool_execution_span

Stack Traces | 0.17s run time

.../integrations/openai_agents/test_openai_agents.py:562: in test_tool_execution_span
    assert re.sub(
E   assert '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\\"message\\": \\"hello\\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool_call", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]' == '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\\"message\\": \\"hello\\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]'
E     
E     - [{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\"message\": \"hello\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]
E     ?                                                                                                                                                                                                                                                                                                                                                                                                                    ^^^^^^^^^^^^^
E     + [{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\"message\": \"hello\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool_call", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]
E     ?                                                                                                                                                                                                                                                                                                                                                                                                                    ^^^^^^^^^^^^^^^^^^

tests.integrations.openai_agents.test_openai_agents::test_tool_execution_span

Stack Traces | 0.195s run time

.../integrations/openai_agents/test_openai_agents.py:562: in test_tool_execution_span
    assert re.sub(
E   assert '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\\"message\\": \\"hello\\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool_call", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]' == '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\\"message\\": \\"hello\\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]'
E     
E     - [{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\"message\": \"hello\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]
E     ?                                                                                                                                                                                                                                                                                                                                                                                                                    ^^^^^^^^^^^^^
E     + [{"role": "system", "content": [{"type": "text", "text": "You are a helpful test assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Please use the simple test tool"}]}, {"role": "assistant", "content": [{"arguments": "{\"message\": \"hello\"}", "call_id": "call_123", "name": "simple_test_tool", "type": "function_call", "id": "call_123", "function": "NOT_CHECKED"}]}, {"role": "tool_call", "content": [{"call_id": "call_123", "output": "Tool executed with: hello", "type": "function_call_output"}]}]
E     ?                                                                                                                                                                                                                                                                                                                                                                                                                    ^^^^^^^^^^^^^^^^^^

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

sentrivana

LGTM, ty!

Left one nit.

Regarding the failing Check CI config check, merging master into this should fix it (took the liberty to sync this for you), if not, I'll take care of it. (We'll be removing the check altogether soon.)

sentrivana · 2025-10-07T12:20:05Z

tests/integrations/anthropic/test_anthropic.py

+    assert SPANDATA.GEN_AI_REQUEST_MESSAGES in span["data"]
+
+    # Parse the stored messages
+    import json


Any reason not to have this at the top with the other imports?

Nope, I'll move it up

fix(ai): add mapping for gen_ai message roles

ae8e359

shellmayr added 6 commits October 7, 2025 10:56

map openai correctly

f680921

map openai

906da3d

don't try to convert messages that don't have a role

0081cf8

constant

e7bf105

remove superfluous comments and use the constants everywhere

ab4dec0

fix tool call role name

0d7102e

shellmayr marked this pull request as ready for review October 7, 2025 12:07

shellmayr requested a review from a team as a code owner October 7, 2025 12:07

This comment was marked as outdated.

Sign in to view

fix missing tool

8810ee6

This comment was marked as outdated.

Sign in to view

sentrivana approved these changes Oct 7, 2025

View reviewed changes

sentrivana and others added 3 commits October 7, 2025 14:23

Merge branch 'master' into shellmayr/fix/gen_ai-map-message-roles

96b3ad6

move import up

16342b1

add to langchain

3286292

This comment was marked as outdated.

Sign in to view

add langchain test

eb8c05d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(ai): add mapping for gen_ai message roles #4884

fix(ai): add mapping for gen_ai message roles #4884

Uh oh!

shellmayr commented Oct 3, 2025 •

edited

Loading

Uh oh!

linear bot commented Oct 3, 2025

Uh oh!

codecov bot commented Oct 3, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

sentrivana left a comment •

edited

Loading

Uh oh!

sentrivana Oct 7, 2025

Uh oh!

shellmayr Oct 7, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

fix(ai): add mapping for gen_ai message roles #4884

Are you sure you want to change the base?

fix(ai): add mapping for gen_ai message roles #4884

Uh oh!

Conversation

shellmayr commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear bot commented Oct 3, 2025

Uh oh!

codecov bot commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 4 Tests Failed:

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

sentrivana left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sentrivana Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

shellmayr Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

shellmayr commented Oct 3, 2025 •

edited

Loading

codecov bot commented Oct 3, 2025 •

edited

Loading

sentrivana left a comment •

edited

Loading