Converter from AI Service threads/runs to evaluator-compatible schema

[thecsw]

Description

This pull request introduces a new converter, AIAgentConverter, that translates agentic conversations into evaluator-friendly schema. Given a thread_id and run_id of a conversation from AI Projects/Services, we will convert that interaction into a schema that is compatible with evaluator SDKs, OpenAI schemas, etc.

Intended pattern on how to call and use,

from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
from azure.ai.evaluation import AIAgentConverter

# Create your instance to talk to AI services.
project_client = AIProjectClient.from_connection_string(
    credential=DefaultAzureCredential(),
    conn_str="<CONNECTION_STRING>"
)

# Initialize the converter that will be backed by the project.
converter = AIAgentConverter(project_client)

# These can be retrieved from the UI or after running `project_client.agents.create_run`
thread_id = "<THREAD_ID>"
run_id = "<RUN_ID>"

# Convert the agent run to a format suitable for the OpenAI API.
converted_data: dict = converter.convert(thread_id, run_id)

This will return a dictionary, which will produce the following output showcasing the system message, user messages, tool interactions, and assistant's response.

The top-level structure is as follows:

• query: List[dict] — this is the conversation history and context. For backward-compatibility with Azure AI Foundry evaluations, we named it query. This field will have all of the interactions in the thread before the requested run's agent's response. This will include the system message, any other previous assistant messages (including tool calls and tool results), and user queries. Previous runs' tool calls introduce a performance hit, which can be disabled with exclude_tool_calls_previous_runs=True.
• response: List[dict] — assistant's response, containing any number of tool calls, their results, and assistant's formatted responses.
• tool_definitions: List[dict] — definitions of the tools that assistant can use in this run.

Similarly, there is an API to feed a local file, in an offline fashion to produce the same output,

from azure.ai.evaluation import AIAgentConverter

converted_data: dict = AIAgentConverter.convert_from_file("FILE_PATH", "<RUN_ID>")

Here is a non-trivial example with multiple tools, multiple tool calls/results, and multiple turns in the given conversation,

{
  "query": [
    {
      "role": "system",
      "content": "You are a helpful assistant"
    },
    {
      "createdAt": "2025-03-12T20:03:19Z",
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Hello, what is the weather in New York? Make sure to give it to me in Fahrenheit."
        }
      ]
    },
    {
      "createdAt": "2025-03-12T20:03:21Z",
      "run_id": "run_drO1YX3nw1GLp0ZYtIi2nGb3",
      "role": "assistant",
      "content": [
        {
          "type": "tool_call",
          "tool_call": {
            "id": "call_bWZiARx69AevDyjpjP1o8vkx",
            "type": "function",
            "function": {
              "name": "fetch_weather",
              "arguments": {
                "location": "New York"
              }
            }
          }
        }
      ]
    },
    {
      "createdAt": "2025-03-12T20:03:22Z",
      "run_id": "run_drO1YX3nw1GLp0ZYtIi2nGb3",
      "tool_call_id": "call_bWZiARx69AevDyjpjP1o8vkx",
      "role": "tool",
      "content": [
        {
          "type": "tool_result",
          "tool_result": {
            "weather": "Sunny, 25°C"
          }
        }
      ]
    },
    {
      "createdAt": "2025-03-12T20:03:26Z",
      "run_id": "run_drO1YX3nw1GLp0ZYtIi2nGb3",
      "role": "assistant",
      "content": [
        {
          "type": "tool_call",
          "tool_call": {
            "id": "call_oBJAKB3vLZ5wpMkCg2kfLm2F",
            "type": "function",
            "function": {
              "name": "in_farenheit",
              "arguments": {
                "celsius": 25
              }
            }
          }
        }
      ]
    },
    {
      "createdAt": "2025-03-12T20:03:28Z",
      "run_id": "run_drO1YX3nw1GLp0ZYtIi2nGb3",
      "tool_call_id": "call_oBJAKB3vLZ5wpMkCg2kfLm2F",
      "role": "tool",
      "content": [
        {
          "type": "tool_result",
          "tool_result": "77.0°F"
        }
      ]
    },
    {
      "createdAt": "2025-03-12T20:03:29Z",
      "run_id": "run_drO1YX3nw1GLp0ZYtIi2nGb3",
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "The current weather in New York is sunny with a temperature of 77°F."
        }
      ]
    },
    {
      "createdAt": "2025-03-12T20:03:32Z",
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Actually, nevermind, I need the weather in Tokyo. Since they use Celsius, I want it in Celsius too."
        }
      ]
    }
  ],
  "response": [
    {
      "createdAt": "2025-03-12T20:03:33Z",
      "run_id": "run_Y1jbkKYmw5HWtwmK12yfi0Jb",
      "role": "assistant",
      "content": [
        {
          "type": "tool_call",
          "tool_call": {
            "id": "call_eYtq7fMyHxDWIgeG2s26h0lJ",
            "type": "function",
            "function": {
              "name": "fetch_weather",
              "arguments": {
                "location": "Tokyo"
              }
            }
          }
        }
      ]
    },
    {
      "createdAt": "2025-03-12T20:03:34Z",
      "run_id": "run_Y1jbkKYmw5HWtwmK12yfi0Jb",
      "tool_call_id": "call_eYtq7fMyHxDWIgeG2s26h0lJ",
      "role": "tool",
      "content": [
        {
          "type": "tool_result",
          "tool_result": {
            "weather": "Rainy, 22°C"
          }
        }
      ]
    },
    {
      "createdAt": "2025-03-12T20:03:35Z",
      "run_id": "run_Y1jbkKYmw5HWtwmK12yfi0Jb",
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "The current weather in Tokyo is rainy with a temperature of 22°C."
        }
      ]
    }
  ],
  "tool_definitions": [
    {
      "name": "in_farenheit",
      "description": "Converts Celsius to Fahrenheit.",
      "parameters": {
        "type": "object",
        "properties": {
          "celsius": {
            "type": "number",
            "description": "The temperature in Celsius."
          }
        }
      }
    },
    {
      "name": "fetch_weather",
      "description": "Fetches the weather information for the specified location.",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The location to fetch weather for."
          }
        }
      }
    }
  ]
}

This work is actively in progress and sponsored by AI Foundry.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[github-actions]

Thank you for your contribution @thecsw! We will review the pull request and get back to you soon.

Pull Request Overview

This PR introduces the AIAgentConverter to transform agentic conversation data from AI Projects/Services into an evaluator-friendly schema. It defines several Pydantic models for messages and tool definitions, implements a helper function to break tool call data into messages, and adds conversion methods to return different response formats.

• Introduces new models for system, user, assistant, and tool messages.
• Adds a helper function to split tool calls into separate messages.
• Provides the AIAgentConverter class with methods to convert agent runs into various output schemas.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_converters/_models.py	New models and helper functions supporting the conversion process
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_converters/_ai_services.py	Implements the AIAgentConverter and methods to translate conversation data
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_converters/init.py	Added copyright/license header

Comments suppressed due to low confidence (3)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_converters/_models.py:182

• The comment at this location suggests that a tool call might be better represented as a tool message, yet an AssistantMessage is used. Verify whether the tool call should be represented with the tool message role to align with the intended schema.

messages.append(AssistantMessage(run_id=run_id, content=[to_dict(content_tool_call)], createdAt=tool_call.created))

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_converters/_ai_services.py:126

• [nitpick] The sort key uses a boolean expression to order messages with the same timestamp. Consider using an explicit numeric value (e.g., (x.createdAt, 1 if x.role == _AGENT else 0)) for greater clarity.

final_messages.sort(key=lambda x: (x.createdAt, x.role == _AGENT))

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_converters/_models.py:208

• The to_dict function converts objects by performing a round-trip JSON serialization. If the object is a Pydantic model, using its built-in .dict() method might be more efficient and direct.

return json.loads(json.dumps(obj))

[azure-sdk]

API change check

API changes are not detected in this pull request.

[luigiw]

consider marking this as experimental

[thecsw]

This is going for private preview, and if we were to, where does @experimental decorator come from? I see it used sporadically, but this project doesn't seem to have that dependency.

[luigiw]

Nit: List comprehension may be a better way to find all messages belong to a run id. See here as an example. https://www.w3schools.com/python/python_lists_comprehension.asp

[thecsw]

Agreed that list comprehension is neat, but notice that this is a greedy loop, where I want to add commentary on why each filtering is happening per a business and design decision reached. List comprehension due to its neatness would lose that non-trivial context.

I prefer to commenting on top of a statement than in-line within the list comprehension.

[thecsw]

Azure:prp/agent_evaluators is not up to date with the most recent main, so there are more files changed than there really are—only 3. Base branch needs to be rebased to get a clean diff.

[stevepon]

Newer APIs use "developer" here (they're essentially interchangeable, just depends on API version). So might want to make sure that's supported too.

[thecsw]

Interesting, didn't know that. What API version should we be bound to or set it to developer by default? I'll look into versioning.

[stevepon]

I think for output, we can just use system, at least for now. But when reading in messages from threads/etc., you might need to cover the case where it could say "developer". Or maybe we just default to use whatever the thread itself is using, actually.

[stevepon]

Actually, for this converter's purpose I don't think it matters. I take it back.

[thecsw]

Let's write system for now—I was thinking of possibly capturing the model it ran on, so if it were o1, we could mark the instructions as coming from developer. The thread itself only gives us instructions without an explicit marking for either.

[stevepon]

Not for this PR, it can be a quick follown so we don't need to be blocked, but I've suggested we include name in order to support multi-agent flows. Any objections? @singankit ?

[singankit]

Can user provide a name ? I do not see it being generated right now by service.