[AI Evaluation] Tool messages (or anything without 'Text' contents) are not rendered in reports

[jtschuster]

In Microsoft.Extensions.AI.Evaluations.Reporting, the report created does not render tool calls with FunctionCallContents or FunctionResponseContents because the content type is not 'text'. I've worked around this by using the messages with FunctionCallContents to get the response, and changing the tool messages contents for the EvaluateAsync method, but it was frustrating to debug the library to figure this out.

[shyamnamboodiripad]

Thanks for the feedback @jtschuster. We have discussed displaying tool call messages as part of the conversation history that is displayed in the report but haven't got round to implementing this support yet. We'll use the current issue to track this.

None of the evaluators that ship as part of the library currently consider tool call messages (as part of their eval prompts). However, some new evaluators that we are planning to introduce will need this (see #6350). It would be especially useful to include this info as part of the displayed conversation history once we have those - but I agree they would also be useful in general.

Just curious: Did you run into the above when using custom authored evaluators that consider the tool messages as part of the evaluation they perform? Or did you run into it for the evaluators that ship as part of the library?

(cc @peterwald FYI)

[jtschuster]

Did you run into the above when using custom authored evaluators that consider the tool messages as part of the evaluation they perform? Or did you run into it for the evaluators that ship as part of the library?

Both, I have tests that prefill FunctionCall and FunctionResponse messages in the chat history when calling IChatClient.GetResponseAsync() and use the EquivalenceEvaluator to compare an expected response to the tool call result. But I also have another IEvaluator that does add the ToolCall data into the prompt.