.Net: Bug: .Net Plugin return[Description] being totaly ignored?

[christianarg]

When I define un a plugin [return:Description("Some description")] as documented in the official documentation https://learn.microsoft.com/en-us/semantic-kernel/concepts/plugins/?pivots=programming-language-csharp#1-define-your-plugin the return description seems to be completely ignored.
After testing many times I've observed that the indications in [return:Description("...")] where completely ignored by the model. Since there is a large discussion about how to capture exact request that is made to azure open ai when using chat completion service here #1239 I made some code to inspect the exact request

public class DiagnosticListenerObserver : IObserver<DiagnosticListener>
{
    private readonly IObserver<KeyValuePair<string, object>> _innerObserver;

    public DiagnosticListenerObserver(IObserver<KeyValuePair<string, object>> innerObserver)
    {
        _innerObserver = innerObserver;
    }

    public void OnNext(DiagnosticListener diagnosticListener)
    {
        if (diagnosticListener.Name == "HttpHandlerDiagnosticListener")
        {
            diagnosticListener.Subscribe(_innerObserver);
        }
    }

    public void OnError(Exception error)
    {
        // Handle errors here, if needed
    }

    public void OnCompleted()
    {
        // Handle completion, if needed
    }
}

public class HttpClientLoggingListener : IObserver<KeyValuePair<string, object>>
{
    public void OnNext(KeyValuePair<string, object> value)
    {
        if (value.Key == "System.Net.Http.HttpRequestOut.Start")
        {
            var request = (HttpRequestMessage)value.Value.GetType().GetProperty("Request")?.GetValue(value.Value);

            //Console.WriteLine($"Request: {request?.Method} {request?.RequestUri}");
            if (request.Method == HttpMethod.Post && request.RequestUri.ToString().Contains("openai.azure.com"))
            {
                Console.WriteLine($"POST Request to {request.RequestUri}");

                if (request.Content != null)
                {
                    // Read the body as a string
                    var contentTask = request.Content.ReadAsStringAsync();
                    string requestBody = contentTask.Result;    // CUIDADO! sync-over-async

                    Console.WriteLine("Request Body:");
                    Console.WriteLine(requestBody);
                }
                else
                {
                    Console.WriteLine("Request Body: <No Content>");
                }
            }
        }
    }

    public void OnError(Exception error) { }

    public void OnCompleted() { }
}

Then having 2 simple plugins

public class M365InvoicePlugin
{
    [KernelFunction()]
    [Description("Obtener facturas de Microsoft 365")]
    public string GetM365Invoices()
    {
        return "Facturas de Microsoft 365. Septiembre 2024:80€";
    }
}


public class AzureInvoicePlugin
{
    [KernelFunction]
    
    [Description("Obtener facturas de Azure")]
    [return:Description("Facturas de Azure del último año")] 
    public string GetAzureInvoices()
    {
        return "Facturas de Azure. Octubre 2024:100€";
    }
}

When invoking with chatCompletions (many lines shortened for brevity, I assume you get the idea)

var listener = new HttpClientLoggingListener();
DiagnosticListener.AllListeners.Subscribe(new DiagnosticListenerObserver(listener));

var builder = Kernel.CreateBuilder();
builder.Plugins.AddFromType<M365InvoicePlugin>();
builder.Plugins.AddFromType<AzureInvoicePlugin>()
// configure azure openai etc
var kernel = builder.Build();
var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();
var chat = new ChatHistory();
chat.AddUserMessage("Hola");
var result = await chatCompletionService.GetChatMessageContentAsync(
     chat,
     executionSettings: openAIPromptExecutionSettings,
     kernel: kernel);

The captured post does NOT show the text in the [return:Description("")] in the tools parameter:

{
    "messages": [
        {
            "role": "user",
            "content": "hola"
        }
    ],
    "model": "CENSORED",
    "temperature": 0.1,
    "top_p": 1,
    "tools": [
        {
            "type": "function",
            "function": {
                "description": "Obtener facturas de Microsoft 365",
                "name": "M365InvoicePlugin-GetM365Invoices",
                "parameters": {
                    "type": "object",
                    "required": [],
                    "properties": {}
                }
            }
        },
        {
            "type": "function",
            "function": {
                "description": "Obtener facturas de Azure",
                "name": "AzureInvoicePlugin-GetAzureInvoices",
                "parameters": {
                    "type": "object",
                    "required": [],
                    "properties": {}
                }
            }
        }
    ],
    "tool_choice": "auto"
}

[SergeyMenshykh]

Hi @christianarg, thanks for reporting the issue. Please consider one of the techniques described here - https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/FunctionCalling/FunctionCalling_ReturnMetadata.cs to provide function return metadata to AI model. We'll improve our documentation to make it clearer.

[SergeyMenshykh]

@christianarg, may I ask you to share scenario(s) where the LLM was not able to reason about the function result, and you tried to help it by including the return type description?

[christianarg]

@SergeyMenshykh I don't need it, with the normal [Description ("some description")] is more than enough IMO. But literally your official docs https://learn.microsoft.com/en-us/semantic-kernel/concepts/plugins/?pivots=programming-language-csharp#1-define-your-plugin are telling developers to use [return: Description ("information about function return")].
It's pretty hard to argue with a team member telling him that return attribute does nothing when it's in your official docs. I'm not telling you "implement it" (that's up to you really). I'm telling, if this doesn't work, remove it from your docs (and maybe even put a note clarifying that return: Description is ignored.)

[SergeyMenshykh]

@christianarg, got it; we are improving the documentation at the moment.

[SergeyMenshykh]

The documentation has been updated. Closing the issue for now.