Add Support for Image Return in Agent Tools

[gabriansa]

Describe the feature

I would like to propose adding support for tools that can return images. This would enable agents to generate, process, and analyze visual data as part of their workflow.

Use Cases:

1. Data Visualization

   • Agents could generate plots and charts from numerical data
   • Analyze trends and patterns visually
   • Make data-driven decisions based on visual representations

2. Image Processing

   • Generate or modify images based on specifications
   • Create diagrams or flowcharts
   • Process and return modified images

Dummy Example Scenario:

@function_tool
def create_plot(data: list[float]) -> Image:  # New Image return type
    """Creates a line plot from the given data and returns it as an image."""
    plt.figure()
    plt.plot(data)
    # Convert plot to image
    return plot_to_image()

[rm-openai]

We should definitely do this at some point. Main blocker is that the OpenAI API doesn't support this yet.

[gabriansa]

What about as an additional message after the tool call?

Dummy Example

messages.append({
    "role": "assistant",
    "content": [
        {
            "type": "image_url",
            "image_url": {
                "url": f"data:image/jpeg;base64,{base64_image}",
            },
        },
    ],
})

[rm-openai]

Yeah that would be cool. Would require a somewhat large refactor to the tools pipeline, to allow returning Content from a tool call instead of just str

[weathon]

Yes, I would like this to be added as well