Add JSON Schema Support for Structured AI Outputs

[oleander]

What you'd like to see

Proposal: Add JSON Schema support for structured AI outputs. This would allow agents to define output schemas that get validated at the API level for supported models (OpenAI, Grok), ensuring consistent and reliable structured responses from AI models.

Key components:

• New schemas section in YAML configuration files with native YAML syntax
• Schema references using $ref for composition and reuse
• Model-specific schema configuration in agent definitions
• Support for both inline schemas and external schema files
• Configuration validation that fails when schemas are specified for unsupported models

Example configuration:

schemas:
  event:
    type: object
    required: [title]
    properties:
      title: 
        type: string
        description: "Event title"
  response:
    type: object
    required: [events]
    properties:
      events: 
        type: array
        items: 
          $ref: "#/schemas/event"

agents:
  root:
    model: openai
    description: "Event extractor"
    instruction: "Extract events from text"
    schema:
      output: event

Why you'd like to see it

Structured outputs would help me build more reliable AI agents that can consistently return data in expected formats. Currently, I have to implement post-processing validation and retry logic when AI responses don't match expected structures, which adds complexity and reduces reliability. This feature would eliminate the need for manual validation and retry mechanisms, making agent responses more predictable and reducing development overhead.

The feature would be particularly valuable for:

• Data extraction agents that need consistent JSON output
• API integration scenarios requiring specific response formats
• Multi-agent workflows where downstream agents depend on structured data from upstream agents

Workarounds?

Currently using manual JSON parsing and validation in post-processing steps, along with retry logic when responses don't match expected formats. This involves:

• Custom validation functions for each expected output structure
• Retry mechanisms with different prompts when validation fails
• Error handling for malformed JSON responses
• Manual schema enforcement in agent instructions

Additional context

• The approach is inspired by jsonSchema in GitHub's gh-models project
• OpenAI (structured output using response_format)
• Grok (structured outputs)

Implementation would use github.com/xeipuuv/gojsonschema for full JSON Schema Draft 7 support, with schema resolution and $ref handling for composition. The feature would be backwards compatible - existing agent configurations without schema definitions would continue to work unchanged.

Future considerations include extending support to input validation.

[krissetto]

The basics of structured output support were implemented with this PR.

I'd suggest giving it a try and then opening more specific issues for other parts of your request (e.g.the schemas section in the yaml)

[krissetto]

Closing for now as most of this has been implemented in this PR. Feel free to open separate issues for things like schemas from external files or your other requests if they don't work as expected