Load balancing

[TimW9]

Description

I am running into trouble with rate limits in some scenarios. I would love to see a feature in pydantic ai where I can provide the agent with (multiple) models/fallback models, so the agent could then use the other models if rate limits are reached.

Right now, the only way to achieve this is to define multiple agents, give each a model and wrap them all over the system prompts and tools, which is already really messy and then try catch the rate limits and balance the load elsewhere.

References

No response

[aristideubertas]

There is a Fallback model https://ai.pydantic.dev/api/models/fallback/

[DouweM]

@TimW9 The FallbackModel @aristideubertas pointed out is the recommended way to go. There's also people working on handling rate limits directly: #1734. I'll close the issue considering the question answered, but let me know if anything's still unclear.