[Feature]: Prompt caching friendly routing strategy

[jeromeroussin]

The Feature

Prompt caching is harder to trigger when litellm load balances across several deployments (using Azure as an example). If the litellm gateway is configured with, say, 3 deployments for a specific model, it may take 3 or more calls before prompt caching takes place, and cost saving+lower latency achieved. The more deployments for a single model, the more calls it will take to "warm up" prompt caching in each deployment.

I am suggesting the following prompt caching friendly routing strategy: whenever a prompt of over 1024 tokens is detected, litellm would cache the beginning of the prompt along with the model-id it landed on. On subsequent calls with the same first 1024 tokens in the prompt, litellm would route the request to the same cached model-id. The cache entries would only need to live in the cache for as long as the prompt caching TTL of the LLM providers themselves (which varies from 5m to one hour)

Motivation, pitch

Lower costs, lower latencies with prompt caching that kicks in immediately while not sacrificing load-balancing.

Twitter / LinkedIn details

https://www.linkedin.com/in/jeromeroussin/

[krrishdholakia]

Need a way to:

• identify if a request has prompt caching enabled
• check cache for model id, for that request
• if found, route to that model_id
• if none, pick from loadbalancing logic AND store model id for that call (parallel to completion call)

[jeromeroussin]

Thanks. As I have made you aware, the current implementation of this new routing scheme is going to have limited impact in our environment. Azure OpenAi does not align prompt caching with message boundaries, whereas your current algorithm does.

[jeromeroussin]

Can you reopen this since you reverted the PR?