feat: add cache_control support for Claude models in OpenAI provider

[HikaruEgashira]

Adds conditional cache_control functionality to OpenAI provider when model name contains "claude", enabling prompt caching for LiteLLM and other OpenAI-compatible services.

Fixes #3333

[michaelneale]

@HikaruEgashira thanks for this - can you run cargo fmt/check clippy etc - also, is will this work specifically with openrouter - what provider are you testing it with that uses the openai provider in this case?

[michaelneale]

/// Update the request when using anthropic model.
/// For anthropic model, we can enable prompt caching to save cost. Since openrouter is the OpenAI compatible
/// endpoint, we need to modify the open ai request to have anthropic cache control field.

Does openrouter not do this as it is offering the openai compatible api? that seem really odd if they aren't? why would the offer an openai like api?

[michaelneale]

any chance we can confirm before/after caching with openai to know it does need this header? (still seems odd to me)

[HikaruEgashira]

I tested this program with LiteLLM. This header does not work with pure OpenAI but most OpenAI Compatible will work. I provide the usage from the system log.

before (1.0.29)

\"model\": \"anthropic.claude-3-5-haiku-20241022-v1:0\",\n  \"object\": \"chat.completion\",\n  \"system_fingerprint\": null,\n  \"usage\": {\n    \"cache_creation_input_tokens\": 0,\n    \"cache_read_input_tokens\": 0,\n    \"completion_tokens\": 554,\n    \"completion_tokens_details\": null,\n    \"prompt_tokens\": 5902,\n    \"prompt_tokens_details\": {\n      \"audio_tokens\": null,\n      \"cached_tokens\": 0\n    },\n    \"total_tokens\": 6456\n  }\n}","input_tokens":"5902","output_tokens":"554","total_tokens":"6456"},"target":"goose::providers::utils","span":{"name":"complete"},"spans":[{"name":"complete"}]}

after

\"model\": \"anthropic.claude-3-5-haiku-20241022-v1:0\",\n  \"object\": \"chat.completion\",\n  \"system_fingerprint\": null,\n  \"usage\": {\n    \"cache_creation_input_tokens\": 0,\n    \"cache_read_input_tokens\": 7593,\n    \"completion_tokens\": 68,\n    \"completion_tokens_details\": null,\n    \"prompt_tokens\": 8025,\n    \"prompt_tokens_details\": {\n      \"audio_tokens\": null,\n      \"cached_tokens\": 7593\n    },\n    \"total_tokens\": 8093\n  }\n}","input_tokens":"8025","output_tokens":"68","total_tokens":"8093"},"target":"goose::providers::utils","span":{"name":"complete"},"spans":[{"name":"complete"}]}

https://docs.litellm.ai/docs/completion/prompt_caching

[cgwalters]

Shouldn't we add a method to the Provider trait for this instead?

[michaelneale]

I think if liteLLM offers an openai api - that should include things like caching abstraction on it, that should be part of its job.

[michaelneale]

I think we should have a liteLLM provider specifically - we shouldn't be changing the openai provider for specific middleware routers which are lacking features (but if we have a liteLLM provider - that would be ideal - and leave the openai one as it is).

This could be done by cloning the openai provider, and adding in liteLLM specific code (and things like this) which is a better experience as well (as there likely will be other things like this which are liteLLM specific - as liteLLM is important enough I think to justify this)

[HikaruEgashira]

OK! I'll add new provider. Thanks for reviewing.