Enable bedrock prompt cache

[fbalicchia]

Summary

Implemented prompt caching for Anthropic Claude models on AWS Bedrock to reduce costs
Introduced an intelligent cache point placement strategy that complies with AWS Bedrock’s four cache point limitation.
Added the BEDROCK_ENABLE_CACHING configuration parameter

Testing

• Verify caching is enabled for Claude models by default
• Test that cache points are correctly placed in system prompts
• Test that cache points are correctly placed in user/assistant messages
• Verify cache point limit (4 max) is respected

[fbalicchia]

@jh-block and @blackgirlbytes, I’ve created a fresh PR. Thanks!

[jh-block]

Thanks! The implementation looks good. I have a couple of questions:

1. It wasn't totally clear to me how tool messages are being handled, since they can't be cache points. Do we explicitly filter them out somewhere? Could we verify that with test(s)?
2. Did the strategy of "most recent 3" come from any analysis? A comment explaining the choice would be good. I (perhaps naïvely) would have thought caching the earliest messages would have better ROI. (This is just for discussion, I think it's perfectly fine to merge the implementation and then iterate on the cache strategy if necessary.)

[fbalicchia]

Thanks for your feedback

For question 1:

Current state: Cache points are added at the message level (after all content blocks), regardless of whether the message contains a ToolRequest or a ToolResponse.
There are currently no tests verifying that this behavior is correct. A comment in bedrock.rs:178 mentions that “tools don’t support cache points,” but the intent and correctness of this are unclear.
I plan to fix this and add appropriate tests.

For question 2:
AWS Bedrock enforces a strict limit of 4 cache checkpoints per request. The current implementation correctly allocates them as 1 system message + 3 user/assistant messages, for a total of 4.
https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html

This means you can only cache 3 messages in addition to the system message.
At the moment, the system caches the most recent 3 messages, so every time a new message is added, the “last 3” shift. This is not very efficient.

A better approach might be to cache the earliest messages instead of the most recent ones, since early messages tend to remain stable, while recent messages change on every turn and frequently invalidate the cache.

Tell me what you think

[fbalicchia]

@jh-block and @blackgirlbytes, I’ve applied the requested changes

[fbalicchia]

Hi @michaelneale, when you have a moment, could you kindly review this PR? Thank you.”

[fbalicchia]

Hi @jh-block ,
Is there any chance this could be reviewed?
Thank you!

[jh-block]

Thanks for the changes. I have a couple comments:

• In bedrock.rs we're now caching the earliest 3 messages which is good (I think we can improve further, but this is fine for now), but some tests still seem to test the "last 3 messages" strategy: test_cache_point_allocation_without_tools, test_cache_point_allocation_with_tools, test_cache_point_limit_respected_with_few_messages, test_cache_points_with_few_messages. Actually, looking at those tests, they don't seem to be actually testing anything since they duplicate the logic inside the test rather than calling the actual code. There are other tests like this, e.g. test_max_four_cache_points_respected is just testing that 1 + 3 == 4. These tests can just be removed, or if appropriate, updated to actually test the production code.
• to_bedrock_message() in formats/bedrock.rs is now dead code, as it's only used in a test: test_to_bedrock_message_without_caching. If we're never using bedrock without caching, this test and to_bedrock_message() can be removed.

[jh-block]

Hey @fbalicchia, let me know when this is ready for review, I have a few small fixes but I can just push them up if that's OK with you, so we can get this merged sooner

[fbalicchia]

@jh-block go ahead thanks