refactor: extract shared prompt cache breakpoint layer from 4 providers#11415
refactor: extract shared prompt cache breakpoint layer from 4 providers#11415hannesrudolph wants to merge 1 commit intomainfrom
Conversation
Move cache breakpoint logic from individual providers to a shared utility called from Task.ts before createMessage(). Messages arrive at providers pre-annotated with providerOptions, and the AI SDK routes the correct options to the active provider automatically. New files: - src/api/transform/prompt-cache.ts: resolveCacheProviderOptions() + applyCacheBreakpoints() with provider adapter mapping - src/api/transform/__tests__/prompt-cache.spec.ts: 14 test cases Changes per provider: - anthropic.ts: removed targeting block + applyCacheControlToAiSdkMessages() - anthropic-vertex.ts: same - minimax.ts: same - bedrock.ts: removed targeting block + applyCachePointsToAiSdkMessages() Key improvements: - Targets non-assistant batches (user + tool) instead of only role=user. After PR #11409, tool results are separate role=tool messages that now correctly receive cache breakpoints. - Single source of truth: cache strategy defined once in prompt-cache.ts - Provider-specific config preserved: Bedrock gets 3 breakpoints + anchor, Anthropic family gets 2 breakpoints Preserved (untouched): - systemProviderOptions in all providers' streamText() calls - OpenAI Native promptCacheRetention (provider-level, not per-message) - Bedrock usePromptCache opt-in + supportsAwsPromptCache() 5,491 tests pass, 0 regressions.
Clean refactor overall. One issue flagged around Bedrock's cache checkpoint budget.
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
| applyCacheBreakpoints( | ||
| cleanConversationHistory, | ||
| cacheOptions, | ||
| isBedrock ? 3 : 2, |
There was a problem hiding this comment.
The old bedrock code documented a hard limit of 4 cache checkpoints and budgeted them as 1 system + 2 trailing + 1 anchor = 4. Here, maxMessageBreakpoints = 3 for Bedrock plus useAnchor = true yields up to 3 trailing + 1 anchor = 4 message-level breakpoints, plus the system prompt checkpoint applied in bedrock.ts via systemProviderOptions. That totals 5, which exceeds the documented 4-checkpoint limit and could cause Bedrock API errors once a conversation is long enough to trigger the anchor. Changing the trailing count to 2 (matching the old behavior) would keep the total at 4.
| isBedrock ? 3 : 2, | |
| isBedrock ? 2 : 2, |
Fix it with Roo Code or mention @roomote and request a fix.
Summary
Moves cache breakpoint placement from individual providers to a shared utility called from
Task.tsbeforecreateMessage(). Messages arrive at providers pre-annotated withproviderOptions, and the AI SDK routes the correct options to the active provider automatically.Architecture
The AI SDK's
providerOptionsnamespacing ensures each provider only picks up its own cache hints —anthropic.cacheControlfor Anthropic family,bedrock.cachePointfor Bedrock.New Files
src/api/transform/prompt-cache.tsresolveCacheProviderOptions()— maps provider name → cache config;applyCacheBreakpoints()— annotates last N non-assistant batchessrc/api/transform/__tests__/prompt-cache.spec.tsProvider Changes
anthropic.tsapplyCacheControlToAiSdkMessages()systemProviderOptionsinstreamText()anthropic-vertex.tsapplyCacheControlToAiSdkMessages()systemProviderOptionsinstreamText()minimax.tsapplyCacheControlToAiSdkMessages()systemProviderOptionsinstreamText()bedrock.tsapplyCachePointsToAiSdkMessages()systemProviderOptions,usePromptCache,supportsAwsPromptCache()Key Improvements
Fixes tool message caching gap — Targets non-assistant batches (
role: user+role: tool) instead of onlyrole: user. After PR feat: implement ModelMessage storage layer with AI SDK response messages #11409, tool results are separaterole: toolmessages that now correctly receive cache breakpoints.Single source of truth — Cache strategy defined once in
prompt-cache.tswith provider adapter mapping.Provider-specific config preserved — Bedrock: 3 breakpoints + anchor at ~1/3 for 20-block lookback coverage. Anthropic family: 2 breakpoints. Bedrock opt-in via
awsUsePromptCache.Not Changed
systemProviderOptionsin all 4 providerspromptCacheRetention(provider-level option, not per-message)Stats
+426 / -216 lines • 14 new tests • 5,491 total tests pass • 0 regressions