Skip to content

refactor: extract shared prompt cache breakpoint layer from 4 providers#11415

Closed
hannesrudolph wants to merge 1 commit intomainfrom
refactor/shared-prompt-cache-layer
Closed

refactor: extract shared prompt cache breakpoint layer from 4 providers#11415
hannesrudolph wants to merge 1 commit intomainfrom
refactor/shared-prompt-cache-layer

Conversation

@hannesrudolph
Copy link
Collaborator

Summary

Moves cache breakpoint placement from individual providers to a shared utility called from Task.ts before createMessage(). Messages arrive at providers pre-annotated with providerOptions, and the AI SDK routes the correct options to the active provider automatically.

Architecture

Task.ts message preparation:
  cleanConversationHistory
    → resolveCacheProviderOptions(provider, modelInfo, settings)
    → applyCacheBreakpoints(messages, options, config)
    → api.createMessage(systemPrompt, annotatedMessages, metadata)

The AI SDK's providerOptions namespacing ensures each provider only picks up its own cache hints — anthropic.cacheControl for Anthropic family, bedrock.cachePoint for Bedrock.

New Files

File Purpose
src/api/transform/prompt-cache.ts resolveCacheProviderOptions() — maps provider name → cache config; applyCacheBreakpoints() — annotates last N non-assistant batches
src/api/transform/__tests__/prompt-cache.spec.ts 14 test cases

Provider Changes

Provider Removed Preserved
anthropic.ts targeting block + applyCacheControlToAiSdkMessages() systemProviderOptions in streamText()
anthropic-vertex.ts targeting block + applyCacheControlToAiSdkMessages() systemProviderOptions in streamText()
minimax.ts targeting block + applyCacheControlToAiSdkMessages() systemProviderOptions in streamText()
bedrock.ts targeting block + applyCachePointsToAiSdkMessages() systemProviderOptions, usePromptCache, supportsAwsPromptCache()

Key Improvements

  1. Fixes tool message caching gap — Targets non-assistant batches (role: user + role: tool) instead of only role: user. After PR feat: implement ModelMessage storage layer with AI SDK response messages #11409, tool results are separate role: tool messages that now correctly receive cache breakpoints.

  2. Single source of truth — Cache strategy defined once in prompt-cache.ts with provider adapter mapping.

  3. Provider-specific config preserved — Bedrock: 3 breakpoints + anchor at ~1/3 for 20-block lookback coverage. Anthropic family: 2 breakpoints. Bedrock opt-in via awsUsePromptCache.

Not Changed

  • systemProviderOptions in all 4 providers
  • OpenAI Native promptCacheRetention (provider-level option, not per-message)
  • No settings types or UI changes (separate scope)

Stats

+426 / -216 lines • 14 new tests • 5,491 total tests pass • 0 regressions

Move cache breakpoint logic from individual providers to a shared utility
called from Task.ts before createMessage(). Messages arrive at providers
pre-annotated with providerOptions, and the AI SDK routes the correct
options to the active provider automatically.

New files:
- src/api/transform/prompt-cache.ts: resolveCacheProviderOptions() +
  applyCacheBreakpoints() with provider adapter mapping
- src/api/transform/__tests__/prompt-cache.spec.ts: 14 test cases

Changes per provider:
- anthropic.ts: removed targeting block + applyCacheControlToAiSdkMessages()
- anthropic-vertex.ts: same
- minimax.ts: same
- bedrock.ts: removed targeting block + applyCachePointsToAiSdkMessages()

Key improvements:
- Targets non-assistant batches (user + tool) instead of only role=user.
  After PR #11409, tool results are separate role=tool messages that now
  correctly receive cache breakpoints.
- Single source of truth: cache strategy defined once in prompt-cache.ts
- Provider-specific config preserved: Bedrock gets 3 breakpoints + anchor,
  Anthropic family gets 2 breakpoints

Preserved (untouched):
- systemProviderOptions in all providers' streamText() calls
- OpenAI Native promptCacheRetention (provider-level, not per-message)
- Bedrock usePromptCache opt-in + supportsAwsPromptCache()

5,491 tests pass, 0 regressions.
@dosubot dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Feb 11, 2026
@roomote
Copy link
Contributor

roomote bot commented Feb 11, 2026

Rooviewer Clock   See task

Clean refactor overall. One issue flagged around Bedrock's cache checkpoint budget.

  • Bedrock maxMessageBreakpoints = 3 + anchor can produce 5 total checkpoints (including system), exceeding the documented 4-checkpoint limit

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.

applyCacheBreakpoints(
cleanConversationHistory,
cacheOptions,
isBedrock ? 3 : 2,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old bedrock code documented a hard limit of 4 cache checkpoints and budgeted them as 1 system + 2 trailing + 1 anchor = 4. Here, maxMessageBreakpoints = 3 for Bedrock plus useAnchor = true yields up to 3 trailing + 1 anchor = 4 message-level breakpoints, plus the system prompt checkpoint applied in bedrock.ts via systemProviderOptions. That totals 5, which exceeds the documented 4-checkpoint limit and could cause Bedrock API errors once a conversation is long enough to trigger the anchor. Changing the trailing count to 2 (matching the old behavior) would keep the total at 4.

Suggested change
isBedrock ? 3 : 2,
isBedrock ? 2 : 2,

Fix it with Roo Code or mention @roomote and request a fix.

@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap Feb 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments