-
Notifications
You must be signed in to change notification settings - Fork 1
Closed
Labels
coreenhancementNew feature or requestNew feature or requestmemoryPersistence and memoryPersistence and memorysize/S
Description
Parent: #740 (P0)
Problem
Current estimate_tokens() uses bytes/3 heuristic which overestimates on multibyte text (Cyrillic, CJK — up to 2-3x). This causes premature context compaction and inaccurate budget allocation.
Solution
- Replace
text.len() / 3withtext.chars().count() / 4 - Add configurable safety margin (default 1.0, recommended 1.2 for production)
- Optionally support
tiktoken-rsbehind a feature flag for precise counting with cloud providers
Affected crates
zeph-memory(estimate_tokensfunction)zeph-core(all call sites)
Acceptance criteria
- Estimation accuracy within 20% on mixed ASCII/Cyrillic/CJK text
- Safety margin configurable via
memory.token_safety_margin - Existing tests updated
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
coreenhancementNew feature or requestNew feature or requestmemoryPersistence and memoryPersistence and memorysize/S