Port Python Temporal Scoring System to Rust

[nikomatsakis]

Port Python Temporal Scoring System to Rust

Status: In Progress - Implementation Complete, Debugging Required

Overview

Port the sophisticated temporal scoring system from the Python implementation to Rust to achieve feature parity. The Python version uses a research-based formula: 30% recency + 20% frequency + 35% importance + 15% context with access tracking and metadata management.

Work Completed Today (2025-08-02)

✅ Core Implementation

• Added daily_access_counts field to Insight struct for tracking access patterns
• Created constants module with all temporal scoring parameters matching Python values
• Implemented temporal scoring methods on Insight:
  • record_access() - tracks accesses by active day
  • calculate_frequency() - sliding window frequency calculation
  • calculate_recency_score() - exponential decay based on days since last access
• Rewrote search algorithm to use sophisticated composite scoring instead of simple semantic similarity
• Updated MCP server to get current active day and record initial access on insight creation
• Added access tracking to search results for frequency learning

✅ Infrastructure Changes

• Added constants.rs module with all scoring weights and parameters
• Updated search method signature to include current_active_day parameter
• Fixed all tests to use new search signature
• Added record_insight_access() method to storage layer

Current Issue: Serialization/Execution Problem

Symptom: The daily_access_counts field is not appearing in saved JSON files, and no metadata.json file is being created.

Analysis: This suggests the new temporal scoring code paths are not being executed at all, despite successful compilation.

Debugging Attempted:

• Added debug logging to insight creation and access recording
• Verified server binary is updated and installed
• Checked MCP server registration includes debug flags
• Confirmed code compiles and tests pass

Next Steps Needed:

1. Verify the new code is actually being called by adding simple print statements
2. Check if there are silent errors in get_current_active_day() method
3. Test metadata system in isolation to confirm it works
4. Investigate if MCP server error handling is swallowing exceptions

Technical Details

Key Files Modified:

• rs/src/models.rs - Added temporal scoring methods and daily_access_counts field
• rs/src/constants.rs - New file with all scoring parameters
• rs/src/search.rs - Complete rewrite of search algorithm
• rs/src/main.rs - Updated MCP server to use temporal scoring
• rs/src/storage.rs - Added access tracking method

Algorithm Implementation:

let final_relevance = RELEVANCE_WEIGHT_RECENCY * recency_score
    + RELEVANCE_WEIGHT_FREQUENCY * normalized_frequency  
    + RELEVANCE_WEIGHT_IMPORTANCE * current_importance
    + RELEVANCE_WEIGHT_CONTEXT * situation_relevance;

Constants Match Python:

• RELEVANCE_WEIGHT_RECENCY = 0.30
• RELEVANCE_WEIGHT_FREQUENCY = 0.20
• RELEVANCE_WEIGHT_IMPORTANCE = 0.35
• RELEVANCE_WEIGHT_CONTEXT = 0.15
• RECENCY_DECAY_RATE = 0.05
• FREQUENCY_WINDOW_DAYS = 30

Success Criteria

• daily_access_counts field appears in insight JSON files
• metadata.json file is created and updated
• Search results show temporal relevance scoring (not just semantic similarity)
• Access tracking works (insights accessed via search get frequency updates)
• All existing tests pass
• New temporal scoring matches Python behavior

Notes

The implementation is architecturally complete and follows the Python design exactly. The issue appears to be in execution/debugging rather than design. Once the serialization issue is resolved, the temporal scoring system should be fully functional.