Replace SQLite with in-memory pickle-based index #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Major architectural change: removes SQLite dependency and replaces it with a simpler in-memory dictionary backed by pickle persistence. Changes: - Remove sqlite3 dependency and all SQL queries - Replace index.db with index.pkl (pickled Python dict) - Add crash recovery: can rebuild index from Arrow segment files - Add atomic index persistence with temp file swap - Update tests to verify index persistence and rebuild - Update README documentation (partial) - Add debug logging in decorator for cache operations Benefits: - Simpler architecture with fewer dependencies - Faster startup (no database initialization) - Better crash recovery (auto-rebuild from segments) - Same O(1) lookup performance - Fewer moving parts to debug Breaking change: Existing caches with index.db will need to be rebuilt. The old index.db file will be ignored and a new index.pkl will be created. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Complete documentation update removing all SQLite references and updating to describe the new pickle-based in-memory index architecture. Changes: - Update README.md architecture section with pickle index details - Completely rewrite ARCHITECTURE.md SQLite sections - Update module docstrings (__init__.py, backend.py) - Update test comments and implementation in test_recovery.py - Replace SQLite B-tree references with dict/pickle terminology - Update performance characteristics and memory usage notes - Replace SQLite WAL reference with Pickle Protocol in references Key updates: - Storage: index.db → index.pkl - Index type: SQLite B-tree → Python dict with pickle persistence - Crash recovery: WAL → auto-rebuild from segments - Memory model: Updated to reflect in-memory index scaling 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Update test files that still referenced old SQLite attributes: - test_edge_cases.py: Change backend.db_path to backend.index_path - test_recovery.py: Add missing pyarrow import, update conn references to use backend.index All 72 tests now passing (2 skipped for CUDA unavailability) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Remove unused imports (time, shutil, os) - Replace typing.List with list (modern Python syntax) - Prefix unused variables with underscore - Remove unused f-string prefixes - Remove unused HAS_PSUTIL variable All ruff checks now passing. All tests still passing (72 passed, 2 skipped). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment Thanks for integrating Codecov - We've got you covered ☂️ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces the SQLite-based index with a simpler in-memory pickle-based index, reducing complexity while maintaining performance and improving crash recovery.
Changes
Core Migration
index.db(SQLite) →index.pkl(pickled Python dict)sqlite3imports and SQL queriesKey Files Modified
src/torchcachex/backend.py- Complete rewrite of index management_load_index()- loads pickle or rebuilds from segments_save_index()- atomic pickle persistence_rebuild_index_from_segments()- crash recoverytests/test_backend.py- Added new tests for index persistence and rebuildtests/test_recovery.py- Updated to test pickle-based recoverytests/test_edge_cases.py- Fixed references to new index attributesDocumentation Updates
__init__.pyandbackend.pyCode Quality
Benefits
✅ Simpler architecture
✅ Faster lookups
✅ Better crash recovery
✅ More compact storage
Trade-offs
Testing
All tests passing:
```
======================== 72 passed, 2 skipped in 0.96s =========================
```
Breaking Changes
No user code changes required - the migration is transparent.
Performance
No regression in performance:
🤖 Generated with Claude Code