Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add CLI indexing and improve notes processing #3

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

camwest
Copy link

@camwest camwest commented Dec 18, 2024

Context:

  • I have ~3000 notes and it was just hanging
  • I added batching with an async iterator and it works pretty well now

Changes:

  • Add direct CLI command for indexing notes (bun run index-notes)
  • Implement batched note processing to handle large collections
  • Add timeout handling and progress reporting
  • Improve error handling and recovery
  • Add comprehensive test coverage for note processing
  • Clean up database connections on shutdown

- Add direct CLI command for indexing notes (`bun run index-notes`)
- Implement batched note processing to handle large collections
- Add timeout handling and progress reporting
- Improve error handling and recovery
- Add comprehensive test coverage for note processing
- Clean up database connections on shutdown
@timdesrochers
Copy link

timdesrochers commented Dec 22, 2024

Running bun run index-notes... I had to change the timeout value from 60000 to 120000 on lines 168 and 190 of index.ts in order to prevent timeout errors. I've got 1721 notes... so far it seems to be indexing just fine (425/1721 as of right now).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants