DrizzleORM - Db providers (SQLlite, MYSQL, PostgreSQL)#9
Draft
voarsh2 wants to merge 24 commits intosnipeship:mainfrom
Draft
DrizzleORM - Db providers (SQLlite, MYSQL, PostgreSQL)#9voarsh2 wants to merge 24 commits intosnipeship:mainfrom
voarsh2 wants to merge 24 commits intosnipeship:mainfrom
Conversation
Renames all references from "claudeflare" to "ccflare" across documentation, scripts, and metadata for branding consistency. Adds .npmignore to exclude development files from npm package publishing. Introduces a minimal entry point for the upcoming implementation. Updates instructions and scripts to reflect new naming and structure.
Renames all "claudeflare" references (package names, imports, environment variables, config files, DB paths, documentation, branding, and CLI usage) to "ccflare" for clearer, shorter project identity. Updates .gitignore, all package.json files, published binary names, docs, environment variable prefixes, platform directories, and user-facing text. No logic or functional behavior changes; ensures uniform branding and naming consistency across the codebase, CLI, and documentation.
Revises the description in package.json for improved clarity and marketing, changing it from "Claude load balancer proxy" to "Ultimate CC Proxy". Enhances readability and better reflects the project's purpose.
…imizations - Add comprehensive retry logic with exponential backoff for SQLITE_BUSY errors - Configure SQLite with WAL mode, busy timeout, and optimizations for Rook Ceph - Extend configuration system to support database-specific settings - Apply retry logic to critical read operations (getAllAccounts, getAccount, etc.) - Add retry support to key write operations (updateAccountTokens, markAccountRateLimited) - Maintain backward compatibility with existing async writer and proxy systems Resolves database locking issues causing 'loading requests' page hangs on distributed storage
- Fix N+1 query problem in getRequests() by using JOIN instead of individual getAccount() calls - Eliminate redundant API calls in dashboard by enhancing detail handler to include summary data - Add database index on requests.account_used for better JOIN performance - Apply retry logic consistently to getRequestSummaries() function - Reduce database queries from ~201 to 1 for 200 request entries - Reduce API calls from 2 to 1 in dashboard requests page Performance improvements: - Database queries: ~99.5% reduction (201 → 1) - Network requests: 50% reduction (2 → 1) - Better resilience with consistent retry mechanisms
…eneck - Replace dual-query approach with single optimized query from requests table - Eliminate expensive JSON parsing of 200 request payloads on page load - Return only essential summary data (timestamps, status, tokens, cost) for initial view - Add composite index on (timestamp DESC, account_used) for faster query execution - Add lazy loading endpoint /api/requests/payload/:id for individual request details - Reduce data transfer by ~90% and CPU usage by ~95% Performance improvements: - Database queries: 2 → 1 (50% reduction) - JSON parsing: 200 operations → 0 (100% elimination) - Memory usage: ~90% reduction (no large JSON objects) - Network transfer: ~90% smaller response size Resolves slow loading of requests page with 200 entries
CRITICAL FIXES: - Fix TUI core to use optimized queries instead of JSON parsing 200 payloads - Add lazy loading for request details in both web dashboard and TUI - Implement database corruption protection for production stability - Add graceful fallbacks when full payload data unavailable PERFORMANCE IMPROVEMENTS: - TUI: Eliminate 200 JSON.parse() calls, use single optimized query - Dashboard: Add lazy loading with loading states and error handling - Database: Conservative settings for distributed filesystems (Rook Ceph) - API: Dynamic route handling for individual request payloads PRODUCTION STABILITY: - Database integrity checks before configuration - Fallback to DELETE mode if WAL fails on distributed storage - Reduced cache size and disabled memory-mapped I/O for stability - Increased busy timeout for distributed storage latency CONSISTENCY: - Both web and TUI now use same optimized approach - Unified lazy loading pattern across components - Consistent error handling and fallback mechanisms Resolves database corruption issues in production and eliminates performance bottlenecks in requests page loading.
Enhances database configuration management by adding validation to configuration parameters, providing default values for database settings, and improving retry logic. Adds validation to database configuration parameters to prevent invalid settings. Introduces default values for database settings to ensure consistent behavior. Enhances database retry logic by implementing a synchronous retry mechanism and improving error handling. Also, this commit ensures the ccflare user is correctly added to its group in the Dockerfile and improves script execution permissions.
Migrates to a Drizzle ORM-based system supporting SQLite, PostgreSQL, and MySQL. This change provides: - Database provider selection via environment variables or configuration - Proper migration system using `drizzle-kit` - Automated migration and compatibility with legacy SQLite databases - New testing framework - Updated documentation The previous SQLite-only system is now deprecated.
- Resolved conflicts in packages/config/src/index.ts (kept both database config and default_agent_model) - Resolved conflicts in packages/http-api/src/router.ts (added requests stream handler) - Brings in latest features: agent filtering, rate limit progress, real-time streaming, model preferences - Maintains remote-run branch database resilience improvements
Brings in latest features from remote-run branch: - Agent filtering and model preferences - Real-time request streaming via SSE - Account renaming functionality - Rate limit progress indicators - Advanced dashboard filtering - Analytics and model stats - Chart improvements and compact formatting - Browser launching improvements Conflicts resolved: - bun.lock: Regenerated to include all dependencies Note: TypeScript compatibility between DatabaseOperations and DrizzleDatabaseOperations will be addressed in follow-up commits as part of the database modernization effort.
Refactors database interactions to utilize repository methods for enhanced database provider compatibility and adds a fallback to raw SQL queries for specific SQLite analysis. The changes address the issue of SQLite-specific queries within the codebase, promoting a more provider-agnostic approach using DrizzleORM repositories. Updates health check and request handling to use async methods and repository patterns when available, improving overall resilience.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates to a Drizzle ORM-based system supporting SQLite, PostgreSQL, and MySQL.
This change provides:
drizzle-kitThe previous SQLite-only system is now deprecated.
More work is to be done to fully remove the old database code, update TUI.... but given how many changes I made, I prefer a review before continuing - plus I anticipate merge hell.
This looks larger, because it is based on my first PR that is still not merged yet.