-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Description
Currently, the entire database lives in memory and is persisted to a single file on disk. Consider making each registered id be persisted to a separate file. By loading/unloading the file (containing the data from a single id) on user request(s) we will likely make the memory footprint a lot smaller.
Pros:
- Smaller, more manageable files.
- Data from each
idis isolated making them easier to move around and share - Data is loaded/saved based on ID, reducing how
- Its harder to corrupt the entire data set just because of one unclean shutdown.
Cons:
- Although we already persist to disk with every request, PyXIE will be doing a lot more reads from disk, making things slower
- The
DATABASE_FILEconfiguration would change to something likeDATA_PATHwhich some may consider a breaking change, so a major version bump would be recommended. - A data migration process from the single file database to multiple a multi-file database would need to be defined.
This ticket may lend itself well to #25 where each thread that PyXIE runs would receive a request from a queue, then loads the correct file for the id, and processes the request. Unclear how well that strategy would work with Flask, however.
Metadata
Metadata
Assignees
Labels
No labels