Skip to content

Reduce memory dependence; persist to multiple files #31

@dev-dull

Description

@dev-dull

Currently, the entire database lives in memory and is persisted to a single file on disk. Consider making each registered id be persisted to a separate file. By loading/unloading the file (containing the data from a single id) on user request(s) we will likely make the memory footprint a lot smaller.

Pros:

  • Smaller, more manageable files.
  • Data from each id is isolated making them easier to move around and share
  • Data is loaded/saved based on ID, reducing how
  • Its harder to corrupt the entire data set just because of one unclean shutdown.

Cons:

  • Although we already persist to disk with every request, PyXIE will be doing a lot more reads from disk, making things slower
  • The DATABASE_FILE configuration would change to something like DATA_PATH which some may consider a breaking change, so a major version bump would be recommended.
  • A data migration process from the single file database to multiple a multi-file database would need to be defined.

This ticket may lend itself well to #25 where each thread that PyXIE runs would receive a request from a queue, then loads the correct file for the id, and processes the request. Unclear how well that strategy would work with Flask, however.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions