Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor to go fully in-memory first, and persistent second #99

Open
matko opened this issue Jun 27, 2022 · 0 comments
Open

Refactor to go fully in-memory first, and persistent second #99

matko opened this issue Jun 27, 2022 · 0 comments
Labels
triage issue to be triage

Comments

@matko
Copy link
Member

matko commented Jun 27, 2022

A lot of the data structures are currently written around the idea that we're persisting layers directly to some persistent storage. The memory store is then implemented as a special kind of persistence which just saves things in a hashmap.

This is kinda weird and the wrong way around. We actually almost always want to build up stuff in memory first, and only then (possibly after some more checks, like schema checking) persist to disk.

Therefore, we should consider refactoring terminusdb-store to be in-memory first. That means that none of the data structures, nor the layer structures, should have to know about an underlying storage method (FileLoad, FileStore, etc), but instead directly work on Bytes. Builders likewise should not have to know anything about storage, but build directly into Bytes.

Doing all this will also reduce the complexity of the typing in store, as a lot of that is now complicated by the fact that in many places we want to work with builders without knowing this underlying storage method. If all actual persisting logic is moved into the various LayerStores, we probably don't have to include FileLoad or FileStore in any of the typing anymore.

@github-actions github-actions bot added the triage issue to be triage label Jun 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
triage issue to be triage
Projects
None yet
Development

No branches or pull requests

1 participant