Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate/implement SQLite-based or other BlockStore implementation for Windows #86

Closed
pchiusano opened this issue Aug 12, 2016 · 0 comments

Comments

@pchiusano
Copy link
Member

At the moment, we do not have a fast, low-memory, BlockStore implementation that runs on all platforms. The current implementations have the following limitations:

  • MemBlockStore is totally ephemeral - it's state doesn't survive node container shutdown. Useful mainly for testing and for use by a standalone JS editor.
  • FileBlockStore uses acid-state, so it persists its state, but keeps the entire store in memory. It also is apparently somewhat slow? I'm guessing the performance could be improved with tuning of how we use acid-state, but keeping the entire store in memory is not going to work long-term.
  • LevelDbStore uses LevelDB, which works nicely but is painful to build on windows. We want people to be able to easily run Unison nodes on their windows machines, too, so that is a problem (unless we can figure out a good story for building on windows).

So, this task is to investigate a SQLite-backed implementation of BlockStore. SQLite runs on pretty much everything (Windows phone 8 anyone??) and I'm guessing would be easy to integrate into our build. For implementation strategy, it seems like it would work to represent the BlockStore with one two-column table, with columns (hash, blob), with an index on the hash column. Then one more table for tracking the series - (seriesid, hash), with an index on seriesid.

If there's another good embedded key-value store option on Windows, that would be fine also.

There seem to be a couple sqlite bindings on hackage. Not sure which is best.

For questions, @sfultong is something of an expert on BlockStore (he produced the three existing implementations). I'm also available for general support. @sfultong if you decide you'd like to implement this, feel free to claim it - just assign it to yourself. But this might be a good project for someone else to get involved with as a way to get their feet wet with the project.

One additional requirement - the implementation strategy should be obviously safe against SQL injection attacks. No manually gluing raw strings together to construct queries, obviously. Tests should include some 'Bobby tables' style queries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant