Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deep dive into underlying databases: RocksDB and BadgerDB #10

Closed
i-norden opened this issue Apr 27, 2021 · 1 comment
Closed

Deep dive into underlying databases: RocksDB and BadgerDB #10

i-norden opened this issue Apr 27, 2021 · 1 comment

Comments

@i-norden
Copy link
Collaborator

i-norden commented Apr 27, 2021

Outline precisely how we will support both using the MapStore interface (with batching, versioning, and DB snapshotting).

@roysc
Copy link
Collaborator

roysc commented May 7, 2021

ADR-040: Updates for SMT

As part of the proposed ADR-040, we want to update the SMT data structure so it can use Tendermint DB as a node database. This will entail implementing the MapStore interface for tm-db and its new backends, RocksDB and BadgerDB.

Batching

Batched writes are already implemented in tm-db for both backends. These could simply be wrapped in a method on SmapStore, but there is no clear benefit to using them in the SMT for the currently implemented operations. However, if we want to implement efficient batched writes to the SMT itself, that could also be supported in the following way.

The tree will need to be able to read as well as write to the batch itself while building it, using something closer to a transaction object.

  • RocksDB has a Write Batch With Index utility which allows this.
  • BadgerDB's WriteBatch does not support Gets, but, as it is a relatively simple wrapper around a transaction, we could just use Txn.

These types would be wrapped in the MapStore interface. The SMT would need a new interface type to represent the batch (e.g. WriteBatch). This would wrap a derived SparseMerkleTree object using the transaction object as its MapStore.

Versioning & snapshots

Creating state sync snapshots is not required for SMT data. cosmos#8430 (comment).

It seems versioning is also not needed, but it's not entirely clear. However, if so, this could be implemented with a MapStore.GetAt([]byte, uint64) method, which forwards to a (new) tmdb.DB.GetAt([]byte, uint64) method, implemented like so:

  • BadgerDB in managed mode allows arbitrary integers to be used as "timestamp" versions, so this can basically be forwarded directly.
  • For RocksDB, a Checkpoint can be created for each commit in a path corresponding to the version integer, and then used as a read-only copy of the DB.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants