WIP: Introduce Pruning to IAVL #151
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Storing intermidiary IAVL versions in memory and not to disk
Motivation: Both Cosmos and Loom Network save an IAVL version per block, then go back and delete these versions. So you have constant churn on the IAVL and underlying Leveldb database. When realistically what you want is to only store every X Blocks.
At Berlin Tendermint Conference, Zaki and I surmised a plan where new versions are in memory, while still pointing back to nodes on disk to prevent needing to load entire IAVL into main memory. Loom IAVL tree is around 256gb so this is not feasible otherwise.
Usage
OLD Code would be like
```go
hash, version, err := s.tree.SaveVersion()
```
New Caller code would look like
```go
oldVersion := s.Version()
var version int64
var hash []byte
//Every X versions we should persist to disk
if s.flushInterval == 0 || ((oldVersion+1)%s.flushInterval == 0) {
if s.flushInterval != 0 {
log.Error(fmt.Sprintf("Flushing mem to disk at version %d\n", oldVersion+1))
hash, version, err = s.tree.FlushMemVersionDisk()
} else {
hash, version, err = s.tree.SaveVersion()
}
} else {
hash, version, err = s.tree.SaveVersionMem()
}
```
FlushMemVersionDisk:
Flushes the current memory version to disk
SaveVersionMem:
Saves the current tree to memory instead of disk and gives you back an apphash
This is an opt in feature, you have to call new apis to get it.
We also have a PR that demonstrates its usage https://github.com/loomnetwork/loomchain/pull/1232/files
We are now commiting every 1000 blocks, so we store 1000x less. Also we have signficant improves in IO at least double from not having to Prune old versions of the IAVL Tree
|
TODO: Move all pruning logic out of mutableTree and into nodedb |
AdityaSripal
changed the title
|
Had to change base branch to develop as we haven't nuked develop yet on this repo. Once we get this done, then will move base back to master |
|
TODO:
|
|
Closing to create in-org branch |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces notion of pruning to IAVL.
MutableTreehaskeepRecentandkeepEveryfields. Recent versions get saved in memDB while snapshot versions get saved to levelDB. For more details see:#144 (comment)
Based off Loom PR (#150).