Storing intermediary IAVL versions in memory and not to disk #150
Conversation
Storing intermidiary IAVL versions in memory and not to disk
Motivation: Both Cosmos and Loom Network save an IAVL version per block, then go back and delete these versions. So you have constant churn on the IAVL and underlying Leveldb database. When realistically what you want is to only store every X Blocks.
At Berlin Tendermint Conference, Zaki and I surmised a plan where new versions are in memory, while still pointing back to nodes on disk to prevent needing to load entire IAVL into main memory. Loom IAVL tree is around 256gb so this is not feasible otherwise.
Usage
OLD Code would be like
```go
hash, version, err := s.tree.SaveVersion()
```
New Caller code would look like
```go
oldVersion := s.Version()
var version int64
var hash []byte
//Every X versions we should persist to disk
if s.flushInterval == 0 || ((oldVersion+1)%s.flushInterval == 0) {
if s.flushInterval != 0 {
log.Error(fmt.Sprintf("Flushing mem to disk at version %d\n", oldVersion+1))
hash, version, err = s.tree.FlushMemVersionDisk()
} else {
hash, version, err = s.tree.SaveVersion()
}
} else {
hash, version, err = s.tree.SaveVersionMem()
}
```
FlushMemVersionDisk:
Flushes the current memory version to disk
SaveVersionMem:
Saves the current tree to memory instead of disk and gives you back an apphash
This is an opt in feature, you have to call new apis to get it.
We also have a PR that demonstrates its usage https://github.com/loomnetwork/loomchain/pull/1232/files
We are now commiting every 1000 blocks, so we store 1000x less. Also we have signficant improves in IO at least double from not having to Prune old versions of the IAVL Tree
|
Thank you for the PR @mattkanwisher!! Getting some folks to review this! |
| @@ -165,11 +185,16 @@ func (ndb *nodeDB) DeleteVersion(version int64, checkLatestVersion bool) { | |||
| // Saves orphaned nodes to disk under a special prefix. | |||
| // version: the new version being saved. | |||
| // orphans: the orphan nodes created since version-1 | |||
| func (ndb *nodeDB) SaveOrphans(version int64, orphans map[string]int64) { | |||
| func (ndb *nodeDB) SaveOrphans(version int64, orphans map[string]int64, flushToDisk bool) { | |||
There was a problem hiding this comment.
What is the purpose of parameter flushToDisk bool? It seems useless in SaveOrphans
|
When we restart the node, un-persisted changes in memory will be lost and that means the latest version of state may not persisted. will it work when the node restart and find out could not load the root node of the latest version. should we replay the blocks to get the versions not persisted when restart. It seems in loomchain there is just one app store, so it may works for tendermint will relay all blocks automatically if the height of state and block height do not match. but it may does not work in cosmos for it have multi app store. |
|
Concerned with the parametrization of the new functions, in particular
|
|
|
||
| // DeleteVersionFull deletes a tree version from disk or memory based on the flag. The version can then no | ||
| // longer be accessed. | ||
| func (tree *MutableTree) DeleteVersionFull(version int64, memDeleteAlso bool) error { |
There was a problem hiding this comment.
Can this be refactored into two functions DeleteVersionMem and DeleteVersionDisk? Then DeleteVersion can just call both functions.
|
|
||
| // FlushMemDisk saves a new tree to disk and removes all the versions in memory | ||
| func (tree *MutableTree) FlushMemVersionDisk() ([]byte, int64, error) { | ||
| x, y, err := tree.saveVersion(true) |
There was a problem hiding this comment.
could we have more descriptive vars
|
on second thought, we can also flush mem version asynchronously. Assume that we have high throughput, though we save state every x blocks, but when it comes to that block, the blocking time may increased. So if we can flush mem version to disk asynchronously, we will have the same performance for each block. |
|
@yutianwu yeah doing this in a goroutine is def best practice |
| func (tree *MutableTree) FlushMemVersionDisk() ([]byte, int64, error) { | ||
| x, y, err := tree.saveVersion(true) | ||
| tree.ndb.dbMem = dbm.NewMemDB() | ||
| tree.memVersions = map[int64]bool{} |
There was a problem hiding this comment.
IMHO, if we do not delete all memVersions and keep recent n version would be better, for it would fail when query recent version of state like querystore, if height is not exactly the latest one, it would fail for the version is just deleted.
|
Closing this in favor of #158 |
|
@mattkanwisher how often are you flushing versions to disk with loom? Have you compared the results of various pruning strategies. Are you benchmarking your application with various pruning strategies or are you benchmarking by just testing IAVL in isolation? Would be very useful to hear your approach to benchmarking this and the results you got. thanks |
Storing intermediary IAVL versions in memory and not to disk
Motivation: Both Cosmos and Loom Network save an IAVL version per block, then go back and delete these versions. So you have constant churn on the IAVL and underlying Leveldb database. When realistically what you want is to only store every X Blocks.
At Berlin Tendermint Conference, Zaki and I surmised a plan where new versions are in memory, while still pointing back to nodes on disk to prevent needing to load entire IAVL into main memory. Loom IAVL tree is around 256gb so this is not feasible otherwise.
Usage
OLD Code would be like
New Caller code would look like
FlushMemVersionDisk:
Flushes the current memory version to disk
SaveVersionMem:
Saves the current tree to memory instead of disk and gives you back an apphash
This is an opt in feature, you have to call new apis to get it.
We also have a PR that demonstrates its usage https://github.com/loomnetwork/loomchain/pull/1232/files
We are now commiting every 1000 blocks, so we store 1000x less. Also we have signficant improves in IO at least double from not having to Prune old versions of the IAVL Tree