Allow EVM state to be flushed to disk every Nth block instead of every block #1532

pathornteng · 2019-10-09T06:29:32Z

Currently, we flush Patricia trie every tx. This commit operation is really expensive and it causes a lot of IOPS. Furthermore, once the trie is committed to disk, web3/truffle normally queries the newly committed data so the app loads the recently committed trie from disk shortly after.

This PR makes commit more efficient by holding multiple tries in memory and only flushing them every N blocks. This should significantly reduce the disk usage (less trie versions) and also IOPS (recent tries cached in memory)

Dependency: loomnetwork/go-ethereum#9

I added unit tests for any code that added
I updated the CHANGELOG.md
All IP is original and not copied from another source
I assign all copyright to Loom Network for the code in the pull request

app.go

pathornteng · 2019-11-04T03:42:57Z

According to my own performance test, this PR

Reduce the growth rate up tp 50%
Reduce processing time for block replay up to 45%

store/evmstore.go

rpc/query_server.go

Due to the flush interval in the EvmStore some EVM state roots may never be written to disk, so it's important to check that the root of the state returned by EVMState.GetSnapshot() at a particular height matches the EVM state root stored in the app store at that height, otherwise the EVM state and app state returned by Application.ReadOnlyState() may end up being out of sync.

The tests are supposed to test how the vm-prefixed keys are handled, replacing the prefix makes the tests fairly pointless, so the prefix has been restored. A couple of the snapshot tests have been removed because the EvmStore no longer caches writes until Commit, it now writes them out immediately.

The EVM root in the VersionedCachingStore cache wasn't being updated in VersionedCachingStore.SaveVersion() which meant that the snapshots obtained from the VersionedCachingStore contained a stale EVM root that didn't match the EVM state in the snapshot. This hack updates the cached EVM root in VersionedCachingStore.SaveVersion() but it's not a proper fix for the issue, the VersionedCachingStore shouldn't be aware of the EVM root at all. The most sensible thing to do is probably to eliminate the MultiWriterAppStore itself and update Application.Commit() to call EvmStore.Commit(), and then store the EVM root in the VersionedCachingStore.

e2e/tests/truffle/test/EvmSnapshot.js

store/iavlstore.go

…mit()

pathornteng · 2019-12-11T07:32:08Z

rebuild sesame

This setting works similarly to AppStore.IAVLFlushInterval.

enlight · 2019-12-12T06:50:53Z

Previously the EVM accessed data like so EVM -> VersionedCachingStore -> MultiWriterAppStore -> EvmStore -> evm.db, this PR changes it to EVM -> EVMState -> EvmStore -> evm.db, which means that the VersionedCachingStore is completely bypassed for EVM reads & writes, leaving only LevelDB caching.

Newer versions of go-ethereum (https://github.com/ethereum/go-ethereum/releases/tag/v1.8.19 or later) have added a cache to the trie.Database (ethereum/go-ethereum#18087) created by the EvmStore, this trie cache should compensate for the loss of VersionedCachingStore so we should update our go-ethereum fork.

https://github.com/ethereum/go-ethereum/releases/tag/v1.8.19 is the release in which the trie cache was initially added, it was optimized in https://github.com/ethereum/go-ethereum/releases/tag/v1.9.0 (ethereum/go-ethereum#19307), and then again in https://github.com/ethereum/go-ethereum/releases/tag/v1.9.8 (ethereum/go-ethereum#19971). Need to figure out if we can just bump our go-ethereum to v1.9.8.

enlight · 2019-12-13T07:21:41Z

There's also ethereum/go-ethereum#19953 (and a follow up PR ethereum/go-ethereum#20113 to fix some issues introduced by that PR) that was merged in go-ethereum v1.9.4 that may impact the trie.Database performance, will need to review it to make sure our usage of trie.Database is compatible with the changes in that PR.

enlight · 2019-12-16T04:57:52Z

In a follow up PR:

Eliminate call to EvmStore.GetSnapshot() from MultiWriterAppStore.GetSnapshotAt(). In fact we can probably eliminate GetSnapshot() from all the stores, only GetSnapshotAt() is being used now. (see Eliminate VersionedKVStore.GetSnapshot() #1607)
Eliminate writes to the EvmStore from MultiWriterAppStore. (Cleanup MultiWriterAppStore #1609)
Eliminate Get/Has/Set/Delete from EvmStore. (Cleanup MultiWriterAppStore #1609)
Fix VersionedCachingStore.GetSnapshotAt() to be atomic. (see Eliminate VersionedKVStore.GetSnapshot() #1607)
~~Remove the hack from VersionedCachingStore.SaveVersion() and fix Application.Commit() instead. This should be done last probably since it's not that important.~~ (Don't remember what this was about anymore)

enlight · 2019-12-17T04:34:10Z

Previously the EVM accessed data like so EVM -> VersionedCachingStore -> MultiWriterAppStore -> EvmStore -> evm.db, this PR changes it to EVM -> EVMState -> EvmStore -> evm.db, which means that the VersionedCachingStore is completely bypassed for EVM reads & writes, leaving only LevelDB caching.

Actually this data access flow is only accurate when considering queries. Tx handlers never read from VersionedCachingStore, they only write to it, which means that the loss of the VersionedCachingStore shouldn't affect performance of validator and sentry nodes that aren't queried often... in fact we probably should've disabled the VersionedCachingStore by default on all those nodes a long time ago to save a bit of CPU & RAM.

# Conflicts: # Makefile

enlight

LGTM

This is a follow up to PR #1532, cleans up a couple of issues left over from that PR: - Eliminate GetSnapshot() from all the stores, only GetSnapshotAt() is used now. - Fix VersionedCachingStore.GetSnapshotAt() to be atomic. Also cleaned up the tests a little bit so they test using real stores instead of buggy mock stores.

MultiWriterAppStore was used to migrate EVM state keys from app.db to evm.db, since all clusters have now been migrated this functionality is no longer necessary, and is in fact incompatible with the changes introduced in PR #1532. This set of changes cleans out most of the obsolete code from that store: - Eliminate writes to the EvmStore from MultiWriterAppStore. - Eliminate Get/Has/Set/Delete from EvmStore. - Eliminate use of AppStore.SaveEVMStateToIAVL config setting.

… of every block (#1532)" This reverts commit a6fba1c.

pathornteng added 4 commits October 9, 2019 13:28

experiment

097dc40

fix unit tests

ea8a66b

only save committed evmroot

de298e0

allow flush interval for evm state to be configured in loom.yaml

5fda6e4

pathornteng commented Oct 9, 2019

View reviewed changes

app.go Outdated Show resolved Hide resolved

pathornteng added 3 commits October 9, 2019 15:27

fix unit test

8c04bfc

fix unit test

9180eb1

save default root to evm.db

4193068

enlight reviewed Oct 9, 2019

View reviewed changes

app.go Outdated Show resolved Hide resolved

address PR comment

a4863cf

enlight reviewed Oct 9, 2019

View reviewed changes

app.go Outdated Show resolved Hide resolved

pathornteng added 16 commits October 9, 2019 18:14

move EVMStateDB from readonlystate to state

2af31ad

refactor

28376ff

clean up

393c94e

update tg test

15a2264

revert change

0ca6a89

only save default root once

7f92b44

remove legacy evm dump commands and clean up

749052c

Merge branch 'master' into in-memory-evmstate

fc265e7

fix unit tests

88aafca

fix a unit test

8853c38

add evmstore to pluginvm

2861d52

merge master

ccc4ffd

simplify test

8198833

Merge branch 'master' into in-memory-evmstate

56cfef7

merge master

8a6be85

simplify commit EVM root logic

1b7540c

enlight reviewed Nov 4, 2019

View reviewed changes

store/evmstore.go Outdated Show resolved Hide resolved

rpc/query_server.go Outdated Show resolved Hide resolved

address PR comment

0b70a21

enlight added 6 commits December 6, 2019 17:28

Remove unused Application.EvmStore field

6ecf12f

Check that EVMState isn't nil in Application.Commit()

e996c14

Check that EVMState isn't nil in Application.CheckTx()

8b8a3cb

enlight reviewed Dec 11, 2019

View reviewed changes

e2e/tests/truffle/test/EvmSnapshot.js Show resolved Hide resolved

e2e/tests/truffle/test/EvmSnapshot.js Show resolved Hide resolved

enlight reviewed Dec 11, 2019

View reviewed changes

store/iavlstore.go Show resolved Hide resolved

address PR comments and make evm state snapshot panic if it calls Com…

b41cb2a

…mit()

Add EvmStore.FlushInterval setting to loom.yml

ad002bb

This setting works similarly to AppStore.IAVLFlushInterval.

enlight changed the title ~~[WIP] Keep Patricia tries in memory and only commit every N blocks~~ Allow EVM state to be flushed to disk on every Nth block instead of every block Dec 17, 2019

enlight changed the title ~~Allow EVM state to be flushed to disk on every Nth block instead of every block~~ Allow EVM state to be flushed to disk every Nth block instead of every block Dec 17, 2019

enlight added 3 commits December 17, 2019 11:45

Switch back to go-ethereum/loomchain branch

b1d871b

Add back a lost comment

412d987

Merge branch 'master' into in-memory-evmstate

c69f3c2

# Conflicts: # Makefile

enlight approved these changes Jan 28, 2020

View reviewed changes

enlight merged commit a6fba1c into master Jan 28, 2020

enlight mentioned this pull request Jan 29, 2020

Eliminate VersionedKVStore.GetSnapshot() #1607

Merged

enlight mentioned this pull request Jan 30, 2020

Cleanup MultiWriterAppStore #1609

Merged

enlight deleted the in-memory-evmstate branch February 27, 2020 07:46

enlight added a commit that referenced this pull request Mar 13, 2020

Revert "Allow EVM state to be flushed to disk every Nth block instead…

53fbe89

… of every block (#1532)" This reverts commit a6fba1c.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow EVM state to be flushed to disk every Nth block instead of every block #1532

Allow EVM state to be flushed to disk every Nth block instead of every block #1532

pathornteng commented Oct 9, 2019 •

edited

Loading

pathornteng commented Nov 4, 2019 •

edited

Loading

pathornteng commented Dec 11, 2019

enlight commented Dec 12, 2019

enlight commented Dec 13, 2019 •

edited

Loading

enlight commented Dec 16, 2019 •

edited

Loading

enlight commented Dec 17, 2019

enlight left a comment

Allow EVM state to be flushed to disk every Nth block instead of every block #1532

Allow EVM state to be flushed to disk every Nth block instead of every block #1532

Conversation

pathornteng commented Oct 9, 2019 • edited Loading

pathornteng commented Nov 4, 2019 • edited Loading

pathornteng commented Dec 11, 2019

enlight commented Dec 12, 2019

enlight commented Dec 13, 2019 • edited Loading

enlight commented Dec 16, 2019 • edited Loading

enlight commented Dec 17, 2019

enlight left a comment

Choose a reason for hiding this comment

pathornteng commented Oct 9, 2019 •

edited

Loading

pathornteng commented Nov 4, 2019 •

edited

Loading

enlight commented Dec 13, 2019 •

edited

Loading

enlight commented Dec 16, 2019 •

edited

Loading