[R4R] implement State Verification && Snapshot Commit pipeline #668
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The state verification && Snapshot/MPT commit takes a lot of time. While in most cases, the verification can pass, there is no need to wait for the verification result serially. For snapshot/MPT commit, we only persist the bottom diff layer to disk, there is no need to wait for it to finish. We propose an asynchronous state verification and asynchronous snapshot/MPT commit solution.
Spec
The insert blockchain can be split into three-phase:
The three-phase are executed serially.
Phase2 occupies about 15-20% of the time, and in most case, it should succeed.
Phase3 occupies about 10-15% of the time.
State Verification Pipeline
A pipeline workflow is introduced in this PR:
Once Block Process and Snapshot Commit are done, there is no need to wait for State Verification, the blockchain can process the next block. While for the next block, before it starts State Verification, it has to wait until the State Verification of its parent finish.
Persistent Storage Pipeline
Both MPT and Snapshot need to be persisted into storage, the storage can be Memory DB and Disk DB, this PR will introduce the asynchronous persistence of Disk DB without breaking the processing of the following blocks.
Key Points
How to store the Verification Result
Since Snapshot already has a very good layout for the execution result of the block, we try to reuse it by adding a verifiedCh channel as the verification result.
How to Rollback if Verification Failed
There is an auto rollback mechanism when verification is failed.
How to avoid race condition
Introduce a few locks, but ensure it won't bring too much overhead.
Performance
Following is the 24hours comparison of a Pipeline node(10.211.32.131) and a normal node(10.211.31.49)。
Hardware: 24core, 96G, SSD, storage occupation 2.5T
Execution Time
Before:
After:
The execution time is mainly improved by this PR: #704, while in this PR, we further improve it by avoiding the CPU usage burst.
Validation Time
Before:
After:
Commit Time
Before:
After:
CommitTxs Time
Before:
After:
Further Discussion
Merge Multi Difflayer and Do Verification Once
Improve
AccountsIntermediateRoot
Improve Miner Module