Merge block-stm to develop #840
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rite dependencies between transactions
* basic txn prioritizer implemented using mutex map
* Re-execute parallel tasks when there is a read in coinbase or burn address
* Re-execute parallel tasks when there is a read in coinbase or burn address
* using *sync.RWMutex{} in mutexMap
Co-authored-by: Jerry <jerrycgh@gmail.com>
Added tests for executor and some improvements: 1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure. 2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings. 3. Use sync.Map to reduce the time spent in global mutex. 4. Skip applying intermediate states. 5. Estimate dependency when an execution fails without dependency information. 6. Divide execution task queue into two separate queues. One for relatively certain transactions, and the other for speculative future transactions. 7. Setting dependencies of Txs coming from the same sender before starting parallel execution. 8. Process results in their semantic order (transaction index) instead of the order when they arrive. Replace result channel with a priority queue.
* Create MVHashMap and use it StateDB * Parallel state processor * Move fee burning and tipping out of state transition to reduce read/write dependencies between transactions * Re-execute parallel tasks when there is a read in coinbase or burn address * Block-stm optimization Added tests for executor and two major improvements: 1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure. 2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings. * Remove cache from executor test * added mvhashmap unit tests (with as key) * Shard mvhashmap to reduce the time spent in global mutex * Skip applying intermediate states * Dependency improvement * added test for status * Create MVHashMap and use it StateDB * Parallel state processor * Move fee burning and tipping out of state transition to reduce read/write dependencies between transactions * Re-execute parallel tasks when there is a read in coinbase or burn address * Txn prioritizer implemented using mutex map (#487) * basic txn prioritizer implemented using mutex map * Re-execute parallel tasks when there is a read in coinbase or burn address * Re-execute parallel tasks when there is a read in coinbase or burn address * using *sync.RWMutex{} in mutexMap Co-authored-by: Jerry <jerrycgh@gmail.com> * added getReadMap and getWriteMap (#473) * Block-stm optimization Added tests for executor and some improvements: 1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure. 2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings. 3. Use sync.Map to reduce the time spent in global mutex. 4. Skip applying intermediate states. 5. Estimate dependency when an execution fails without dependency information. 6. Divide execution task queue into two separate queues. One for relatively certain transactions, and the other for speculative future transactions. 7. Setting dependencies of Txs coming from the same sender before starting parallel execution. 8. Process results in their semantic order (transaction index) instead of the order when they arrive. Replace result channel with a priority queue. * Do not write entire objects directly when applying write set in blockstm * fixed a small bug in the Report function (#530) * linters Co-authored-by: Jerry <jerrycgh@gmail.com>
There seems to be an issue when hasher is used concurrently in parallel execution. This change will ensure no hasher is used by multiple executors at the same time.
* added support for dependencies (executor_tests) * added a function to get dependency map * getting all dependencies in the GetDep function * updated GetDep function * changed the type of AllDeps * added a function to get dependency map * updated GetDep function * generate and get dependencies from block producer * optimized getDep function * bug fix regarding txn index and dep structure * fixed gas bug * optimized getDep function * tests updated/added * few updates regarding dependencies * added channel to calculate the dependencies in a separate go routine * minor changes in the executor which uses latest changes of dependecies + removed metadata flag/argument * Use channel when metadata is available * small bug fix * getting reads and writes only when the transaction succeeds * fixed bug in adding dependencies * updated logic for delay/not delay * bug fix (shouldDelayFeeCal) in parallel state processor * lint fix * using EnableMVHashMap flag * fixed worker and stateProcessor and removed SetMVHashMapNil fumction from stateDB * addredded few comments and fixed bug in executor tests * commented executor tests with metadata (panic: test timed out after 5m0s) * added a check to check len(mvReadMapList) > 0 in miner * addressed comments, minor refactoring in dag.go * moved blockContext out of Execute and adding it in execution task * removed Author() from Settle() and added in execution task * not calling block.Header() again and again, using instead * removed EnableMVHashMap flag, and updated applyTransaction function * addressed comments * added unit test to check dependencies in the block header Co-authored-by: Jerry <jerrycgh@gmail.com>
* removed the first 2 element in TxDependency[i] * addressed comments
Merge remote-tracking branch 'upstream/develop' into block-stm
…rupt State processor interrupt
Parallel execution can fail due to bad dependencies. In the case of bad dependencies, the result of serial execution should be used.
Run serial and parallel processor at the same time
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## develop #840 +/- ##
===========================================
+ Coverage 56.60% 56.83% +0.22%
===========================================
Files 611 617 +6
Lines 72430 73600 +1170
===========================================
+ Hits 40999 41828 +829
- Misses 27917 28236 +319
- Partials 3514 3536 +22
☔ View full report in Codecov by Sentry. |
Closed
marcello33
approved these changes
May 9, 2023
* added check for circular and out-of-range dependency problem * addressed comment * addressed comments
If a transaction is reverted, its write records should be excluded from dependency calculation.
pratikspatil024
approved these changes
May 12, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Enables parallel execution of EVM.
Changes
Breaking changes
Please complete this section if any breaking changes have been made, otherwise delete it
Nodes audience
In case this PR includes changes that must be applied only to a subset of nodes, please specify how you handled it (e.g. by adding a flag with a default value...)
Checklist
Cross repository changes
Testing
Manual tests
Please complete this section with the steps you performed if you ran manual tests for this functionality, otherwise delete it
Additional comments
Please post additional comments in this section if you have them, otherwise delete it