Merge develop (v0.4.0-beta-candidate) to master for v0.4.0 #904
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rite dependencies between transactions
* basic txn prioritizer implemented using mutex map
* Re-execute parallel tasks when there is a read in coinbase or burn address
* Re-execute parallel tasks when there is a read in coinbase or burn address
* using *sync.RWMutex{} in mutexMap
Co-authored-by: Jerry <jerrycgh@gmail.com>
Added tests for executor and some improvements: 1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure. 2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings. 3. Use sync.Map to reduce the time spent in global mutex. 4. Skip applying intermediate states. 5. Estimate dependency when an execution fails without dependency information. 6. Divide execution task queue into two separate queues. One for relatively certain transactions, and the other for speculative future transactions. 7. Setting dependencies of Txs coming from the same sender before starting parallel execution. 8. Process results in their semantic order (transaction index) instead of the order when they arrive. Replace result channel with a priority queue.
* Create MVHashMap and use it StateDB * Parallel state processor * Move fee burning and tipping out of state transition to reduce read/write dependencies between transactions * Re-execute parallel tasks when there is a read in coinbase or burn address * Block-stm optimization Added tests for executor and two major improvements: 1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure. 2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings. * Remove cache from executor test * added mvhashmap unit tests (with as key) * Shard mvhashmap to reduce the time spent in global mutex * Skip applying intermediate states * Dependency improvement * added test for status * Create MVHashMap and use it StateDB * Parallel state processor * Move fee burning and tipping out of state transition to reduce read/write dependencies between transactions * Re-execute parallel tasks when there is a read in coinbase or burn address * Txn prioritizer implemented using mutex map (#487) * basic txn prioritizer implemented using mutex map * Re-execute parallel tasks when there is a read in coinbase or burn address * Re-execute parallel tasks when there is a read in coinbase or burn address * using *sync.RWMutex{} in mutexMap Co-authored-by: Jerry <jerrycgh@gmail.com> * added getReadMap and getWriteMap (#473) * Block-stm optimization Added tests for executor and some improvements: 1. Add a dependency map during execution. This will prevent aborted tasks from being sent for execution immedaitely after failure. 2. Change the key of MVHashMap from string to a byte array. This will reduce time to convert byte slices to strings. 3. Use sync.Map to reduce the time spent in global mutex. 4. Skip applying intermediate states. 5. Estimate dependency when an execution fails without dependency information. 6. Divide execution task queue into two separate queues. One for relatively certain transactions, and the other for speculative future transactions. 7. Setting dependencies of Txs coming from the same sender before starting parallel execution. 8. Process results in their semantic order (transaction index) instead of the order when they arrive. Replace result channel with a priority queue. * Do not write entire objects directly when applying write set in blockstm * fixed a small bug in the Report function (#530) * linters Co-authored-by: Jerry <jerrycgh@gmail.com>
There seems to be an issue when hasher is used concurrently in parallel execution. This change will ensure no hasher is used by multiple executors at the same time.
CI: Optimize devnet setup
* dev: add: pos-944 security ci and readme * dev: add: pos-944 remove linters as this is included already in build ci * dev: chg: pos-947 dependencies upgrade to solve snyk security issues * dev: chg: update security-ci * dev: chg: remove linter to allow replacements for security issues * dev: add: pos-944 verify path when updating metrics from config * dev: add: pos-944 fix linter * dev: add: pos-944 add .snyk policy file / fix snyk code vulnerabilities * dev: fix: pos-944 import common package / gitignore snyk dccache file * dev: fix: pos-944 verify canonical path for crashers * dev: fix: pos-944 linter * dev: add: pos-976 add govuln check * dev: add: pos-976 test upload with permissions * dev: add: pos-976 remove duplicated upload * dev: add: pos-976 report upload * dev: add: pos-976 remove upload * dev: fix: pos-944 fix govuln action * dev: fix: pos-944 move govulncheck to security-ci * dev: fix: pos-944 bump golvun action and golang versions * dev: fix: pos-944 remove persmissions and fix conflicts * dev: chg: restore err msg * dev: chg: remove duplicated function * dev: chg: sort import * dev: chg: fix linter * dev: add: use common VerifyCrasher function to avoid duplications / replace deprecated ioutils.ReadFile * dev: fix: typo
* dev: add: additional elements in PR checklist * dev: chg: erigon task in the checklist
…cs-format DEVOPS-829 format fix for prometheus metrics
…ted (#889) An address's nonce and balance will get updated even if its transaction is reverted. Therefore, we need to keep balance and nonce in write list in order to check the transaction's validity in parallel execution.
* updated Indore HF related changes by adding stateSyncConfirmationDelay * converted StateSyncConfirmationDelay to map[string]uint64 and cleanup * calculate last state ID from incoming state object with eth call (#883) * removed IndoreBlock from configs * fix * remove code duplication and refactor --------- Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
A reverted transaction, X, can incorrectly write to a slot, which might be read by another transaction, Y. If we skip checking states written by X, then Y might end up with reading the wrong value and we won't be able to detect this error.
This reverts commit 6dee9b5.
Currently, we determine if MVHashMap should be enabled by checking the block number with parallelUniverse flag in genesis, which has a value of 0. Some headers' block number is 0, so MVHashMap was enabled, which led to a few errors. Disable this flag explicitly until we figure out a proper hardfork solution.
…e. This issue only is present due to difference in test versus primary
Addressing an issue with packager using any versus mumbai network nam…
Raneet10
approved these changes
Jun 27, 2023
marcello33
approved these changes
Jun 27, 2023
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #904 +/- ##
==========================================
+ Coverage 56.78% 56.80% +0.02%
==========================================
Files 611 617 +6
Lines 71390 73610 +2220
==========================================
+ Hits 40539 41817 +1278
- Misses 27394 28251 +857
- Partials 3457 3542 +85
☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.