feat: compute receipts root #111

ricomateo · 2024-07-02T18:18:42Z

Closes #106

**Motivation** Replace evm code in eftests with an implementation that uses ethrex types **Description** * Implement `execute_tx` using revm * Move `evm` crate into `core` crate * Move `Transaction` into its own module * Implement multiple getters for `Transaction` fields * Implement address recovery from signed transactions * Implement conversions between ef tests types and ethrex types * Add one more ef test  Closes #21

the value must be rlp(receipt) instead of tx_type || rlp(receipt) (if tx_type != 0)

the previous way in which the receipts trie values were computed was the right way

crates/core/src/types/block.rs

MegaRedHand

LGTM

**Motivation**  **Description**   Closes #issue_number

**Motivation** Obtain world state from account info stored in db **Description** * Implement storage root computation * Implement word state  Closes #44

**Motivation** Add support for EIP2930 transaction **Description** Implements EIP2930Transaction + required methods & traits   Closes None, but will enable #135 to also close #24

This PR is based on #138 (as they modify the same code), please merge it first **Motivation** Add support for EIP4844 transactions  **Description** Add `EIP4844` transactions + needed trait and methods and integrate it into existing transaction-related code   Closes #26 (evm already performs validations before executing so adding the transaction itself will sufice)

**Motivation** Add error handling to `execute_tx` and remove unwrap  **Description** Add `EvmError` Map revm's `EVMError` to `EvmError` Add error handling to `execute_tx`   Closes None

**Motivation**  **Description**   Closes #125 --------- Co-authored-by: Tomás Grüner <47506558+MegaRedHand@users.noreply.github.com>

**Motivation** Being able to store and fetch blocks from the db via `Store` api **Description** * Replace `Transaction::encode` with `Transaction::encode_with_type` logic (Reasoning: We cannot decode the transactions without knowing their type with the current behaviour, making it impossible to implement `RLPDecode` for `Transaction`) * Implement `RLPDecode` for `Transaction` (Using the logic that was previoulsy used in `EncodedTransaction::decode` (payload module)) * Implement `RLPDecode` for `Withdrawal`, `BlockHeader` and `BlockBody` * Add the folloewng methods to `Store` and `StoreEngine`: `add_block_header`, `add_block_body`, `get_block_header`, `get_block_body`, and implement them for `InMemory` and `Libmdbx` engine types   Closes None, but is needed for #145

**Motivation** Being able to serialize blocks matching the output required by the rpc spec  **Description** *Add struct `BlockSerializable` which can be serialized to match the specifications of the rpc. It can contain either the full transactions or just their hashes. *Implement serialization for `BlockSerializable`, `BlockHeader`, `BlockBody`, `Withdrawal` and `Transaction`. Other changes: *Replace `Bloom` with `ethereum_types::Bloom` *Rename some fields in `Eip1559Transaction` to match other transaction types' fields   Closes None, but is needed in order to implement #31

**Motivation**  Ref #1741 **Description**  Replaces specId with Fork - First change is in file `crates/vm/levm/src/environment.rs` - - Changes in other relevant files to accept the changes and fix cascading errors  Closes [#1741](#1741) --------- Co-authored-by: Tomas Fabrizio Orsi <tomas.orsi@lambdaclass.com> Co-authored-by: Federico Borello <156438142+fborello-lambda@users.noreply.github.com> Co-authored-by: fborello-lambda <federicoborello@lambdaclass.com> Co-authored-by: Leandro Ferrigno <leanrafa@gmail.com>

**Motivation** This PR introduces the following upgrades for snap-sync: - Use DB-persisted checkpoints so we can persist the sync progress throughout restarts & cycles - Stop ForckChoices & NewPayloads being applied while syncing - Improved handling of stale pivot during sub-processes - Improved handling of pending requests when aborting due to stale pivot - Fetching of large storage tries (that don't fit in a single range request) - Safer (but a bit slower) healing that can be restarted - Faster storage fetching (multiple parallel fetches) And also simplifies it by removing the following logic: - No longer downloads bodies and receipts for blocks before the pivot during snap sync (WARNING: this goes against the spec but shouldn't be a problem for the time being) - Removes restart from latest block when latest - 64 becomes stale. (By this point it is more effective to wait for the next fork choice update) - Periodically shows state sync progress  **Description** - Stores the last downloaded block's hash in the DB during snap sync to serve as a checkpoint if the sync is aborted halfway (common case when syncing from genesis). This checkpoint is cleared upon succesful snap sync. - No longer fetches receipts or block bodies past the pivot block during snap sync - Add method `sync_status` which returns an enum with the current sync status (either Inactive, Active or Pending) and uses it in the ForkChoiceUpdate & NewPayload engine rpc endpoints so that we don't apply their logic during an active or pending sync. - Fetcher process now identify stale pivots and remain passive until they receive the end signal - Fetcher processes now return their current queue upon return so that it can be persisted into the next cycle - Stores the latest state root during state sync and healing as a checkpoint - Stores the last fetched key during state sync as a checkpoint - Healing no longer stores the nodes received via p2p, it instead inserts the leaf values and rebuilds it to avoid trie corruption between restarts. - The current progress percentage and estimated time to finish is periodically reported during state sync - Disables the following Paris & Cancun engine hive tests that previously yielded false positives due to new payloads being accepted on top of a syncing chain: * Invalid NewPayload (family) * Re-Org Back to Canonical Chain From Syncing Chain * Unknown HeadBlockHash * In-Order Consecutive Payload Execution (Flaky) * Valid NewPayload->ForkchoiceUpdated on Syncing Client * Invalid Missing Ancestor ReOrg * * Payload Build after New Invalid Payload  (only Cancun) - And also disables the following tests that fail with the flag Syncing=true for the same reason : * Bad Hash on NewPayload * ParentHash equals BlockHash on NewPayload (only for Paris) * Invalid PayloadAttributes (family) Misc: - Replaces some noisy unwraps in networking module with errors - Applies annotated hacky fixes for problems reported in #1684 #1685 & #1686   Closes None   Closes #issue_number

**Motivation** Since the recently #1803 PR we now have new benchmarks. We need to modify the CI to make the comparison. I couldn't do that in the previous PR because since the new benchmarks were not in main yet, the CI would always fail. **Description** I will fix this in the next benchmark PR so we don't have to create 2 PRs for each new benchmark.

**Motivation** There is no reason to attempt to resume a state sync if the previous cycle already moved on to healing, it could lead to corrupted state as the trie is currently being healed and in an inconsistent state  **Description** * Skip to healing if a previous cycle already completed state sync   Closes #issue_number

**Motivation** Adding more benchmarks to have an idea of how we fare vs other implementations and find possible improvements. **Description** We add a new benchmark: - `BubbleSort`: Creates an array of n random elements, sorts it via bubble-sort and then checks the array is sorted.

…1828) **Motivation** Accounts are downloaded at a much faster pace compared to storages, causing bottlenecks in storage download. A previous PR sped up storage download by parallelizing requests but we still had many requests piling up in set ups with multiple peers. This PR aims to alleviate this issue by reading multiple incoming requests at once and avoid piling requests in the storage fetcher's receiver  **Description** * Read multiple messages at once from the storage fetcher's receiver channel during snap sync   Closes #issue_number

**Motivation** sync code should not be aware of how peers are handled and only care about what they ask the peer and what it returns to them. This PR aims to abstract peer handling logic into a `PeerHandler` so that sync code no longer has to care about finding a suitable peer and managing retries for failing requests. The `PeerHandler` api will provide request methods that return a suitable peer response unless there was no suitable peer or none of them returned a suitable response. This will also allow us to refactor peer code without affecting sync code & add other behaviors such as punishing peers that return invalid responses in the future  **Description** * Add `PeerHandler` abstraction (which will take over the request methods of `PeerChannels`) * Peer request methods now also find a suitable peer for the request and retry it if it fails * Removed peer selection & request retry logic from sync code   Closes #issue_number

**Motivation** This testnet is much newer than holesky, so It should allow us to test snap sync much faster  **Description**   Closes #issue_number

… on beacon call (#1864) **Motivation**  In this PR we change the check for doing the beacon root call to make it work for future forks begging in Cancun **Description**  Change the equal assertion of the fork to a greater or equal assertion

**Motivation**  This pull request introduces updates to the `blockchain` crate, primarily focusing on handling transactions and withdrawals when the `levm` feature is enabled. **Description**  ##### Dependency Management: * Added `ethrex-levm` as an optional dependency in `Cargo.toml` to support `levm` feature. * Updated the `levm` feature to include `ethrex-levm` in `Cargo.toml`. ##### Function Modifications: * Modified various functions in `payload.rs` to support `block_cache` when the `levm` feature is enabled: * `build_payload` now initializes and passes `block_cache` through the transaction processing pipeline. * `apply_withdrawals` returns a `block_cache` and updates state based on the `levm` feature. * `fill_transactions` and its helper functions (`apply_transaction`, `apply_blob_transaction`, `apply_plain_transaction`) now accept and update `block_cache`. * `finalize_payload` updates the state root and other headers using `block_cache` when the `levm` feature is enabled. ##### Code Organization: * Added conditional imports for `levm` feature in `payload.rs` to ensure compatibility and proper functionality.  Closes #1777 --------- Co-authored-by: Tomas Fabrizio Orsi <tomas.orsi@lambdaclass.com> Co-authored-by: Ivan Litteri <67517699+ilitteri@users.noreply.github.com>

…ng a known network (#1866) **Motivation** When we use a known network name as a `--network` argument we should not ignore the boot nodes passed though the command line but instead extend them with the preset bootnodes.  **Description** * Extends current bootnodes with preset boot nodes instead of replacing them when running a preset network * Documents preset networks feature in cli documentation   Closes #issue_number

**Motivation**  This PR fix the last test of the regression when using LEVM with the feature flag. **Description**  ### Enhancements to nonce handling: * Added a `tx_nonce` field to the `Environment` struct . * Introduced a new `NonceMismatch` variant in the `TxValidationError` enum to handle nonce mismatch errors. * Added check for mismatch nonce value in the `prepare_execution`.  Closes #1865 --------- Co-authored-by: Tomas Fabrizio Orsi <tomas.orsi@lambdaclass.com>

**Motivation** Reverts the "slow but safe" approach to healing taken by #1763 and instead adds the intermediate nodes received from peers to the trie.  **Description** * Write nodes fetched during healing to the state instead of writing only leaf values * Track paths left in queue when a healing cycle ends due to pivot staleness * Condense snap checkpoint clearing to one single method * (Misc) Move some noisy p2p info tracing to debug * (Fix) When executing full blocks after the pivot block during snap sync execute the block first before setting it as canonical   **Comments** A not so pretty solution was implemented on the storage_healer (the startup flag) to avoid blocking storage healing at the start. This can and will be improved but doing so in this PR is not worth it when follow up PRs will already touch up on this logic when parallelizing storage node fetching (similar to storage ranges) **Testing** This proved to work fine in `Mekong` testnet when introducing forced sleeps and delays to force healing. But it is still too slow to work on Holesky testnet, this should improve once we add more parallelization to state sync & healing Closes #issue_number

…1872) **Motivation** Spawn multiple concurrent fetch processes for state & storage nodes (similar to storage ranges) in order to speed up healing phase.  **Description** * Add `MAX_PARALLEL_FETCHES` constant to limit and customize how many concurrent fetch processes can be spawned at a time when fetching storage ranges, state nodes and storage nodes * Spawn multiple sub-processes at a time to fetch state & storage nodes * Use `recv_many` on the storage healer to avoid piling up storage heal requests   Closes #issue_number

**Motivation**  This pull request introduces changes to the Ethereum Foundation (EF) test framework, specifically enhancing the handling of forks within the test reports. The changes include modifications to the deserialization process, test report structure, and test runner logic to support multiple forks. **Description**   * `cmd/ef_tests/levm/report.rs`: Refactored `EFTestReport` to include `fork_results` instead of a single `fork` field. Introduced `EFTestReportForkResult` to encapsulate fork-specific results. Updated methods to handle the new structure and ensure correct reporting of test results per fork. * `cmd/ef_tests/levm/runner/levm_runner.rs`: Updated the `run_ef_test` and `run_ef_test_tx` functions to iterate over forks and handle fork-specific test execution. Adjusted `prepare_vm_for_tx` and `ensure_post_state` functions to accept a `fork` parameter. * `cmd/ef_tests/levm/runner/mod.rs`: Handled cases where test hashes might be missing to prevent panics. * `cmd/ef_tests/levm/runner/revm_runner.rs`: Updated imports to include `EFTestReportForkResult`. Closes #1701

**Motivation**  This pull request includes updates to the `crates/blockchain/payload.rs` file, grouping the withdrawal for the different evm implementations. Additionally, there are minor changes in `crates/vm/vm.rs` related to state transitions. **Description**  * Added a new function `make_beacon_root_call` to handle beacon root contract calls within the `build_payload` function. * Removed withdrawal processing code from the `finalize_payload` function, as it is now handled within the `apply_withdrawals` function. * Updated the `apply_plain_transaction` function to use `get_state_transitions_levm` for account updates, ensuring consistency with the new beacon root call integration.  --------- Co-authored-by: Francisco Krause Arnim <56402156+fkrause98@users.noreply.github.com> Co-authored-by: Federico Borello <156438142+fborello-lambda@users.noreply.github.com>

**Motivation**  In this PR, part of the gas cost that were in the pre berlin forks are added. **Description** * Added constants for pre-Berlin fork gas costs: `SLOAD_COST_PRE_BERLIN`, `SSTORE_PRE_BERLIN_NON_ZERO`, `SSTORE_PRE_BERLIN`, and `ADDRESS_COST_PRE_BERLIN` in `crates/vm/levm/src/gas_cost.rs`. * Updated `sload`, `sstore`, `address_access_cost`, `balance`, `extcodesize`, `extcodecopy`, `extcodehash`, `call`, `callcode`, `delegatecall`, and `staticcall` functions to include the `Fork` parameter and handle pre-Berlin gas costs. * Updated `balance`, `extcodesize`, `extcodecopy`, `extcodehash`, `sload`, `sstore`, `call`, `callcode`, `delegatecall`, and `staticcall` function calls to pass the `Fork` parameter in `crates/vm/levm/src/opcode_handlers/environment.rs`, `stack_memory_storage_flow.rs`, and `system.rs`.

**Motivation**  In this PR we solved the wrong ef tests summaries due to incorrect fork parsing **Description**  We change the `"ConstantinopleFix"` to be parsed as the `Petersburg` fork, instead of the `Constantinople`

**Motivation**  In this PR we add support for the `eip3651_warm_coinbase` test suite. **Description**  Change the cold/warm cost for `balance` and `extcodesize` of the [eip-2929](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-2929.md#storage-read-changes) to be on forks after Berlin and not after Cancun.  Closes #1916

**Motivation**  In this PR we add support for the `stArgsZeroOneBalance` test suite. **Description**  * We implement the change in the London fork of the quotient in the gas refund. * We remove the cost of cold addresses in the `selfdestruct` opcode for pre Berlin forks. * We add the gas refund in the `selfdestruct` for pre London forks.  Closes #1907

**Motivation**  In this PR we make the stBug test suite pass. **Description**  * We introduce a gas refund of 24000 from the `sstore` opcode to pre Istanbul forks * We take correct gas cost in the `extCodeHash` opcode of 400 to pre Istanbul forks  Closes #1920

**Motivation**  In this PR we add support for the `yul` and `stSStoreTest` test suite. **Description**  * We change and incorrect check of the fork Berlin. Previously it was inclusive when it should be exclusive. * We make a correct gas charge in the `get_intrinsic_gas` function. In a Create type transaction, 2 gas per 32-byte words must be charged in order to follow [EIP-3860](https://eips.ethereum.org/EIPS/eip-3860) but only in Shanghai or later forks  Closes #1938

**Motivation**  In this PR we add increase the test passed in the `stCreate2` test suite **Description**  * We add the correct values for the `ecadd`, `ecmul` and `ecpairing` precompiles to pre Istanbul forks. This is stated [here](https://eips.ethereum.org/EIPS/eip-1108) * We invalidate transactions with the opcode `create2` on forks previous to Constantinople, see [here](https://eips.ethereum.org/EIPS/eip-1014)  Closes #1941

…1899) **Motivation**  In this PR we increase the test passed in the `stDelegatecallTestHomestead` test suite. **Description**  * We add the pre tangerine gas costs for `sload`, `selfdestruct` and `extcodecopy`. * We now don't apply the "all but one 64th" introduced in the [eip-150](https://eips.ethereum.org/EIPS/eip-150). * We invalidate the transaction if we use a `delegatecall` opcode introduced in [eip-7](https://eips.ethereum.org/EIPS/eip-7) for previous forks to Homestead. * We add on previous fork to Spurious Dragon to set the nonce to 0 instead of 1. This is due to [eip-161](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-161.md).  Closes #1886

**Motivation**  In this PR we add the check to not refund gas on accounts that belongs to the selfdestruct set. **Description**  We add a prior check on the gas refund to the account that receives the gas refund to not be in the `selfdestruct_set`

**Motivation** Updates snap sync documentation to reflect changes up to #1936, please merge it first  **Description** * Update snap sync documentation with latest changes   Closes #issue_number

…orage healing (#1936) **Motivation**  Currently, snap sync is working fine on small networks and local test environments but it is still too slow to catch up to a larger network such as holesky and sepolia. This PR aims to speed up the state sync phase by dividing the account state and spawning multiple processes to fetch multiple sections of the state at the same time. While working on this PR I also noticed that the trie rebuild time is quite significant so this PR also includes rebuilding the state & storage tries in a permanent background process while the state sync is active. The PR also includes starting storage healing earlier for those storages that are left in the fetcher's queue when the pivot becomes stale. This change should alleviate the workload of the storage healer in the post-state sync healing phase. **Description** ***State Sync Paralellization*** - Create Snapshot DB tables to store fetched accounts and storages during state sync - Write accounts to the snapshot instead of the tries during state sync - Divide the account state into N segments and fetch all segments at the same time (The number of segments is currently set to 2 but can be adjusted by changing a constant in the storage module) ***Trie Rebuild in Background*** - Add a background state trie rebuild process that will continuously try to rebuild the state trie by reading the latest accounts from the state snapshot, finishing when state sync is finished and there are no more accounts in the state snapshot - Add a background storage trie rebuild process that will listen for incoming messages from the storage fetcher, which will advertise storages that have been fully fetched, and rebuild the full storage tries. It will end when state sync has finished and no more storages are left in queue. - Pass a `CancellationToken` to the background rebuild processes so that they will not be stuck when taking down the node via Ctr+C ***Early Healing*** - The storage healer will be spawned before state sync as opposed to after. - A sender to the storage healer's channel will be passed to the storage fetcher so that it can send unfetched storages after a stale pivot is detected.  Closes #issue_number

…mpiledContracts2` (#1969) **Motivation**  In this PR we check if the fork is enabled to execute precompiles and implement the gas cost for the `call` opcode in [EIP-161](https://eips.ethereum.org/EIPS/eip-161) **Description**  * In the Fork Byzantium the precompiles `modexp`, `ecadd`, `ecmul` and `ecpairing` are introduced. In the previous forks this precompiles must be disable. * For the `call` opcode in the previous forks to Spurious Dragon, the empty account cost only checks for the account to be empty. On Spurious Dragon and after, the value to send must be non zero to be charged with the gas cost.  The current state is: * `stPreCompiledContracts: 298/299 (99.67%)` * `stPreCompiledContracts2: 648/762 (85.04%)` --------- Co-authored-by: Tomas Fabrizio Orsi <tomas.orsi@lambdaclass.com> Co-authored-by: Federico Borello <156438142+fborello-lambda@users.noreply.github.com>

…#1961) **Motivation** The code for sync has grown significantly since its start, and having it all under the same module can make it very hard to follow. This PR aims to solve this by separating each different job into different modules and adding module-level documentation for each, explaining how each part of the sync works and integrates to the system as a whole  **Description** * Separate sync code into different modules, one for each separate functionality * Add doc comments for each individual module explaining its purpose and mechanics   Notes for merging: This PR will likely conflict with #1936 but it has the same exact code, only moving it across modules, so it should be safe to solve all conflicts by keeping the changes from this PR alone. Alternatively, you can also merge this PR into #1936 as it has no conflicts and then merge #1936 --------- Co-authored-by: Martin Paulucci <martin.c.paulucci@gmail.com>

**Motivation** This PR introduces a `CommonBridgeL2` system contract (that comes with the L2 genesis) at address `0x000000000000000000000000000000000000FFFF` that handles withdrawals. To issue a withdraw, users simply send a transaction to the `withdraw` function with the address of the account that will claim the funds on L1. This was done because the previous approach to withdrawals was extremely hacky and error prone, issuing a transaction to the L1 address that would claim, that was then internally changed to the zero address before vm execution. This approach is a lot simpler and does not require any hackyness of that kind. **Description**   Closes #issue_number

MegaRedHand and others added 10 commits July 1, 2024 13:03

feat: compute transactions root for a block

a685ff2

fix: was encoding LegacyTransaction incorrectly

611bc86

fix: RLP encode index

49f0374

Merge branch 'main' into compute-transactions-root

7080893

Merge branch 'main' into compute-transactions-root

5f9e6eb

refactor: add Transaction::encode_with_type

a81897f

refactor: add TxType enum for encoding tx type

b966cf4

refactor: add TxKind struct

2c59b6d

Merge branch 'main' into compute-transactions-root

830ecac

add receipts root computation

67f1db0

ricomateo added 4 commits July 2, 2024 17:07

fix: the receipt trie value was being computed wrong

3f2dac6

the value must be rlp(receipt) instead of tx_type || rlp(receipt) (if tx_type != 0)

fix: compute receipts trie value correctly

8e23699

the previous way in which the receipts trie values were computed was the right way

remove unnecessary constructor

3b32d29

merge main

ff376e1

ricomateo marked this pull request as ready for review July 2, 2024 20:59

ricomateo requested a review from a team as a code owner July 2, 2024 20:59

MegaRedHand reviewed Jul 2, 2024

View reviewed changes

crates/core/src/types/block.rs Outdated Show resolved Hide resolved

fix: move compute_receipts_root out of BlockBody impl

f02abf1

MegaRedHand approved these changes Jul 3, 2024

View reviewed changes

MegaRedHand merged commit b3a16bf into main Jul 3, 2024
3 checks passed

MegaRedHand deleted the compute-receipts-root branch July 3, 2024 13:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: compute receipts root #111

feat: compute receipts root #111

ricomateo commented Jul 2, 2024

MegaRedHand left a comment

feat: compute receipts root #111

feat: compute receipts root #111

Conversation

ricomateo commented Jul 2, 2024

MegaRedHand left a comment

Choose a reason for hiding this comment