feat: `engine_newPayloadV3`: validate, execute & store block #222

fmoletta · 2024-08-05T19:25:55Z

Motivation

Being able to fully validate execute and store blocks received by engine_newPayloadV3

Description

Add block validation, execution & storing to engine_newPayloadV3 endpoint

Fixes:

Genesis.get_block:
- Now uses the INITIAL_BASE_FEE as base_fee_per_gas
- Now computes roots instead of using zero as default
- Now uses default values for cancun fields instead of leaving them as None

(With these fixes the genesis' block hash now matches the parentBlockHash of the next block when running with kurtosis)

beacon_root_contract_call now sets the block's gas_limit to avoid tx validation errors

Misc:

compute_transactions_root is now a standalone function matching the other compute functions
Log more steps of the engine_newPayloadV3 endpoint
Improve api of ExecutionPayloadV3 & PayloadStatus

Other
We can now execute payloads when running with kurtosis 🚀

Disclaimer: We are still getting some execution errors in later blocks that we need to look into (They are all currently passing the block validations)

Closes #51

…re-cancun-time

…idation (#220) **Motivation** Fetch cancun time from DB when validating payload v3 timestamp  **Description** * Store cancun_time in db * Use the stored cancun_time when validating payload timestamp in `eth_newPayloadV3` * Replace update methods for chain data in `Store` with `set_chain_config` Bonus: * Move `NewPayloadV3Request` to its corresponding module and update is parsing to match the rest of the codebase   Closes None, but is part of #51

**Motivation** Having a way to obtain lates/earliest/pending/etc block numbers  **Description** * Add get and update methods for earliest, latest, finalized, safe & pending block number to `Store` & `StoreEngine` * Resolve block numbers from tag in rpc methods   Closes None, but fixes many and enables others

…pts & withdrawals (#225) **Motivation** These roots are currently being calculated using `from_sorted_iter` but without being sorted beforehand. This PR replaces this behavior with inserting directly into the trie to ensure that it is ordered, then computing the root (The same fix that has been previously applied to storage root) **Description** Fixes `compute_transactions_root`, `compute_receipts_root` & `compute_withdrawals_root`  **Notes** After this change, the payloads created by kurtosis local net now pass the block hash validations in `engine_NewPayloadV3`  Closes None, but is needed for #51 Co-authored-by: Federica Moletta <federicamoletta@MacBook-Pro-de-Federica.local>

…date-payload

**Motivation**  We are using `String` and `&str` for URLs instead of the more specific `Url`. **Description**  Replace CLI and occurrences, primarly in `EthClient` to use `Url` instead.  Related: #2673

**Motivation** The CI is broken due to not having space left to run the jobs **Description**   Closes #issue_number

…4646) **Motivation** Our last attempt at making a release failed because protobuf is now a dependency for building (see [this action](https://github.com/lambdaclass/ethrex/actions/runs/17991396735/job/51181982770)). This should fix it. **Description**   Closes #issue_number

**Motivation**  **Description**  Execution with eth_getProof had some caveats regarding the trie. For making it work we need to do 2 things: 1. Pre-inject possible nodes to the trie 2. Make a modification to the MPT in case we don't find a missing node. The `1` was already done in the code but wasn't working properly, the problem was just that the root node wasn't being inserted into the nodes that were used for building the trie that's used for getting potential child nodes. The 2 is a modification that we don't want in our MPT so we didn't carry along [with that](#2952). However, in execution without zkVM we have the possibility of inserting arbitrary nodes with the hashes that we want and RLP content that doesn't necessarily match to that hash, so we can "deceive" the trie into thinking it has some nodes when in fact they are just dummy nodes. How can we use it to our advantage? We insert dummy branch nodes in every single place where a storage node is being referenced but it's missing. This way, if we want to do a trie restructuring we will find the node that otherwise we wouldn't have found, and the good thing is that we don't care about the contents of this node because they won't change, in this particular case the node will be referenced by a one nibble extension node. For more info read https://github.com/kkrt-labs/zk-pig/blob/main/docs/modified-mpt.md#modified-mpt-implementation  Closes #issue_number --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

**Motivation** We want to stop using the keccak-hash crate and implement it with sha3.  **Description** Removed the keccak-hash crate and implemented it with sha3   Closes #4535 --------- Co-authored-by: Ivan Litteri <67517699+ilitteri@users.noreply.github.com>

**Motivation** The mempool has several locks being used at the same time, which is prone to concurrency issues  **Description** Unify the mempool state in a single struct and use only one lock for it, since we write everything at the same time, this shouldn't affect performance, since we can still do concurrent readings   Closes #4592 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…4656) **Motivation** We currently return a "NonceTooLow" error when we fail to replace a mempool transaction (aka we receive a transaction with the same sender and nonce as an existing one but without a higher fee value). This doesn't make any sense and can be quite confusing. This PR adds a MempoolError enum variant for this specific case.  **Description** * Use a specific error variant when we fail to replace a mempool transaction   Closes None

**Motivation** The removal of elements from the mempool when it is full is not efficient.  **Description** Now the array that represent the order of insertion to the mempool is a vecdeque, which is not changed when an arbitrary element is removed from the mempool. Only when the mempool is full, we remove from the vecdeque until we find an element present in the mempool. Also if the vecdeque is greater than 150% of the maximum mempool size, we prune it, keeping only the txs that are actually in the mempool, preserving the order.   Closes #4605 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

**Motivation**  CLI reference is hard to find across de docs. **Description**  Add links to CLI reference at startup and configuration docs  --------- Co-authored-by: Martin Paulucci <martin.c.paulucci@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

**Motivation**  We don't set the `extra_data` field in the block header. **Description**  Use a CLI provided `extra_data` string or a default one.  Closes #4662

**Motivation**  Currently when someone wants to upgrade the ethrex version of an L2, they have to perform an UUPS upgrade to the contract in order to upgrade the verification keys. **Description**  Add a function to upgrade SP1 and RISC0 VKs without the need of a full upgrade to the contract

**Motivation**  **Description**  Block number being displayed on screen was wrong. We now do store the actual block number to execute and this is used for the post-state proofs, whereas the `block number - 1` is used for the pre-state proofs.  Closes #issue_number

**Motivation** We want to make the mempool max size a parameter  **Description** Adds the mempool max size as a parameter to ethrex command   Closes #4600

…s in `TxBroadcaster` (#4675) **Motivation** We were seeing spikes of 3~10Gb on post-sync execution, most of it was due to `resize_rehash` during the `add_txs` of the `TxBroadcaster`. **Description** This PR replace the current hashmap of `(tx_hash, peer_id) -> Instant` for a less duplicated `tx_hash -> (peer_bitmap, last_seen)` + another `peer_id -> peer_index`. This removes orders of magnitud of entries, going from 27M to less than 100k. Now all the memory taken from the TxBroadcaster disappeared from our memory flamegraphs. WIP: ~I'm right now analyzing the execution of the client throughout the weekend, we still appear to have a memory issue.~ This was related to a change in rocksdb that enabled transaction. It didn't affected this fix.  Closes #issue_number

**Motivation**  **Description**  Closes #issue_number --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ivan Litteri <67517699+ilitteri@users.noreply.github.com>

**Motivation**  **Description**   Closes #issue_number

**Motivation** Remove rem as a workspace dependency  **Description** * Remove `revm` feature from all workspace crates * Remove revm backend from `ethrex-vm` crate * Remove `tooling/ef_tests/state` and `crates/vm/levm/bench/revm_comparison` from workspace (They still retain revm as a dependency)   Closes #issue_number