Experience writing an ExEx as a Rust noob (random thoughts)

[unitezen]

I'm currently working on an ExEx to stream various chain data to external consumers and serve queries for data that the standard ETH execution layer APIs do not provide. As a Rust noob, here are some of what I'm trying to implement and the challenges I face (random ramblings, in no particular order):

Re-execute committed/re-orged blocks using custom revm inspector

I need to setup the revm (env/db) to do so. The env involves me constructing all of the required parts with a lot of ::default() for
CfgEnv, HandlerCfg, BlockEnv, CfgEnvWithHandlerCfg, EnvWithHandlerCfg and filling them up with .fill_env_with_header from EvmEnvProvider, then for each transaction I do fill_tx_env_with_recovered with revm's modify_tx_env() then rebuild the revm using build().

For the database I first came up with using things such as state_by_block_number_or_tag on the provider, wrapping it in StateProviderDatabase, then wrap it in CacheDB but upon further looking it seems that i'm being very inefficient -- the Chain from ExExNotification has a BundleStateWithReceipt appears to have all of the data required to reexecute the block (so no need to touch the DB on disk again for account/storage loads, the recent reads should still be in memory due to mmapped nature of mdbx but overhead might be large for all the code path required to read the db?) but I struggle to find the proper way to wrap them up.
Probably need to look at revm's State/BundleState as well for retrieving the account/storage changesets afterwards. Revm is another beast of its own. I saw the BlockExecutor but was not sure how I would integrate a custom inspector to it, probably has to do with the EvmConfig.

Simulate pending transactions with newest payload attributes and custom revm inspector

I'm not sure about the meaning of "pending" in provider methods and I would like to simulate transactions with proper payload attributes (timestamp, prevrandao), etc so I manually subscribe to the provided ExExContext's components.payload_builder() for payload attributes then construct an EnvWithHandlerCfg behind an Arc+RwLock for each new payload attribute which is then cloned and used by multiple spawned worker task that subscribes to the txpool and trace transactions as they come in.

Maintain consistent state on the consumer side

Let say I wrote an external consumer in Python that subscribes to a stream of storage/account changesets and I use it to maintain a derived state of a contract. E.g. track internal storage data that is not emitted as an event and derive it as something meaningful. I need to maintain a journal which tags transformation with the block number/block hash of up to N blocks until the blocks are finalized on the consensus side, which allows me to revert each change when the chain reorgs. If the client loses an update for whatever reason, I would query a custom endpoint to check whether the current last N chain of blocks I've seen matches the canonical block number -> block hash chain and if not revert those changes and backfill if the chain has advanced beyond those points.

Efficient backfills by only reexecuting blocks and transactions that affects an address

Let say I'm 1000 blocks behind, instead of reexecuting all 1000 blocks I would reexecute only certain blocks that have resulted in a state change by walking through the History/Changeset tables (History [account->slot] for contract with small amount of storage keys, changeset [block number -> account] for contract with large amount of keys). May need to try a custom indexing stage for [account -> block number] as table key. Gonna implement an endpoint to get required block range to execute and an endpoint to execute them.

Storing shadow contracts to augment events

Need to look into the mdbx abstraction so I might be able to store these in a seperate mdbx db file and look here before looking at Bytecodes table when executing transactions. Would also be useful to store other arbitrary data.

tl;dr:

• i'm not very good at rust and i'm trying to write as little of it as possible to do the heavy lifting and hose all the data to a more dynamic language where I can experiment and implement new ideas faster.
• there seems to be many ways to accomplish the same one thing, but some are a lot more/less efficient than others. sometimes it's not very obvious if you're not very familiar with the architecture of the node.
• looking at the codebase to try and copy the pattern that the node does things (which is probably a lot more efficient than what I can hack together) is quite painful, due to the understandably abstracted layers of generics and traits. so many types.
• i try to stick with SealedBlockWithSenders and TransactionSignedEcRecovered, with U256, B256 and Address.
• what do I need to wrap this with, and where is it? what traits do I need to bring in scope? what trait bounds do I need for these generics? is a common theme. Ok I want to walk some db tables, what do I need to import. most of it is probably just my skill issue
• simple abstraction/helpers for common tasks for noobs like me would make life a lot easier.
• not complaining though. every contributor did an awesome job, and has been more than helpful in telegram chat. love you guys, am very grateful for the work you guys have done. exex rocks. one day I wanna contribute too.

[mattsse]

this is very helpful tysm.

we want to make all of this easier and also improve examples + docs.