execute eth_estimateGas and eth_createAccessList on latest block

[MartinquaXD]

Description

For a while we had mysteriously failing transactions. What I mean is that we execute eth_estimateGas to see if a transaction would still work. If it fails we log all the data needed to re-simulate it on tenderly. But when you then actually re-simulate on tenderly the transaction would work.

The solution unravels with this issue.
The spec for eth_estimateGas mentions 2 arguments: the tx and an identifier for a block. The block is supposed to be optional but geth had a bug for a very long time. When you don't pass a block argument it should be sent over the wire as null. But geth tried to deserialize null which then caused these calls to fail.

As a workaround the web3 crate stopped serializing the block tag if it was None. Eventually geth fixed this issue and implemented it such that a missing block tag would default to pending.
Also a rule of thumb in node development is "if in doubt do whatever geth does" so reth also implemented the default to pending.

Defaulting to pending is what can make these reverts unpredictable. When you simulate the tx on pending the node takes the state of the latest block PLUS transactions that it currently has in its mempool. So whenever we log a reverting tx that works on tenderly it only failed in our node because some tx in the nodes mempool interfered with the solution.
Interestingly there also a lot of cases where you can even simulate the tx on the next block. In those cases simulating on pending caused the driver completely erroneously since the problematic tx didn't even make it into the next block.

example

log

2025-01-06T06:33:25.748Z  INFO request{id="4114843"}:/solve{solver=extzeroex-solve auction_id=9990122}: driver::infra::observe: discarded solution: settlement encoding id=Id { id: 17739308, merged_solutions: [22] } err=Simulation(Revert(RevertError { err: Blockchain(AccessList(String("execution reverted"))), tx: Tx { from: Address(0x28b1bd44996105b5c14c4de41093226ff78a4eb1), to: Address(0x9008d19f58aabd9ed0d60971565aa8510560ab41), value: Ether(0), input: 0x13d79a0b0000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000012000000000000000000000000000000000000000000000000000000000000001c000000000000000000000000000000000000000000000000000000000000003e00000000000000000000000000000000000000000000000000000000000000004000000000000000000000000839d4a05d6d23a22365bd90c2114c199d53e674c000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee000000000000000000000000839d4a05d6d23a22365bd90c2114c199d53e674c000000000000000000000000eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee0000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000017c836d081141b7000000000000000000000000000000000000000000000034cafbfac2c90f727a000000000000000000000000000000000000000000000000017c836d081141b700000000000000000000000000000000000000000000003635c9adc5dea0000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000030000000000000000000000006a7fe93d1e194d31b8076c255779c4b12a45192c00000000000000000000000000000000000000000000003635c9adc5dea00000000000000000000000000000000000000000000000000000010df3f85654f86700000000000000000000000000000000000000000000000000000000677b7e7d5b203ebd4ff6f6667daa775bd3de4410b0b834f059fccecd85660cb72bf031e30000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003635c9adc5dea00000000000000000000000000000000000000000000000000000000000000000016000000000000000000000000000000000000000000000000000000000000000410fd3b1f82fc78a190ccf38bbfa08652ca518ef025802fe671c0b7e1b8cc1c8e762470c92b313dd597fde1022168575929770076f30cc6581377d174f76ce675e1c0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000046000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000001400000000000000000000000000000000000000000000000000000000000000300000000000000000000000000839d4a05d6d23a22365bd90c2114c199d53e674c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000044095ea7b3000000000000000000000000def1c0ded9bec7f1a1670819833240f027b25eff00000000000000000000000000000000000000000000003635c9adc5dea0000000000000000000000000000000000000000000000000000000000000000000000000000000000000def1c0ded9bec7f1a1670819833240f027b25eff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000128d9627aa4000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000003635c9adc5dea000000000000000000000000000000000000000000000000000000186565d4040733b00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000839d4a05d6d23a22365bd90c2114c199d53e674c000000000000000000000000c02aaa39b223fe8d0a0e5c4f27ead9083c756cc2869584cd0000000000000000000000009008d19f58aabd9ed0d60971565aa8510560ab410000000000000000000000000000000000000000b507dba868e8950578d34d3f000000000000000000000000000000000000000000000000000000000000000000000000c02aaa39b223fe8d0a0e5c4f27ead9083c756cc20000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000242e1a7d4d000000000000000000000000000000000000000000000000017c836d081141b70000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000986fea, access_list: AccessList({}) }, block: BlockNo(21563660) }))

Results in this simulation that even works 2 blocks past the block we logged.

Changes

Always estimate gas and create access lists on the latest block instead of pending.
This should be the better default for 2 reasons:

1. it makes our simulation reverts consistent with what we see on tenderly
2. most blocks get built by sophisticated algorithms including and arranging blocks in a way that maximizes MEV. That means whatever transactions ordering the node applies in the pending likely isn't accurate anyway.

[marcovc]

Thanks Martin for PR and awesome detective work!

[mstrug]

Maybe instead of unwrap() we can return an error (function returns Result)?

[MartinquaXD]

this would require a new error type and given that this unwrap never caused issues (all possible values are serializable) I prefer not to touch it.

[sunce86]

Did you test this somehow?

Btw, I thought it's known that simulating on block X on client means "on top of block X", while on tenderly it means "on top of X-1 block", so you actually always have to add 1 block to tenderly to achieve the same. So, I guess with this change we should always be 1:1 if we add that +1 to tenderly simulations.

[MartinquaXD]

Did you test this somehow?

Not so much. The e2e tests are still passing (including the forked ones) so all the nodes we care about support passing the latest block tag with the request.
Other than that it's not straight forward to test this. If you think it's very important I could try to build a test case that shows a revert using pending but not with latest.

Btw, I thought it's known that simulating on block X on client means "on top of block X", while on tenderly it means "on top of X-1 block", so you actually always have to add 1 block to tenderly to achieve the same.

Overall the system we have is a bit racy. If the node becomes aware of a new block before out blockstream we could show incorrect blocks regardless. Sometimes the using the same block on tenderly as in the logs results in a revert and sometimes it doesn't. And as shown in the example sometimes the tx works even multiple blocks ahead.

So, I guess with this change we should always be 1:1 if we add that +1 to tenderly simulations.

Modulo race conditions with the block stream. I would love it if nodes simply returned the block they used when you do anything on the latest block. :/

[sunce86]

The e2e tests are still passing (including the forked ones) so all the nodes we care about support passing the latest block tag with the request.

This would be enough for me.

[sunce86]

Overall the system we have is a bit racy. If the node becomes aware of a new block before out blockstream we could show incorrect blocks regardless. Sometimes the using the same block on tenderly as in the logs results in a revert and sometimes it doesn't. And as shown in the example sometimes the tx works even multiple blocks ahead.

Ah true, forgot about it. We would have to implement ethereum/execution-apis#484 when ti lands.