EngineError when requesting block at slot 43200 on Zhejiang testnet

[peterbitfly]

System information

Erigon version: erigon version 2.39.0-dev-f01e3d17

OS & Version: Linux

Commit hash:

Erigon Command (with flags/config):

/usr/bin/erigon
  --datadir=/home/ubuntu/eldata
  --networkid=1337803
  --http
  --http.api eth,erigon,engine,debug,trace,txpool
  --http.port 8545
  --http.addr 0.0.0.0
  --http.vhosts *
  --http.corsdomain *
  --ws
  --private.api.addr=127.0.0.1:9999
  --externalcl
  --rpc.batch.limit 10000000
  --rpc.returndata.limit 1000000000
  --authrpc.port=8551
  --authrpc.addr=127.0.0.1
  --authrpc.jwtsecret=/home/ubuntu/jwtsecret
  --batchSize=32m
 
--bootnodes=enode://691c66d0ce351633b2ef8b4e4ef7db9966915ca0937415bd2b408df22923f274873b4d4438929e029a13a680140223dcf701cabe22df7d8870044321022dfefa@64.225.78.1:30303,enode://89347b9461727ee1849256d78e84d5c86cc3b4c6c5347650093982b726d71f3d08027e280b399b7b6604ceeda863283dcfe1a01e93728b4883114e9f8c7cc8ef@146.190.238.212:30303,enode://c2892072efe247f21ed7ebea6637ade38512a0ae7c5cffa1bf0786d5e3be1e7f40ff71252a21b36aa9de54e49edbcfc6962a98032adadfa29c8524262e484ad3@165.232.84.160:30303,enode://71e862580d3177a99e9837bd9e9c13c83bde63d3dba1d5cea18e89eb2a17786bbd47a8e7ae690e4d29763b55c205af13965efcaf6105d58e118a5a8ed2b0f6d0@68.183.13.170:30303,enode://2f6cf7f774e4507e7c1b70815f9c0ccd6515ee1170c991ce3137002c6ba9c671af38920f5b8ab8a215b62b3b50388030548f1d826cb6c2b30c0f59472804a045@161.35.147.98:30303

Concensus Layer:

Concensus Layer Command (with flags/config):

/usr/bin/lighthouse
  --datadir /home/ubuntu/beacon
  --testnet-dir=/home/ubuntu/custom_config_data
  bn
  --prune-payloads false
  --disable-enr-auto-update
  --port=9000
  --discovery-port=9000
  --eth1
  --boot-nodes=enr:-Iq4QMCTfIMXnow27baRUb35Q8iiFHSIDBJh6hQM5Axohhf4b6Kr_cOCu0htQ5WvVqKvFgY28893DHAg8gnBAXsAVqmGAX53x8JggmlkgnY0gmlwhLKAlv6Jc2VjcDI1NmsxoQK6S-Cii_KmfFdUJL2TANL3ksaKUnNXvTCv1tLwXs0QgIN1ZHCCIyk,enr:-Ly4QOS00hvPDddEcCpwA1cMykWNdJUK50AjbRgbLZ9FLPyBa78i0NwsQZLSV67elpJU71L1Pt9yqVmE1C6XeSI-LV8Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDuKNezAAAAckYFAAAAAAAAgmlkgnY0gmlwhEDhTgGJc2VjcDI1NmsxoQIgMUMFvJGlr8dI1TEQy-K78u2TJE2rWvah9nGqLQCEGohzeW5jbmV0cwCDdGNwgiMog3VkcIIjKA,enr:-MK4QMlRAwM7E8YBo6fqP7M2IWrjFHP35uC4pWIttUioZWOiaTl5zgZF2OwSxswTQwpiVCnj4n56bhy4NJVHSe682VWGAYYDHkp4h2F0dG5ldHOIAAAAAAAAAACEZXRoMpDuKNezAAAAckYFAAAAAAAAgmlkgnY0gmlwhJK-7tSJc2VjcDI1NmsxoQLDq7LlsXIXAoJXPt7rqf6CES1Q40xPw2yW0RQ-Ly5S1YhzeW5jbmV0cwCDdGNwgiMog3VkcIIjKA,enr:-MS4QCgiQisRxtzXKlBqq_LN1CRUSGIpDKO4e2hLQsffp0BrC3A7-8F6kxHYtATnzcrsVOr8gnwmBnHYTFvE9UmT-0EHh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDuKNezAAAAckYFAAAAAAAAgmlkgnY0gmlwhKXoVKCJc2VjcDI1NmsxoQK6J-uvOXMf44iIlilx1uPWGRrrTntjLEFR2u-lHcHofIhzeW5jbmV0c4gAAAAAAAAAAIN0Y3CCIyiDdWRwgiMo,enr:-LK4QOQd-elgl_-dcSoUyHDbxBFNgQ687lzcKJiSBtpCyPQ0DinWSd2PKdJ4FHMkVLWD-oOquXPKSMtyoKpI0-Wo_38Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDuKNezAAAAckYFAAAAAAAAgmlkgnY0gmlwhES3DaqJc2VjcDI1NmsxoQNIf37JZx-Lc8pnfDwURcHUqLbIEZ1RoxjZuBRtEODseYN0Y3CCIyiDdWRwgiMo,enr:-KG4QLNORYXUK76RPDI4rIVAqX__zSkc5AqMcwAketVzN9YNE8FHSu1im3qJTIeuwqI5JN5SPVsiX7L9nWXgWLRUf6sDhGV0aDKQ7ijXswAAAHJGBQAAAAAAAIJpZIJ2NIJpcIShI5NiiXNlY3AyNTZrMaECpA_KefrVAueFWiLLDZKQPPVOxMuxGogPrI474FaS-x2DdGNwgiMog3VkcIIjKA
  --http
  --http-address=0.0.0.0
  --http-port=4000
  --listen-address=0.0.0.0
  --target-peers=100
  --http-allow-sync-stalled
  --execution-endpoint=http://127.0.0.1:8551
  --jwt-secrets=/home/ubuntu/jwtsecret
  --count-unrealized
  --slots-per-restore-point=32

Chain/Network: Zhejiang

When requesting the block body at slot 80646 via the beacon api curl -v http://localhost:4000/eth/v1/beacon/blocks/43200 we get the following error:

{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0x1c5f2c9cbaa2e7c7abdb6fa73e7562beea21ce6af8a0e52a5634f6283db07418, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

This is also the case for all successive blocks up to block 91765 after which the api returns again valid results.

We are using lh version v3.5.0 together with erigon (latest devel). According to the lh devs (see sigp/lighthouse#4035) this is an issue with erigon.

[michaelsproul]

Lighthouse uses eth_getBlockByHash currently to reconstruct payloads, and is expecting all Capella blocks to have a non-null withdrawals field.

We will soon switch to using the new PayloadBodies methods, which will alleviate the issue on our side. The underlying Erigon issue with eth_getBlockByHash should probably still be resolved though.

[hexoscott]

Hi @peterbitfly, there was a commit merged prior to version 2.39.0 (#6893) that could well fix this issue. It would require a wipe and resync to be applied correctly though. Just checking if you did the re-sync before I investigate further, thanks

[peterbitfly]

Hello, we did a full resync with erigon version 2.40.0-stable-38fb38a9 but the issue still persists.

[hexoscott]

Thanks @peterbitfly, I'll take a further look

[peterbitfly]

Hello, we are now encountering the same issue on goerli with erigon v2.42.0 and lh v4.0.1:

curl localhost:4000/eth/v1/beacon/blocks/5193760
{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

Currently this bug makes retrieval of historical blocks after the shanghai fork impossible.

[michaelsproul]

@peterbitfly We could also work around this on the LH side if we merge this PR: sigp/lighthouse#4102. I'll update it for unstable this weekend

You could also work around this by syncing LH through the merge (not from current finalized) with --prune-payloads false

[Giulio2002]

Lighthouse uses eth_getBlockByHash currently to reconstruct payloads, and is expecting all Capella blocks to have a non-null withdrawals field.

We will soon switch to using the new PayloadBodies methods, which will alleviate the issue on our side. The underlying Erigon issue with eth_getBlockByHash should probably still be resolved though.

what you are saying is that you guys are expecting [] instead of null, is that correct?

[michaelsproul]

Yeah, if it's a Capella execution payload the withdrawals should be [] when they're empty, not null or missing

[Giulio2002]

Ok, I will fix that; however, you should consider properly handling it on your side as well (the null case that is). Just a nice thing to have :).

[yperbasis]

Hello, we are now encountering the same issue on goerli with erigon v2.42.0 and lh v4.0.1:

curl localhost:4000/eth/v1/beacon/blocks/5193760
{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

Currently this bug makes retrieval of historical blocks after the shanghai fork impossible.

@peterbitfly But https://goerli.etherscan.io/block/5193760 predates Shanghai and so shouldn't have withdrawals. Do you observe the issue for Shanghai blocks?

[guybrush]

@peterbitfly But https://goerli.etherscan.io/block/5193760 predates Shanghai and so shouldn't have withdrawals. Do you observe the issue for Shanghai blocks?

https://goerli.beaconcha.in/slot/5193760 does not predate Shanghai: CAPELLA_FORK_EPOCH: 162304

we get the error from the beaconnode-api

curl localhost:4000/eth/v1/beacon/blocks/5193760
{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

curl localhost:4000/eth/v2/beacon/blocks/5193760
{"code":500,"message":"UNHANDLED_ERROR: ExecutionLayerErrorPayloadReconstruction(0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d, EngineError(Api { error: Json(Error(\"missing field `withdrawals`\", line: 0, column: 0)) }))","stacktraces":[]}

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBlockByHash","params":["0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d",false],"id":67}' localhost:8545
{"jsonrpc":"2.0","id":67,"result":{"baseFeePerGas":"0xe3a9b59f","difficulty":"0x0","extraData":"0xd883010b03846765746888676f312e32302e32856c696e7578","gasLimit":"0x1c9c380","gasUsed":"0xb4f86f","hash":"0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d","logsBloom":"0x10062000040944ac011800571c1805d054114000c809144100a06050d415b048002e100004381001028058e021840887500a050a024259b0185816860036620002817198944029484610022b62a0b602000700440580b000a8708108c2a0012142412401078c8c43c26c80ac2ab0a9000110003200025383182b055aac080049071a23748841843b00020440d810094504053489142604140001112452c3800002083003961605390d826082040a06a1b0449a14244d04c08100156c01060083002041325018a40f0402940145124410554c532420994800000638581806a5e25857182800400445e0140110a1e080a18000002808b90948018298000a8278c0","miner":"0x000095e79eac4d76aab57cb2c1f091d553b36ca0","mixHash":"0x0573d7cc2b3385a115860a5cea3f9f8cdd1dfc57dc148415d1aca59650ceb18a","nonce":"0x0000000000000000","number":"0x841508","parentHash":"0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599","receiptsRoot":"0x35862b7877512cbd009455089d2d1b396cef7279b639d982961d84e58c2c1c47","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","size":"0x36390","stateRoot":"0xe5c12ba85ee8852a903361776d3212f02416fb64ac7ad467b0acec19b053b8a5","timestamp":"0x6410f5e0","totalDifficulty":"0xa4a470","transactions":["0x339dbca7d7bea4e12e2051bd844814415b427cb6a94dfa701ac32510f1db076f","0x1ab8b9775ca267fc64accaa78290e55dc5b801475e5e722fe2822815f5809499","0x9234b59f482f1f4df3da90766c93a0d8b97be59fb8a972954c02989e0879224e","0x5d5768ab15eaab6bcfbda8a7858327f3b45e7bc0e504cb6720135b971ab3cfea","0x0eae7d068c2d0ecbcd99653e80d5b38f9fc0ee9df2af824f1c45ab3393d99913","0xd88429518f37d55da414e8db551832c780723da5635c008f9b9802e08094388e","0x56e8285fa24d90aaf35f2d3d83c2411cdbc28d3f190153c2b5681df4ad5b9e1a","0xd30638813dac00827329354a2d4547e30c4076e0f60b448fa5b51a09fc119f5a","0xe78790a6c80057023a8b1d0b8a9b6aa208abbcbd099b5ebc33226aac7fc88fa1","0xff547c8c9851208a531aa46202f50ee56011845f84339576ca122e683d350386","0x21fbfed46ccbbf5268b82704004fb6a0e1f0b8470c0b874aaa476ee43366b2d5","0x22bc8078c3df62da7a14ad5b03736f8e178d4e93c396478767d5ed550cd65a12","0x541616056663f65bdda60acec719db727b7167b1c8c374534783c77111c8e312","0x7002e072d67eb2f6ceecac146330ca0e7cf7a1a7f3cbd88530d72c42690dd03c","0xf84f69287c643cb94cd47127869d43e534d618ad409647c9eb2f4e242afb6fef","0x221fe88f84b3e164b0613a1d245247c86e453844f6f55516b9d6453713592bf1","0xb48a427f2616a33e7fc82b3cd76c0feee2fcc4769fa7ff20092627fc1fe0bdee","0xd993948626c51194e0b421739d1902b0683f3808fe93e9a708b8e82f6599ca07","0xd94d2e33934ada23d68ff201fd5ee07189924117662351b91cf48ac6fb3a0df5","0xa008024264cd3a2a7c06af44c1f62e0f11d025f72d03def6073ae15e6f39437c","0xa704e3f4808656f78aefd4df52620264a1c509f71558e9f4a669aba95fe8d6cc","0x1c1cd78589e65b48f5dfbc999599eed758b8e16b12acb7d65dffa6a9eb1ad325","0xffe5a97aee8822c8ce6a49b27c9d1f8860eb2f76749535310b82f7c5d9a9854d","0xc35495669e9c94a009c40106b58d34a7efa0eb44b8d94c64b7fafcb772249f13","0xbbb5ff0956c38718ddd49fe0269594e1420fb6649f640ffe5afeb02a14f973b9","0x24ef38c89daa772a0e88903004258e7f6a192c190bec61d4c7fb713d8190086d","0xcacc5bcc63ba93b8491cad2c387b8e72622cf4a769faf2aec50dc64b3d7a2afb","0xb363e69abb51913de21365ebf523891488c164cf431a16fdefa1c472041d9846","0xb4764d71b89b4a7a4b3c103d74dd3f1cdb6541285422926da6b437571e9be136","0xc789bdbdf26e9e275f866d77b14a7030c2639088efabf1636c4ca52744b3dda7","0xc0e1ced729d7b4ffdfc1ed1d9b903cc5f7ff996c554f625a685ff660d15621ad","0xb9339fd5b6d275fdc31bd4ee6f9eb7cdc1033d18df78250e85cfa81e39ce2fc0","0x032e447a579546f0a88975ddd3d181c31752947572ab461dff53b9a95bd631b4","0x7b0d55252b51c93be26c944d456a74e48625121c569ab3c57d439f9d0bcf46f2","0x048f7b4d607f9c97c100560fc0e85cab82e0061631d2b5926c21e7e518f8a919","0x0c4800a409f0b3b85d741b9d1a8d58fe34e091c4e3312c31b43c9205228aef5c","0x012d9532ea2f555460a25d49c07b2902b375df9b7d8867626b05dd568027eb13","0xdd11b271d1f70aefba30d667b08fe63998028c88e2ce90c09cfb65f9636ac50a","0x305813f2494b4ede4538f2b3951451e73e1126a7d01324ba3e0c6ce042eff349","0x9efd36cc8a7c5828ce450d92eb58804706d31a5d8d8a75a946a9f1fa98a847f3","0xbfd129fa52d7e46ad31dcda35edcaaaab33b3210289bd532c07f8d60e37f6e0b","0xa6f996fb9413c6f21b65f14bef1e508fdab68861790872c078da46f93b09eae7","0x6df0388cb234d644704e7f9badf11a5c463e736a33fce8d4d4ff69f8cbb1a461","0x9f4a6c11bb4f221b3dd49b948ea4af62ff12196700224352a5f33b849f617022","0x89968267c8bf7c479705ed83c1481cb4fa7c5284305ca762cc8215a6572a9f2d","0xfc795f51ca1bf410f74aff45d6cd5d4fcb30f684d5c1fc4329af04d96eff0e99","0x8e625ab0d0e0a749aed740a7ec7de1e092cd33ddb58f26139bfa01f363e1db95","0xdc786e600f55206c4c4175b5827045891da6bfe81ead6ae25ed6b71137d5dc43","0x98b0cda5e24b6c390902a6e8f2456e18c314dfbba916e327ad30160e9d043a92","0xb2d4f3142d99ae01a8afb874ce34f4cbfa88ad88284f457303ed4a0c402db861","0x359dfd55766d430470c27928f4e061d26453e3132446d22507931b53c74bfa8e","0x2a57f15189d09a504d44c0436ca246e6b9fa87dd97b107898a536a314fdc87bc","0x69edfa771121c67ff95706135c6076a978032ad9102d79e03de4eb6c8c6c1017","0xed63a06251a911bdc212d8cf0c7819f62ddfdea4db4b90c1fab09e587be6aa1a","0x5c7e0aa84bc6a9df143f64182b03d5e58473beec91cd9af54d6d02566dfa65cc","0x72845022bf6c5a75ea3077b60099e74e9dd52a6078d6f4108c9ab2be467d0f9a","0xdd0f5695780f92427ffcf74bb2589cb4c8457e1c0bcf3060c41ecfbed5a95d8a","0x5055598ba3c339b95652813488cddb6e77bb8af4064f22abb4e8f822eaea666d","0x325de12ece40c1ba5966c287baa4026bccf0d3c75d0524a5c4a264103c5f5063","0xa8b20f643544f35de2b36d7b2fe0e102019b845a187d1ccbbecaf3fef03ccce0","0x64aaba902b903f3236cddd86c3665b5e5b2fefe80b17a0b01cbdde4cf37ec273"],"transactionsRoot":"0x2ee5e31f1db763d474a470058c8a3d8d6ab3fc0eebab3519d3bfb68ee4c034c6","uncles":[],"withdrawalsRoot":"0xab5edaeaec5c53fc12822210ece094b8e114bcc4780b5cb249deb44693ae4f57"}}