Node can no longer synchronize with latest versions of op-node and op-reth

[MrFrogoz]

Do you know why this error is generated? op-reth gets stuck because op-node generates this error

ExecStart=/root/build/op-node \
    --network=base-mainnet \
    --l1=..... \
    --l2=http://localhost:8551 \
    --rpc.addr=127.0.0.1 \
    --rpc.port=60004 \
    --l2.jwt-secret=/root/build/jwt.hex \
    --l1.trustrpc \
    --l1.rpckind=basic \
    --l1.beacon=..... \
    --p2p.bootnodes=enr:-J24QNz9lbrKbN4iSmmjtnr7SjUMk4zB7f1krHZcTZx-JRKZd0kA2gjufUROD6T3sOWDVDnFJRvqBBo62zuF-hYCohOGAYiOoEyEgmlkgnY0gmlwhAPniryHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQKNVFlCxh_B-716tTs-h1vMzZkSs1FTu_OYTNjgufplG4N0Y3CCJAaDdWRwgiQG,enr:-J24QH-f1wt99sfpHy4c0QJM-NfmsIfmlLAMMcgZCUEgKG_BBYFc6FwYgaMJMQN5dsRBJApIok0jFn-9CS842lGpLmqGAYiOoDRAgmlkgnY0gmlwhLhIgb2Hb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJ9FTIv8B9myn1MWaC_2lJ-sMoeCDkusCsk4BYHjjCq04N0Y3CCJAaDdWRwgiQG,enr:-J24QDXyyxvQYsd0yfsN0cRr1lZ1N11zGTplMNlW4xNEc7LkPXh0NAJ9iSOVdRO95GPYAIc6xmyoCCG6_0JxdL3a0zaGAYiOoAjFgmlkgnY0gmlwhAPckbGHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQJwoS7tzwxqXSyFL7g0JM-KWVbgvjfB8JA__T7yY_cYboN0Y3CCJAaDdWRwgiQG,enr:-J24QHmGyBwUZXIcsGYMaUqGGSl4CFdx9Tozu-vQCn5bHIQbR7On7dZbU61vYvfrJr30t0iahSqhc64J46MnUO2JvQaGAYiOoCKKgmlkgnY0gmlwhAPnCzSHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQINc4fSijfbNIiGhcgvwjsjxVFJHUstK9L1T8OTKUjgloN0Y3CCJAaDdWRwgiQG,enr:-J24QG3ypT4xSu0gjb5PABCmVxZqBjVw9ca7pvsI8jl4KATYAnxBmfkaIuEqy9sKvDHKuNCsy57WwK9wTt2aQgcaDDyGAYiOoGAXgmlkgnY0gmlwhDbGmZaHb3BzdGFja4OFQgCJc2VjcDI1NmsxoQIeAK_--tcLEiu7HvoUlbV52MspE0uCocsx1f_rYvRenIN0Y3CCJAaDdWRwgiQG \
    --syncmode=execution-layer

op-node

Nov 12 16:00:39  op-node[11346]: t=2024-11-12T16:00:39+0100 lvl=warn msg="Failed to get payload" payload_id=0x0367cca7b5d9ed65 payload_id=0x0367cca7b5d9ed65 err="Unknown payload"
Nov 12 16:00:39  op-node[11346]: t=2024-11-12T16:00:39+0100 lvl=warn msg="Cannot seal block, payload ID is unknown" payloadID=0x0367cca7b5d9ed65 payload_time=1731421391 started_time=2024-11-12T16:00:39+0100
Nov 12 16:00:39  op-node[11346]: t=2024-11-12T16:00:39+0100 lvl=warn msg="Block sealing job of derived attributes expired, job will be re-attempted." build_id=0x0367cca7b5d9ed65 timestamp=1731421391 err="failed to seal execution payload (ID: 0x0367cca7b5d9ed65): input error -38001: Unknown payload"
Nov 12 16:00:39  op-node[11346]: t=2024-11-12T16:00:39+0100 lvl=info msg="processing L2 range request" target=22316021 end=0x53686b95c9200e376335f43490df86dae0b0705f11e05274ba920e86679900c9:22317094 rangeReqId=26
Nov 12 16:00:39  op-node[11346]: t=2024-11-12T16:00:39+0100 lvl=info msg="no peers ready to handle block requests for more P2P requests for L2 block history" target=22316021 end=0x53686b95c9200e376335f43490df86dae0b0705f11e05274ba920e86679900c9:22317094 current=22317008

op-reth

Nov 12 15:52:39 op-reth[11038]: 2024-11-12T14:52:39.333098Z  INFO Status connected_peers=30 latest_block=22316021
Nov 12 15:52:50 op-reth[11038]: 2024-11-12T14:52:50.325326Z  INFO New payload job created id=0x0367cca7b5d9ed65 parent=0x9db221e7994f1c45f6ddc5dd33d5b3981eef33c09c93d82b0934f2893c992f0b
Nov 12 15:53:01 op-reth[11038]: 2024-11-12T14:53:01.970944Z  INFO New payload job created id=0x0367cca7b5d9ed65 parent=0x9db221e7994f1c45f6ddc5dd33d5b3981eef33c09c93d82b0934f2893c992f0b
Nov 12 15:53:04 op-reth[11038]: 2024-11-12T14:53:04.333167Z  INFO Status connected_peers=30 latest_block=22316021
Nov 12 15:53:12 op-reth[11038]: 2024-11-12T14:53:12.133933Z  INFO New payload job created id=0x0367cca7b5d9ed65 parent=0x9db221e7994f1c45f6ddc5dd33d5b3981eef33c09c93d82b0934f2893c992f0b
Nov 12 15:53:23 op-reth[11038]: 2024-11-12T14:53:23.284764Z  INFO New payload job created id=0x0367cca7b5d9ed65 parent=0x9db221e7994f1c45f6ddc5dd33d5b3981eef33c09c93d82b0934f2893c992f0b
Nov 12 15:53:29 op-reth[11038]: 2024-11-12T14:53:29.333350Z  INFO Status connected_peers=30 latest_block=22316021
Nov 12 15:53:33 op-reth[11038]: 2024-11-12T14:53:33.964511Z  INFO New payload job created id=0x0367cca7b5d9ed65 parent=0x9db221e7994f1c45f6ddc5dd33d5b3981eef33c09c93d82b0934f2893c992f0b
Nov 12 15:53:44 op-reth[11038]: 2024-11-12T14:53:44.369836Z  INFO New payload job created id=0x0367cca7b5d9ed65 parent=0x9db221e7994f1c45f6ddc5dd33d5b3981eef33c09c93d82b0934f2893c992f0b
Nov 12 15:53:54 op-reth[11038]: 2024-11-12T14:53:54.333186Z  INFO Status connected_peers=30 latest_block=22316021
Nov 12 15:53:56 op-reth[11038]: 2024-11-12T14:53:56.480768Z  INFO New payload job created id=0x0367cca7b5d9ed65 parent=0x9db221e7994f1c45f6ddc5dd33d5b3981eef33c09c93d82b0934f2893c992f0b

[MrFrogoz]

With geth and erigon the problem does not occur but it seems only with reth, I also add that without --l2.enginekind=reth the node always remains blocked, while if the variable is set, it manages to recover until live and then it blocks again after a couple of minutes. I also tried to change provider like l1, nothing changes

[protolambda]

Nov 12 16:00:39  op-node[11346]: t=2024-11-12T16:00:39+0100 lvl=warn msg="Block sealing job of derived attributes expired, job will be re-attempted." build_id=0x0367cca7b5d9ed65 timestamp=1731421391 

The timestamp 1731421391 is Tue Nov 12 2024 14:23:11 GMT+0000 in UTC, which is not quite the timestamp like seen in the log (T16:00:39). Maybe reth is timing out the block-building job instantly, since it's an older block that is being reproduced from block-attributes? (verification works by processing block inputs, a lot like regular sequencing, just without the transaction pool)

[MrFrogoz]

So how should I proceed to avoid this block? Do you perhaps have to resolve by forwarding the problem to the reth team? for now I have set a restart every 5 minutes to have it somehow the node always live, if I don't restart it it remains blocked all the time