CL<>EL interop: Peers are getting downscored when EL errors on payload execution

[g11tech]

While importing blocks, EL errors with various kind of errors (bad merkle state, internal error etc) as well as possible timeout/connection refused error.

The treatment of all these currently is that that block verification errors and the peer is downscored, which leads to downscoring/banning of peer and soon making lodestar loosing all peers and going out of sync! The lodestar doesn't find peers till the peerstore directory is removed and then the same cycle happens again (based on EL again erroring)

Expected:
The EL executePayload call should be try catched for these errors, and a separate status responded from executionEngine, which should log the error, but lodesstar shouldn;t penalize these peers and also accept this blocks with syncing.

However there is debate in the devnet debug group is that CLs should try discriminate these error (like treat merkle state error as invalid). Waiting till that discussion resolves.

Observed with: lodestar <> nethermind

[dapplion]

Oh good points! With downgrade do you mean downscore?

[g11tech]

Oh good points! With downgrade do you mean downscore?

yes 🙂

[twoeths]

lodestar<>geth could not maintain a good number of peers too

the main reason is lodestar sent goodbye requests to other nodes with "peer score too low"

@g11tech is this the same issue or not, do I need to search for any specific logs to confirm?

[g11tech]

@tuyennhv yes, this is the same issue, I have cleared peerstore and restarted nodes, they seem to now have good peer count. Once this PR #3545 gets merged, peers shouldn't get penalized for EL malfunctions.