Invalid state root error with ZERO_HASH

[twoeths]

Describe the bug

Investigate this log in goerli nodes:

Sep-26 00:48:25.493[chain]           ^[[31merror^[[39m: Block error slot=3970442 root=0x2656995d89a183e60a7199a3e0755af6155dc9c2991af44348bf5f3786e10c2b, expectedRoot=0x0000000000000000000000000000000000000000000000000000000000000000
Error: BLOCK_ERROR_INVALID_STATE_ROOT
    at verifyBlocksStateTransitionOnly (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlocksStateTransitionOnly.ts:51:13)
    at BeaconChain.verifyBlocksInEpoch (file:///usr/app/packages/beacon-node/src/chain/blocks/verifyBlock.ts:70:7)
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at BeaconChain.processBlocks (file:///usr/app/packages/beacon-node/src/chain/blocks/index.ts:71:68)
    at Timeout.JobItemQueue.runJob [as _onTimeout] (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:92:22)

note that the expectedRoot is somehow ZERO_HASH, for Sep 25 it happened 78 times

grep -e expectedRoot=0x0000000000000000000000000000000000000000000000000000000000000000 -rn beacon-2022-09-25.log  | wc -l
78

Expected behavior

No error, or a good explanation of the error

[twoeths]

all of the blocks have ZERO_HASH state root and invalid, if we check the block explorer they are all missed (not included in chain)

beacon-2022-09-26.log:973523:Sep-26 13:51:37.247[chain]           error: Block error slot=3974358 root=0x6f261166376ebb03595a2d488fedc5bfb372a51a9843ec52b13db3c71c9f858f, expectedRoot=0x0000000000000000000000000000000000000000000000000000000000000000
beacon-2022-09-26.log:978334:Sep-26 13:55:49.136[chain]           error: Block error slot=3974379 root=0x6f0c8c539357d76886f942743235a1df06f9705645cd6a2411539e9e0cab1a76, expectedRoot=0x0000000000000000000000000000000000000000000000000000000000000000
beacon-2022-09-26.log:1016735:Sep-26 14:27:25.467[chain]           error: Block error slot=3974537 root=0xcbdc4da67a0d354db075b3996095ca55ab610dead72eaa4f17df38143e0df1a0, expectedRoot=0x0000000000000000000000000000000000000000000000000000000000000000
beacon-2022-09-26.log:1027224:Sep-26 14:36:01.454[chain]           error: Block error slot=3974580 root=0x842690fd90967ae25a67c4e6ac8e81db3c9432c4f28af47f469d446fde7abf13, expectedRoot=0x0000000000000000000000000000000000000000000000000000000000000000
beacon-2022-09-26.log:1028124:Sep-26 14:36:49.064[chain]           error: Block error slot=3974584 root=0xe88b14b11f1f9cc9b128484750ef70b4176b32897f622f30d8dca1f9171cb9c9, expectedRoot=0x0000000000000000000000000000000000000000000000000000000000000000
beacon-2022-09-26.log:1034595:Sep-26 14:42:01.007[chain]           error: Block error slot=3974610 root=0xc994d9913814894cb66002f4d3b614e33d0154d2d92ee339dfb8b05db01619f1, expectedRoot=0x0000000000000000000000000000000000000000000000000000000000000000

[twoeths]

The peer id logged is incorrect

Sep-27 00:01:26.469[network]       ^[[36mverbose^[[39m: Received gossip block slot=3977407, root=0x7bc6…52f8, curentSlot=3977407, peerId=bafzaajiiaijcca2szy2eje5bz2y6w4mucwv3pe7gi33p2b3a3z5u2qszjzt6eqnzxq, delaySec=2.4609999656677246

[twoeths]

we should be able to disconnect peers that send us too many rejected gossip messages

should bring this back #3781

last time the PR was reverted because we don't have a strong mechanism to find subnet peers and a clear gossipsub metrics

[dapplion]

we should be able to disconnect peers that send us too many rejected gossip messages

Sounds good to me

[dapplion]

Closing issues for specific incidents on old versions of Lodestar, please re-open if it happens with latest stable