Handle recurring ETH1_ERROR_DUPLICATE_DISTINCT_LOG

[g11tech]

On doing test runs of kintsugi with nethermind with deposit tracker enabled, following deposit cache error log dump was observed:

{
  depositEvents: [
    { blockNumber: 7521, index: 9, depositData: [Object] },
    { blockNumber: 8167, index: 10, depositData: [Object] }
  ]
}
Jan-27 20:10:02.474 [ETH1]            error: Error updating eth1 chain cache code=ETH1_ERROR_DUPLICATE_DISTINCT_LOG, newIndex=9, prevIndex=11
Error: ETH1_ERROR_DUPLICATE_DISTINCT_LOG
    at Eth1DepositsCache.add (/root/lodestar/packages/lodestar/src/eth1/eth1DepositsCache.ts:45:15)
    at Eth1DepositDataTracker.updateDepositCache (/root/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:164:5)
    at Eth1DepositDataTracker.update (/root/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:144:33)
    at Eth1DepositDataTracker.runAutoUpdate (/root/lodestar/packages/lodestar/src/eth1/eth1DepositDataTracker.ts:118:29)
parseDepositLog {
  log: {

This error shows that the tracker supposedly got a duplicate entry. However once this happens, this error repeats ad infinitum

Expected: Investigate the error , if genuine the error should be handled and tracker should move ahead, else find/fix the bug causing the error.

[dapplion]

This error is so annoying! Thanks for looking into it

[g11tech]

UPDATE:
The error occurs in these two post merge scenarios

• Both EL is in a post merge scenario and is ahead of lodestar: for e.g.
  • When lodestar is started, its deposit tracker correctly identifies that EL is ahead : { remoteHighestBlock: 8885, remoteFollowBlock: 8869 } and hence it starts fetching the deposit events till 8869 for example in this case till index 10, 8167

    {
       blockNumber: 8167,
       index: 10,
       depositData: {
         pubkey: [Uint8Array],
         withdrawalCredentials: [Uint8Array],
         amount: 32000000000,
         signature: [Uint8Array]
       }
    

  • As lodestar goes past the merge block, it "reorgs" the EL with its fcU calls, also causing the tracker to update its remoteHighestBlock and hence remoteFollowBlock as well: { remoteHighestBlock: 7235, remoteFollowBlock: 7219 }
  • now trackers start fetching deposit log again from fromBlock: 7220,toBlock: 7933} causing re-import of the deposits leading to reimporting this deposit event:

     blockNumber: 7521,
     index: 9,
     depositData: {
       pubkey: [Uint8Array],
       withdrawalCredentials: [Uint8Array],
       amount: 32000000000,
       signature: [Uint8Array]
     }
    

  • which throws this error because index 9 event has already been importted:

    Feb-04 08:57:55.720 [ETH1]            error: Error updating eth1 chain cache code=ETH1_ERROR_DUPLICATE_DISTINCT_LOG, newIndex=9, prevIndex=10
    Error: ETH1_ERROR_DUPLICATE_DISTINCT_LOG
        at Eth1DepositsCache.add (/usr/app/packages/lodestar/src/eth1/eth1DepositsCache.ts:46:15)
        at Eth1DepositDataTracker.updateDepositCache (/u
    

• Whenever lodestar restarts it starts from the previous archived finalized state, which could be a few epochs behind the blocks lodestar already fetched last time. This might again lead to "re-org" of the EL as lodestar will start sending fcU calls for previously send slots, again causing above scenario to repeat.

Suggested resolution if such scenario happens

• delete all the logs which are head of the deposit's , as well as unset them from the depositRootTree of DepositDataRootRepository,and then let it proceed from there
• Or ignore these duplicate events and move the underlying tracker along

@dapplion what resolution from above do you think would be suitable.