bug(rln): problems syncing the rln tree, different roots

[alrevuelta]

Problem

Using 0x0A988fd9CA5BAebDf098b8A73621b2AaDa6492E8
Commit 5638bd06bbab5f52aeb399b2cf162c8b834ce011

When continuosly injecting new memberships into the rln contract (at a rate of one per block or so) different nodes calculate different merkle roots when syncing. This creates a "fork" in the network, where some nodes reject valid messages and others accept them. This issue seems to happen (or at least more often) when continuosly injecting new memberships into the contract.

After stopping injecting memberships and inspecting the database of different nodes, they seem to split into:

• Nodes that have synced 5673 memberships
• Nodes with 5672 memberships

This off-by-one membership seems to be the rootcause of the issue. If one of the first memberships was lost, then the nodes will calculate completely different trees and won't be able to validate the same messages.

Notes:

• Cleaning the db and resyncing again seems to fix the issue.
• This happens to 1 every 10 nodes or so.

[alrevuelta]

@rymnc and @alrevuelta are investigating this right now

[alrevuelta]

After a long debugging session with @rymnc, we have concluded that some nodes were missing memberships to be inserted in the tree (1 in most of the cases). Even if just one membership was missing, this led to a completely different tree, with a different merkle root.

Memberships were missing because the node was missing events from the chain. And the node was missing events from the chain because there was a delay between the "initial sync in chunks" and the "real time streaming of blocks". Meaning that in some cases, we were skipping 1 or 2 blocks in the sync process. This is why this issue appeared when constantly injecting new memberships into the contract.

Fix on its way.
#2013