State synchronization/consensus in waku2 store protocol

[staheri14]

Problem

The current state synchronization of the Waku2 store protocol is empowered by the Waku FT-store protocol.

• Store nodes (while being online) listen to the network and store all the messages of the community.
• Store nodes are allowed to go offline and then recover the gap in their message history by performing time-based queries from another store node that has been online for that duration.
• Waku2 FT-store works based on the following assumptions
  • At any time, there is at least one online store node from which the history can be retrieved
  • Nodes (including store nodes and other regular nodes who publish messages) have synchronized clocks with a +-20 seconds error margin
  • There is no network partition, that is all the store nodes will ultimately receive all the messages transmitted in the network

Under the aforementioned assumptions, Waku2 FT-store protocol provides eventual consistency. However, the message state of store nodes may diverge for some points in time due to network delay.

However, we are seeking a better solution to get past the current assumptions and limitations, namely:

• No reliance on the synchronization of nodes' clocks
• Enforces replicated message state across store nodes for some configurable intervals
• Better network partition tolerance

Solution ideas and ongoing efforts

There is ongoing research and development on a more advanced state synchronization protocol. The initial proposal is provided in #73. This proposal has a potential scalability issue and is prone to snowball effect which can eat up a large portion of store nodes bandwidth.
Further solution proposal with better scalability is currently underway (I will link it to this issue when it gets ready).