Epic: Auto-sharded Waku Network that scales

[chair28980]

General Track
This tracks the main body of work to build an autoscaling, autosharded network and provides the foundation for the other tracks.

Milestone 1.1: Network requirements and design
Goal: 31 Aug 2023

This tracks the work necessary to determine the specifications for a public Waku Network and a rough design to achieve this. The deliverable of this milestone is to answer questions such as:

How many shards to we want to support initially? Do we know how to increase this in future?
What is the reasonable bandwidth expectation per shard?
What is a reasonable message rate expectation per shard?
What is a reasonable number of publishers per shard?
What is a reasonable max message size for the shard?
What membership mechanisms and limits should we set and should this be done on a per-shard basis?
Milestone 1.2: Autosharding for autoscaling
Goal: 30 Sept 2023

This milestone tracks designing and implementing an autosharding strategy based on content topics and the design requirements established in 1.1. The first phase would likely be to launch with a limited number of shards (perhaps no more than 128?). More shards can be added as the network grows organically, perhaps using some "generation ID", that allows opening more shards with successive generations.

Milestone 1.3: Node bandwidth management mechanism
Goal: 31 Oct 2023

As the autosharded public network grows and traffic increases per shard, we want to provide some bandwidth management mechanisms for relay nodes to dynamically choose the number of shards they support based on bandwidth availability. For example, when the network launches, it's reasonable for relay nodes to support all shards and gradually unsubscribe from shards as bandwidth increases. The minimum amount of shards to support would be 1, so the network design and DoS mechanisms (see Track 3) would have to provide predictable limits on max bandwidth per shard. We could also envision a bandwidth protection mechanism that drops messages over a threshold, but this will affect the node's scoring so should be carefully planned.

Milestone 1.4: Sharded peer management and discovery
Goal: 30 Nov 2023

This milestone tracks work necessary to allow applications to interact with shards in a transparent manner. Relay nodes should preferably be subscribed to the shards carrying the content topic(s) the application is interested in. Where this is not the case, the application should be able to interact with service peers within the shards it's interested in via a peer management scheme. The peer manager should be tracking peers from all public shards or be able to discover such peers on demand.

Milestone 1.5: Launch and dogfood integrated public Waku Network MVP
Goal: 31 Dec 2023

Combining outputs from all other tracks and milestones, launch and dogfood the Waku Network MVP as set out in waku-org/research#1 by the end of the year.

[jm-clius]

Closing, as epics created for each sub-milestone here.