feat: Peer management with shard as a dimension

[chair28980]

If we assume that peer discovery takes care of filtering peers by shard, these peers must now be managed in a way that makes sense in a dynamic auto-sharded environment. Some requirements/ideas:

• peer store must preferably try to keep at least a target n peers for each subscribed shard (note that dynamically subscribing/unsubscribing from shards could complicate this)
• peer manager must attempt to maintain a healthy relay connectivity for each subscribed shard
• a mechanism to perform ad-hoc discovery (or another solution) to cater for when a node subscribes to a shard for which there are no tracked peers

Priority: Critical for launch

[chair28980]

waku-org/research#3

@danisharora099 @weboko

Per the research issue, can y'all verify if this is required for JSWaku and assign ownership as necessary. Thank you!

[weboko]

Questionable if it belongs more to auto sharding or static sharding.
@chair28980 please, clarify on this.

[fryorcraken]

Questionable if it belongs more to auto sharding or static sharding. @chair28980 please, clarify on this.

This is more of autosharding because we assume for static sharding that shard do not change mid execution.

[jm-clius]

Indeed, this is more relevant within the autosharding effort, but is a general enough concept to apply to static sharding as well: if you want to publish to a specific shard (whether statically configured or hashed from content topic via autosharding), you need to have a reasonable set of steps to find an appropriate peer that supports that shard. This could be:
(1) peer that you're already tracking
(2) new discovery process that you kick off until you find a peer (or timeout?) that supports this shard

The epic here (1.4) is the right one.

[chair28980]

@danisharora099 per discussion in js-waku pm meeting, has this issue been addressed in any of your PRs?

[danisharora099]

@danisharora099 per discussion in js-waku pm meeting, has this issue been addressed in any of your PRs?

This should be more relevant with the work @adklempner is doing wrt autosharding. Not sure about how exactly this fits in with static sharded peer mgmt. Would be curious for some clarity there.

[chair28980]

cc @fryorcraken @jm-clius can you provide context re: Danish's query wrt autosharding above.

[adklempner]

I can take this on after we merge in the current batch of open PRs (autosharding and peer connectivity)

[danisharora099]

should mostly be resolved with #1756

[adklempner]

@danisharora099 does it make sense to create an issue for each of the bullet points in OP?

- peer store must preferably try to keep at least a target n peers for each subscribed shard (note that dynamically subscribing/unsubscribing from shards could complicate this)
- peer manager must attempt to maintain a healthy relay connectivity for each subscribed shard
- a mechanism to perform ad-hoc discovery (or another solution) to cater for when a node subscribes to a shard for which there are no tracked peers

I'm not sure if all of these were resolved by your PR

[chair28980]

Parent epic here: waku-org/pm#67

[danisharora099]

a mechanism to perform ad-hoc discovery (or another solution) to cater for when a node subscribes to a shard for which there are no tracked peers

is outside the scope of js-waku for now as we don't allow subscribing to new shards once the node has started

peer manager must attempt to maintain a healthy relay connectivity for each subscribed shard

i dont think this applies to js-waku either considering we aren't prioritizing relay

peer store must preferably try to keep at least a target n peers for each subscribed shard (note that dynamically subscribing/unsubscribing from shards could complicate this)

connection manager currently connects to all relevant found peers -- perhaps an issue could be opened to add a MAX_CAP on these peer connections