feat: sharded peer management Round 2

[SionoiS]

Description

Peer manager must track peers per shard. Each with a target number of in/out peers and then connect/prune connections accordingly.

This new feature is behind a config flag

N.B. The first version was reverted because of backward-compatibility problems. Named sharding is deprecated but not unsupported.

Changes

• Per shard pruning of connections
• Per shard management of relay in/out connections
• Refactor/clean-up
• config flag

Tracking #1940

[github-actions]

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2332

Built from 4b98690

[SionoiS]

Adding randomized peer selection speed-up reaching steady-state relay connections when using waku-simulator.

[SionoiS]

I will gate the new feature behind a flag so that we can test this if needed.

[github-actions]

This PR may contain changes to configuration options of one of the apps.

If you are introducing a breaking change (i.e. the set of options in latest release would no longer be applicable) make sure the original option is preserved with a deprecation note for 2 following releases before it is actually removed.

Please also make sure the label release-notes is added to make sure any changes to the user interface are properly announced in changelog and release notes.

[gabrielmer]

Looks amazing! Thank you! Added a couple questions :))

[gabrielmer]

Thanks so much! 🔥

[jm-clius]

Thanks. Lgtm in terms of introducing the same functionality as before but now hidden behind a feature flag. Two thoughts:

• could we target this for testing in deployed waku-simulator network and waku.test. I think the testing here is important enough to warrant creating a new issue just to track this, given that peer management is so fundamental
• although named shards should be deprecated, is there a way to reconcile the two peer management paradigms even before named shards disappear? For example, it seems possible to consider a uint32 hash of named shards as the unique index representation of that shard.

[SionoiS]

• could we target this for testing in deployed waku-simulator network and waku.test. I think the testing here is important enough to warrant creating a new issue just to track this, given that peer management is so fundamental

It was tested with waku-simulator but I agree that deploying to a fleet is a good idea.

• although named shards should be deprecated, is there a way to reconcile the two peer management paradigms even before named shards disappear? For example, it seems possible to consider a uint32 hash of named shards as the unique index representation of that shard.

I never though of that but yes it could work. I'll look into it.

edit: The only way it would make sense to me would be if the default pubsub topic was given a shard number but then it breaks backward compatibility and conflict with static shard namespace. Seams a lot of work for not much benefit.

[SionoiS]

status-im/infra-waku#10