Update peer selection options for light protocols #787

chaitanyaprem · 2023-09-29T06:52:11Z

Description

Update req/resp protocols to use new peerSelection logic based on pubSub/contentTopic as per #680

Changes

Update peerSelection in lightpush to use peers filtered based on pubSubTopic.
Update Filter to use peerSelection options based on pubSubTopic
Update Store protocol to also use peerSelection options

Tests

chaitanyaprem · 2023-09-29T06:54:33Z

@richard-ramos , it is becoming challenging to maintain both peerManager and option of using waku protocols without peer-manager (i.e using utils peer code). As you can see in this PR, there already is a lot of checks and duplicate code generated.

Now with peer-selection based on pubSubTopic, which cannot be done by default libp2p peerStore alone(since we store peer specific pubSub topics in wakuPeerStore), I am wondering if it is time to retire this option of peerutils without peerManager. WDYT?

status-im-auto · 2023-09-29T06:54:37Z

Jenkins Builds

Click to see older builds (84)

❔	Commit	#️⃣	Finished (UTC)	Duration	Platform	Result
✔️	`4a18aa3`	#1	2023-09-29 06:54:37	~2 min	`nix-flake`	📄`log`
✔️	`4a18aa3`	#1	2023-09-29 06:55:00	~2 min	`linux`	📦`deb`
✔️	`4a18aa3`	#1	2023-09-29 06:55:27	~2 min	`tests`	📄`log`
✔️	`4a18aa3`	#1	2023-09-29 06:55:35	~3 min	`tests`	📄`log`
✔️	`4a18aa3`	#1	2023-09-29 06:56:28	~4 min	`android`	📦`tgz`
✔️	`4a18aa3`	#1	2023-09-29 06:58:27	~6 min	`ios`	📦`tgz`

✔️	`f1cfc48`	#2	2023-09-29 09:11:23	~1 min	`linux`	📦`deb`
✔️	`f1cfc48`	#2	2023-09-29 09:12:14	~1 min	`nix-flake`	📄`log`
✔️	`f1cfc48`	#2	2023-09-29 09:13:27	~3 min	`tests`	📄`log`
✔️	`f1cfc48`	#2	2023-09-29 09:13:37	~3 min	`tests`	📄`log`
✔️	`f1cfc48`	#2	2023-09-29 09:14:00	~3 min	`android`	📦`tgz`
✔️	`f1cfc48`	#2	2023-09-29 09:14:45	~4 min	`ios`	📦`tgz`

✔️	`31ed0c6`	#3	2023-10-01 08:34:09	~52 sec	`tests`	📄`log`
✔️	`31ed0c6`	#3	2023-10-01 08:35:10	~1 min	`nix-flake`	📄`log`
✔️	`31ed0c6`	#3	2023-10-01 08:35:22	~2 min	`tests`	📄`log`
✔️	`31ed0c6`	#3	2023-10-01 08:35:36	~2 min	`linux`	📦`deb`
✔️	`31ed0c6`	#3	2023-10-01 08:37:03	~3 min	`android`	📦`tgz`
✔️	`31ed0c6`	#3	2023-10-01 08:38:52	~5 min	`ios`	📦`tgz`

✖️	`6aa229b`	#4	2023-10-03 09:13:39	~37 sec	`tests`	📄`log`
✖️	`6aa229b`	#4	2023-10-03 09:14:24	~1 min	`tests`	📄`log`
✔️	`6aa229b`	#4	2023-10-03 09:15:10	~2 min	`nix-flake`	📄`log`
✔️	`6aa229b`	#4	2023-10-03 09:15:27	~2 min	`linux`	📦`deb`
✔️	`6aa229b`	#4	2023-10-03 09:16:45	~3 min	`android`	📦`tgz`
✔️	`6aa229b`	#4	2023-10-03 09:16:46	~3 min	`ios`	📦`tgz`

✖️	`6bb3638`	#5	2023-10-03 09:54:17	~25 sec	`tests`	📄`log`
✔️	`6bb3638`	#5	2023-10-03 09:54:51	~1 min	`linux`	📦`deb`
✖️	`6bb3638`	#5	2023-10-03 09:55:28	~1 min	`tests`	📄`log`
✔️	`6bb3638`	#5	2023-10-03 09:55:55	~2 min	`nix-flake`	📄`log`
✔️	`6bb3638`	#5	2023-10-03 09:57:39	~3 min	`android`	📦`tgz`
✔️	`6bb3638`	#5	2023-10-03 09:57:44	~3 min	`ios`	📦`tgz`

✖️	`1f80dde`	#6	2023-10-03 09:57:53	~1 min	`tests`	📄`log`
✔️	`1f80dde`	#6	2023-10-03 09:57:59	~1 min	`linux`	📦`deb`
✔️	`1f80dde`	#6	2023-10-03 09:58:41	~2 min	`nix-flake`	📄`log`
✖️	`1f80dde`	#6	2023-10-03 09:59:28	~2 min	`tests`	📄`log`
✔️	`1f80dde`	#6	2023-10-03 10:00:32	~3 min	`ios`	📦`tgz`
✔️	`1f80dde`	#6	2023-10-03 10:01:48	~4 min	`android`	📦`tgz`

✔️	`330cf92`	#7	2023-10-03 20:55:44	~1 min	`linux`	📦`deb`
✔️	`330cf92`	#7	2023-10-03 20:56:34	~1 min	`nix-flake`	📄`log`
✔️	`330cf92`	#7	2023-10-03 20:56:37	~2 min	`tests`	📄`log`
✔️	`330cf92`	#7	2023-10-03 20:56:45	~2 min	`tests`	📄`log`
✔️	`330cf92`	#7	2023-10-03 20:58:11	~3 min	`android`	📦`tgz`
✔️	`330cf92`	#7	2023-10-03 21:00:22	~5 min	`ios`	📦`tgz`

✔️	`edf5a85`	#8	2023-10-03 21:08:18	~2 min	`nix-flake`	📄`log`
✔️	`edf5a85`	#8	2023-10-03 21:09:58	~3 min	`tests`	📄`log`
✔️	`edf5a85`	#8	2023-10-03 21:10:26	~4 min	`android`	📦`tgz`
✔️	`edf5a85`	#8	2023-10-03 21:11:05	~5 min	`tests`	📄`log`
✔️	`edf5a85`	#8	2023-10-03 21:12:26	~6 min	`linux`	📦`deb`
✔️	`edf5a85`	#8	2023-10-03 21:13:31	~7 min	`ios`	📦`tgz`

✖️	`a108ee7`	#9	2023-10-12 13:13:06	~26 sec	`tests`	📄`log`
❌	`a108ee7`	#9	2023-10-12 13:13:07	~27 sec	`linux`	📄`log`
✖️	`a108ee7`	#9	2023-10-12 13:13:07	~26 sec	`tests`	📄`log`
✖️	`a108ee7`	#9	2023-10-12 13:13:36	~56 sec	`nix-flake`	📄`log`
❌	`a108ee7`	#9	2023-10-12 13:14:55	~2 min	`android`	📄`log`
❌	`a108ee7`	#9	2023-10-12 13:15:43	~3 min	`ios`	📄`log`

✖️	`0b6d93e`	#10	2023-10-12 14:05:11	~1 min	`tests`	📄`log`
✔️	`0b6d93e`	#10	2023-10-12 14:05:49	~1 min	`linux`	📦`deb`
✔️	`0b6d93e`	#10	2023-10-12 14:05:59	~2 min	`nix-flake`	📄`log`
✔️	`0b6d93e`	#10	2023-10-12 14:06:16	~2 min	`tests`	📄`log`
✔️	`0b6d93e`	#10	2023-10-12 14:08:12	~4 min	`android`	📦`tgz`
✔️	`0b6d93e`	#10	2023-10-12 14:08:22	~4 min	`ios`	📦`tgz`

✔️	`e9fff20`	#11	2023-10-12 17:04:30	~1 min	`tests`	📄`log`
✔️	`e9fff20`	#11	2023-10-12 17:04:41	~1 min	`linux`	📦`deb`
✔️	`e9fff20`	#11	2023-10-12 17:04:49	~1 min	`nix-flake`	📄`log`
✔️	`e9fff20`	#11	2023-10-12 17:05:08	~2 min	`tests`	📄`log`
✔️	`e9fff20`	#11	2023-10-12 17:06:27	~3 min	`android`	📦`tgz`
✔️	`e9fff20`	#11	2023-10-12 17:08:38	~5 min	`ios`	📦`tgz`

✖️	`4890984`	#12	2023-10-12 22:51:00	~23 sec	`tests`	📄`log`
✔️	`4890984`	#12	2023-10-12 22:51:48	~1 min	`linux`	📦`deb`
✖️	`4890984`	#12	2023-10-12 22:51:57	~1 min	`tests`	📄`log`
✔️	`4890984`	#12	2023-10-12 22:52:30	~1 min	`nix-flake`	📄`log`
✔️	`4890984`	#12	2023-10-12 22:54:26	~3 min	`android`	📦`tgz`
✔️	`4890984`	#12	2023-10-12 22:54:40	~4 min	`ios`	📦`tgz`

✔️	`268c41e`	#13	2023-10-12 22:58:00	~1 min	`linux`	📦`deb`
✖️	`268c41e`	#13	2023-10-12 22:58:08	~1 min	`tests`	📄`log`
✖️	`268c41e`	#13	2023-10-12 22:58:08	~1 min	`tests`	📄`log`
✔️	`268c41e`	#13	2023-10-12 22:58:47	~1 min	`nix-flake`	📄`log`
✔️	`268c41e`	#13	2023-10-12 23:00:09	~3 min	`android`	📦`tgz`
✔️	`268c41e`	#13	2023-10-12 23:00:10	~3 min	`ios`	📦`tgz`

✔️	`89da415`	#14	2023-10-13 01:03:21	~1 min	`linux`	📦`deb`
✔️	`89da415`	#14	2023-10-13 01:03:46	~1 min	`tests`	📄`log`
✔️	`89da415`	#14	2023-10-13 01:04:09	~1 min	`nix-flake`	📄`log`
✔️	`89da415`	#14	2023-10-13 01:04:20	~2 min	`tests`	📄`log`
✔️	`89da415`	#14	2023-10-13 01:05:45	~3 min	`android`	📦`tgz`
✔️	`89da415`	#14	2023-10-13 01:06:26	~4 min	`ios`	📦`tgz`

❔	Commit	#️⃣	Finished (UTC)	Duration	Platform	Result
✔️	`7aff588`	#15	2023-10-13 16:54:03	~1 min	`tests`	📄`log`
✔️	`7aff588`	#15	2023-10-13 16:55:31	~3 min	`tests`	📄`log`
✔️	`7aff588`	#15	2023-10-13 16:55:47	~3 min	`android`	📦`tgz`
✔️	`7aff588`	#15	2023-10-13 16:56:06	~4 min	`nix-flake`	📄`log`
✔️	`7aff588`	#15	2023-10-13 16:56:43	~4 min	`ios`	📦`tgz`
✔️	`7aff588`	#16	2023-10-13 17:20:19	~1 min	`linux`	📦`deb`

✔️	`07dfc54`	#17	2023-10-16 16:38:09	~1 min	`linux`	📦`deb`
✔️	`07dfc54`	#16	2023-10-16 16:38:47	~1 min	`tests`	📄`log`
✔️	`07dfc54`	#16	2023-10-16 16:39:11	~2 min	`nix-flake`	📄`log`
✔️	`07dfc54`	#16	2023-10-16 16:39:45	~2 min	`tests`	📄`log`
✔️	`07dfc54`	#16	2023-10-16 16:40:19	~3 min	`android`	📦`tgz`
✔️	`07dfc54`	#16	2023-10-16 16:41:17	~4 min	`ios`	📦`tgz`

richard-ramos · 2023-10-01T16:39:11Z

As long as the scenarios covered by the utils function can be met with the peer manager, it should be okay to retire the functions:

choose a random peer (optionally selecting the peer from a specific list)
chose an specific peer
fastest peer (optionally selecting the peer from a specific list)

chaitanyaprem · 2023-10-03T04:39:13Z

As long as the scenarios covered by the utils function can be met with the peer manager, it should be okay to retire the functions:
* choose a random peer (optionally selecting  the peer from a specific list)

* chose an specific peer

* fastest peer (optionally selecting the peer from a specific list)

The scenarios would be covered in peerManager.
My only concern was that during one of the initial discussions we had, you indicated that we wanted to keep the Waku protocols go-implementation in such a way that they could also be used independently using just a libp2p host and not necessarily by initializing a WakuNode.
If we retire these functions, then these options cannot be used for someone wanting to use Waku protocols without initializing a WakuNode.

richard-ramos

Looking good.
Left some few comments/questions

waku/v2/protocol/filter/client.go

waku/v2/protocol/peer_exchange/waku_peer_exchange_option.go

waku/v2/protocol/store/waku_store_protocol_test.go

waku/v2/peermanager/peer_manager_test.go

waku/v2/protocol/legacy_filter/waku_filter.go

waku/v2/peermanager/peer_manager.go

…dPeer is already specified

waku/v2/node/wakunode2_test.go

Co-authored-by: richΛrd <info@richardramos.me>

Update peer selection options for lightPush

4a18aa3

chaitanyaprem requested review from richard-ramos and harsh-98 September 29, 2023 06:52

Update peer selection options for filter

f1cfc48

chaitanyaprem mentioned this pull request Sep 29, 2023

feat: Service peer selection on specific shards #680

Closed

3 tasks

Merge branch 'master' into feat/lighproto-peer-select-options

31ed0c6

chaitanyaprem self-assigned this Oct 3, 2023

chaitanyaprem added 4 commits October 3, 2023 14:41

move to use peer selection functionality from peer manager

6aa229b

add comments

6bb3638

fix test lint issue

1f80dde

fix peer test failure

330cf92

chaitanyaprem marked this pull request as ready for review October 3, 2023 20:55

chaitanyaprem added 5 commits October 4, 2023 01:05

rename peer manager funcs

edf5a85

Merge branch 'master' into feat/lighproto-peer-select-options

a108ee7

chore: fix new return for options

7e8b51f

chore: fix log

0b6d93e

Merge branch 'master' into feat/lighproto-peer-select-options

e9fff20

richard-ramos reviewed Oct 12, 2023

View reviewed changes

chaitanyaprem mentioned this pull request Oct 12, 2023

libp2p bug: invalid p2p address returned in call to listenAddresses() when intializing libp2p host with localhost #802

Closed

chaitanyaprem added 2 commits October 12, 2023 15:49

chore: address review comments

4890984

chore: fix lint error

268c41e

chaitanyaprem mentioned this pull request Oct 12, 2023

chore: add tests to validate peer selection options in WakuNode #803

Closed

fix: address few issues with test and not doing selection if preferre…

89da415

…dPeer is already specified

chaitanyaprem requested a review from richard-ramos October 13, 2023 01:05

Merge branch 'master' into feat/lighproto-peer-select-options

7aff588

richard-ramos approved these changes Oct 15, 2023

View reviewed changes

waku/v2/node/wakunode2_test.go Outdated Show resolved Hide resolved

Update waku/v2/node/wakunode2_test.go

07dfc54

Co-authored-by: richΛrd <info@richardramos.me>

chaitanyaprem merged commit 6955d01 into master Oct 16, 2023
9 of 10 checks passed

chaitanyaprem deleted the feat/lighproto-peer-select-options branch October 16, 2023 16:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update peer selection options for light protocols #787

Update peer selection options for light protocols #787

chaitanyaprem commented Sep 29, 2023 •

edited

Loading

chaitanyaprem commented Sep 29, 2023 •

edited

Loading

status-im-auto commented Sep 29, 2023 •

edited

Loading

richard-ramos commented Oct 1, 2023 •

edited

Loading

chaitanyaprem commented Oct 3, 2023 •

edited

Loading

richard-ramos left a comment

Update peer selection options for light protocols #787

Update peer selection options for light protocols #787

Conversation

chaitanyaprem commented Sep 29, 2023 • edited Loading

Description

Changes

Tests

chaitanyaprem commented Sep 29, 2023 • edited Loading

status-im-auto commented Sep 29, 2023 • edited Loading

Jenkins Builds

richard-ramos commented Oct 1, 2023 • edited Loading

chaitanyaprem commented Oct 3, 2023 • edited Loading

richard-ramos left a comment

Choose a reason for hiding this comment

chaitanyaprem commented Sep 29, 2023 •

edited

Loading

chaitanyaprem commented Sep 29, 2023 •

edited

Loading

status-im-auto commented Sep 29, 2023 •

edited

Loading

richard-ramos commented Oct 1, 2023 •

edited

Loading

chaitanyaprem commented Oct 3, 2023 •

edited

Loading