Subscribe to the "mailbox topics" of close peers

[lyulka]

Background

Right now, the only peer which subscribes to a particular "mailbox topic" (the topic associated with a particular Public Address) is the replica which is identified by that Public Address.

This means that unicast messages are never sent eagerly and through full-message pairings. Instead, for a unicast message to arrive at its recipient:

1. Its metadata record needs to be gossiped to the recipient.
2. The recipient then needs to state that it wants the message ("IWANT").
3. The recipient then needs to wait until the IWANT arrives at the sending peer.
4. Finally, the recipient needs to wait for the message to finally arrive.

This makes the sending of unicast messages slow.

Reference: libp2p pubsub overview.

Proposed solution

Peers should subscribe to the individual topics of other peers that are close to them. This greatly increases the likelihood that a sending peer which is sending a unicast message has a full-message peering for the mailbox topic of the recipient, and therefore will send the message immediately.

[lyulka]

@manngayin612 will verify our understanding of this problem.

[manngayin612]

The followings are the findings from gossipsub spec.

This explains when they first join a topic, the peer will graft some peers it knows to be in the same topic. And from within, it select some of them to be in the mesh and be full-message pairing.

Edit: After Jonas' testing, it seems like its a misunderstanding from the documentation. For the first point, according to the source code, it is supposed to mean: When a peer join a topic, it will find some connected peers, that are already in the topics, to graft.

The initial construction of the mesh is random. When a peer joins a new topic, it will examine its local state to find other peers that it knows to be members of the topic. It will then select a subset of the topic members, .... These will be added to the mesh for that topic, and the newly added peers will be notified with a GRAFT control message.

As highlighted, the peers we know in a topic which is not in our mesh, will be metadata-only peers.

To allow peers to "reach beyond" their mesh view of a topic, we use gossip to propagate metadata about the message flow throughout the network. This gossip is emitted to a random subset of peers who are not in the mesh. We can think of the mesh members as "full message" peers, to whom we propagate the full content of all messages received in a topic. The remaining peers we're aware of in a topic can be considered "metadata-only" peers, to whom we emit gossip at regular intervals.

The question in my mind after reading this is:

Does the first point, (ie. grafting other peers when a peer join a toipc), also happen when a peer subscribe to a mailbox topic? If it does, isn't it what we are trying to do, (making nearby peers subscribing to our mailbox topic).

[manngayin612]

There was a question during our discussion:
The peers will keep track of what topics their peers subscribed to, but is it tracked or stored?

From libp2p::behaviour sourcecode:

Every time when a peer is subscribing/ unsubscribing to a topic, it will send a RPC to all the connected peers (peer_toipcs), telling the peers which topic its gonna subscribe/ unsubscribe.

And at the end of subscribe(), it call join(), and this is where they GRAFT the peers to the topics and create connection with them. So it seems, the peer is not the only one who subscribe to its own mailbox topic, other connected peers will be grafted to subscribe to its mailbox topics during join().

Edit: After reading the source code carefully, the above paragraph is not true. In join(), it will only graft either peers from the fanout connection of that topic or connected peers that subscribed to the same topic.