protocols/gossipsub: Support peer exchange

[mxinden]

Implement the Gossipsub Peer Exchange mechanism, see specification.

Dependencies:

• Signed Envelope and Peer Record: Added via Implement the rendezvous protocol #2107

Today the mentions of Peer Exchange in the Gossipsub implementation are empty stubs.

See also #2397 (comment).

[Frederik-Baetens]

If I understand the spec correctly, peerexchange is performed as part of pruning, but what if I want peerexchange to occur without pruning nodes to make the network more robust against node failures? Would this also be useful there or is making new connections between nodes already in the network done through another mechanism?

[Frederik-Baetens]

Also, the linked comment makes mention of a requirement for "signed peer records". Has this been implemented? Or was that done as part of the rendezvous protocol implementation?

[AgeManning]

Yep looks like signed peer records have been implemented and we can now add peer exchange to gossipsub.

PR's are welcome, but if not, we will get to this.

[AgeManning]

Could you elaborate on when you would like peer exchange to occur?

It was designed so that if you get removed from someone's mesh they give you options to join others. It probably isn't best to try and use gossipsub as a full discovery mechanism as its not designed to be one. One of the main things discovery mechanisms give you is resistance against eclipse attacks. I'd be worried if you try and use only gossipsub you could be more susceptible

[Frederik-Baetens]

PR's are welcome

Great to hear. I might be interested in contributing if it turns out that libp2p peer exchange is actually what I need. It seems like peerexchange is not meant as a mechanism to make new connections between existing nodes in the network for increasing redundancy, but rather as a mechanism to refer nodes to other nodes when during pruning. And I don't think that would help me.

After re-reading the libp2p docs, what I'm asking for would probably require additions to the libp2p spec itself, so I shouldn't really be asking these questions here. But for those still interested, here's how I'd like it to work:

Could you elaborate on when you would like peer exchange to occur?

I'd like peer exchange to occur right after a new node connects.

What occurs now:
state 1: Node A
----node B dials node A
state 2: Node A & Node B connected
----node C dials node A
state 3: Node A & Node B connected; Node A & Node C connected
----node A blows up
state end: B and C don't know about each other, leaving behind a non functioning gossipsub network

What I'd like to happen:
state 1: Node A
----node B dials node A
state 2: Node A & Node B connected
----node C dials node A
state 3: Node A & Node B connected; Node A & Node C connected
----node A informs node B and C about each others existence.
state 4: Node A & Node B connected; Node A & Node C connected; Node B & Node C connected
----node A blows up
state end: B and C do know about each other, leaving behind a perfectly functional gossipsub network

I'm not worried at all about eclipse attacks, since all nodes in my network are perfectly trusted.

I'm attempting to build a hierarchical database where metadata about which data is stored on which nodes can be shared between database nodes, so requests between nodes don't have to travel all the way up the hierarchy. Idk if that explanation makes sense, but the important bit is that I'm trying to use libp2p as peer to peer communication between fully trusted nodes.

Mostly irrelevant tangent on the additional risk of eclipse attacks (expandable)

After briefly thinking about this, I'm not certain that what I want would increase the risk of eclipse attacks. What I want just lets a node that is already trusted introduce some more nodes in case the trusted node fails. I don't immediately see how this increases the risk of eclipse attacks, since without introducing new nodes, the trusted node to which the initial connection is made is the sole (or one of few) trusted nodes, so it already "eclipses" the nodes that connected to it.

I agree that using gossipsub as your sole peer discovery mechanism probably leaves you vulnerable to eclipse attacks, but the extra redundant connections I'd like don't make it worse. (if you only use the redundant connections when the more secure/trusted peer discovery mechanism hasn't recommended enough nodes for a reliable/redundant connection)

[douglaz]

What @Frederik-Baetens described makes sense. Is this scenario part of Gossipsub Peer Exchange?

[Frederik-Baetens]

From what I understood from briefly reading libp2p docs, peerexchange is only meant to keep the network equally robust when a peer is pruned, by offering it new nodes to connect to. It doesn't do anything to make the network more robust when new nodes are joining.

[AgeManning]

After briefly thinking about this, I'm not certain that what I want would increase the risk of eclipse attacks. What I want just lets a node that is already trusted introduce some more nodes in case the trusted node fails. I don't immediately see how this increases the risk of eclipse attacks, since without introducing new nodes, the trusted node to which the initial connection is made is the sole (or one of few) trusted nodes, so it already "eclipses" the nodes that connected to it.
I agree that using gossipsub as your sole peer discovery mechanism probably leaves you vulnerable to eclipse attacks, but the extra redundant connections I'd like don't make it worse. (if you only use the redundant connections when the more secure/trusted peer discovery mechanism hasn't recommended enough nodes for a reliable/redundant connection)

I think this would modify your trust model. Say you trusted 10 peers. If one of them was malicious, then you have 1/10 malcious peers and the majority of nodes are good so you could identify the bad node. In the changes you propose, (if I understand correctly), the 1 malicious node, could then send you 100 more malicious nodes. So then you'd be connected to 101/110 malicious peers and the malicious peers would be the majority.

The way peer exchange is designed to work, is that if Node A disconnects, rather than blowing up, it informs B and C about each other. I think to make gossipsub behave more like a discovery mechanism we'd need to change the gossipsub specs and it might be worth having a discussion on the libp2p-specs repo.

[nazar-pc]

Created libp2p/specs#391 to discuss suggestion from @Frederik-Baetens

[AgeManning]

This is still not high on our priorities as I'm unsure anyone is actually using this feature.

If this feature is wanted/needed please ping me and we'll get it done sooner rather than later.

[Frederik-Baetens]

By using kademlia & identify, I no longer need peer discovery built into gossipsub. I also don't think I need do_px to work since kademlia offers my nodes more than enough peers. Although I think that do_px can be useful when the connection graph is only 1-connected, but idk how common that is in practice.

[dariusc93]

Any thoughts or plans on implementing peer exchange for gossipsub? Though its not high priority, it may be a nice thing to have and havent seen any updates regarding this.

[thomaseizinger]

Any thoughts or plans on implementing peer exchange for gossipsub? Though its not high priority, it may be a nice thing to have and havent seen any updates regarding this.

I am not aware of anyone planning to implement this at the moment.

[AgeManning]

We've been working on a big update to gossipsub, that we're calling episub.
Its still under testing at the moment, but is entirely built.

It wouldn't be hard to include peer-exchange in that. However I might keep it as a separate PR for ease of review

[dariusc93]

Thanks @AgeManning and @thomaseizinger for the update.

I have seen the specs for episub (meshsub/2.0.0) and honestly thought that was largely abandon. Nice to hear that is being worked on.

[AgeManning]

Yep. We have a rust and go version. We are doing some interop testing and attempting some large simulations to check some of the desired behaviours.

Once it looks like it is doing what we expect, I'll make a PR here to start the review process and get consensus on the spec side to see if its something we want included.

[dapplion]

@AgeManning If ok I can take this issue. Should I build off master or episub? If the latter please link the branch.

[AgeManning]

I was thinking episub may take longer than expected as we need to justify the expected performance improvements.

This could now be a standalone PR put against the master branch of rust-libp2p. i.e Ignore the existence of episub for now.

[AgeManning]

@mxinden @thomaseizinger

I've been discussin with @dapplion about the best way to get the signed peer records into gossipsub. The main source comes from the rendezvous protocol. It keeps a list of records that are still valid and have not expired.
This protocol is optional from a user standpoint, but we are trying to think of the best way to get access or to interact in a default way between the two behaviours.

i.e when gossipsub needs to prune a peer, it would be nice to have access to the list of peer-records from the rendezvous protocol (if it exists).

We can implement some messages that get passed between the two via the swarm, but I guess the user would have to implement them when composing the two behaviours. Do you guys have ideas on the best way to handle this?

[dapplion]

I've pushed a branch that consumes the signed peer records on handle_prune, and forwards the addresses to the dialer if any. master...dapplion:rust-libp2p:px-consume that would cover the client side of the feature

Should I open a PR with that first while figuring out how to seed the signed records?

[thomaseizinger]

I've been discussin with @dapplion about the best way to get the signed peer records into gossipsub. The main source comes from the rendezvous protocol.

They are also used in identify actually (at least for go-libp2p), it is just that the spec hasn't been updated yet, see libp2p/specs#347.

In rust-libp2p, we don't have a central address book / peer store (yet). I think what we'd need for this particular issue here is #2680. More specifically, a NetworkBehaviour should be able to emit a SignedPeerRecord which is then handed back to all behaviours. That would allow protocols like identify and rendevouz to emit these records. libp2p-gossipsub could store and index them by peer ID and topic and then use them to implement PX.

Does that make sense?

[AgeManning]

Yep. I guess we would also need to handle expired records also. Gossipsub would also then have to manage the TTL of records and I imagine other behaviours would also.

The ideal case would be a central address book, but I'm not sure if that is on the roadmap.

As these features are current lacking, what are you thoughts on leaving PX from gossipsub for the time being?

I dont think there is anyone that is using it right now.

[thomaseizinger]

I don't think there is any pressure in implementing it? Happy to let this rest for a bit longer until demand arises.

[dariusc93]

Yea its a nice to have feature (and would make gossipsub be more complete when comparing to other implementations in go and js) but no need for it right now

[dapplion]

Should the consumer side be implemented now? It's pretty easy given the existing from #2107 and does not require to maintain and peer records book

[AgeManning]

We could add the consumer side (if you're interested). We would have to keep another mapping inside gossipsub. I guess make a custom TTL where records expire. We probably also want to remove them if we cannot connect to them and probably also have a size limit people can't keep adding stuff to memory. Maybe an LRUTimeCache or similar data structure.
I find I keep implementing this thing because the one on crates.io is not very good.
https://github.com/sigp/lighthouse/blob/stable/common/lru_cache/src/time.rs
https://github.com/sigp/discv5/blob/master/src/lru_time_cache.rs

If we end up using it here, maybe I'll turn it into a crate.

[thomaseizinger]

I find I keep implementing this thing because the one on crates.io is not very good.

What is your concern? We use it for identify but I haven't looked into it in detail. Would be good to know if we should move away from it!

[AgeManning]

Looks like you're just using an lru cache. I think thats fine. There is an lru_time_cache which does O(n) on lookups/updates.
See here: maidsafe/lru_time_cache#143