feat: rendezvous protocol initial implementation

[vasco-santos]

This PR adds the rendezvous protocol server implementation using the latest js-libp2p.

Spec: https://github.com/libp2p/specs/tree/master/rendezvous

This PR contains the server implementations, as well as some hardening and performance considerations. Take into account that this was a full rewrite of the module, so you do not need to have into account the older code.

Github actions CI only allows services to operate in linux. Other environments are not being used

[vasco-santos]

@jacobheun this should be ready for an initial review on the structure, API and rpc parts. I also left a few questions that would like to discuss ( @vyzo would like your feedback on them as well)

[jacobheun]

Added some initial comments.

One thing I think would be valuable here is to write a separate readme on how this will work with js-libp2p, as I think that will help flush out the API here a bit better. Assume we have a single rendezvous server, a relay, and a few nodes who want to find one another on a certain namespace. libp2p should start by connecting to the rendezvous server and ask for nodes in the relay space (if they aren't dialable) so that they can add that as a listener. The nodes should also query for peers they want to connect to on the same namespace. Once a node as a dialable address (relay or otherwise), it should register on its target namespace.

We should consider what the responsibilities are internally for libp2p and what they are for the application.

• Who is responsible for querying in the first place? Does libp2p handle this, or does the application need to account for it?
• After a query is made, who is responsible for determining if we need more records? (cookie reuse)
• Who is responsible for unregistering? Does libp2p do this on shutdown?

I think the use cases may vary quite a bit so it's reasonable for application logic to handle a lot of this, but clarifying that in usage guides and showcasing how to do those things is valuable and will make sure we've provided the support to do that.

[vasco-santos]

Github actions does not support running services (mysql) when running tests on other OS... We can move into travis, but I feel we can just run it in ubuntu as I don't expect people will run a Rendezvous server in Windows/MacOS

[vasco-santos]

I kept the memory datastore previously implemented in the repo. It might be useful for small tests without spinning up MySQL, as well as to run the tests in the browser.

The browser tests are not useful as no one should run this in the browser, and I can remove them.

For example, for testing the libp2p client in libp2p core, it is easier to use the memory datastore, in order to not have to spin a MySQL db for running the tests

[vasco-santos]

libp2p/specs#302

[carsonfarmer]

How likely is this to be merged any time soon? Looking really good!

[vasco-santos]

How likely is this to be merged any time soon? Looking really good!

Thanks for reaching out. Apologies for the lack of visibility on the state of this. This is currently blocked on a couple of things:

• Message signing Rendezvous message signing specs#303
• Get benchmarking like chore: add benchmarks #8 to a more reliable scenario using testground (js sdk is in development)
• Reviews/Deployments

It will likely not happen in the near future, unless we have more specific requirements from the community that push this work. The priority for this is being tracked in this project PR: protocol/web3-dev-team#67

[vasco-santos]

As a status update on the Rendezvous Protocol, there is a new breaking change to the protocol that needs to be accounted for this PR libp2p/specs#338

[nonchip]

unless we have more specific requirements from the community that push this work

@vasco-santos well one of those requirements is the fact libp2p/js-libp2p#385 exists, the -stars are all deprecated and only work half the time anyway, this project is supposed to replace them, while js-libp2p claims to wanna work in a browser...

not sure how "web3" (:vomiting_face:) relates to this is or what other additional requirements have to be tracked to give "the only transport that actually works in a browser" priority, but actually being able to run libp2p and connect to enough nodes more than 5% of the time to even get a single pubsub message through on the same host even would be kinda nice.

[p-shahi]

@nonchip good news

the -stars are all deprecated and only work half the time anyway, this project is supposed to replace them, while js-libp2p claims to wanna work in a browser...

there are two solutions for js-libp2p to work in the browser. The first is js-libp2p-webtransport. Here's an example of using webtransport to fetch a file from Kubo https://github.com/libp2p/js-libp2p-webtransport/tree/main/examples/fetch-file-from-kubo

the other solution is native support for WebRTC. There are two parts to this. The first is browser-to-server connectivity via js-libp2p-webrtc which enables connecting to server nodes (i.e. a rust-libp2p or go-libp2p node running WebRTC). It's implemented per this specification: https://github.com/libp2p/specs/tree/master/webrtc#browser-to-public-server. This work is largely completed in js-libp2p and rust-libp2p. The go-libp2p pull request is currently in progress:

The second part of WebRTC is browser-to-browser: libp2p/js-libp2p#1462 ; work on this is in progress and we hope to deliver by the end of February. Here's the specification: libp2p/specs#497

So rest assured, there are plenty of ways of getting js-libp2p to work in the browser with support for native WebTransport and WebRTC.

[nonchip]

@p-shahi thanks!

The second part of WebRTC is browser-to-browser: libp2p/js-libp2p#1462 ; work on this is in progress and we hope to deliver by the end of February. Here's the specification: libp2p/specs#497

that's the bit i personally am relying on most because i'm building a 100% selfcontained single-file html "app" which can't expect a server ideally.

so far I'm using webrtc-star for both transport and discovery (the latter being "enhanced" by pubsub-peer-discovery since i'm heavily relying on pubsub anyway), but it's sadly very unreliable as soon as it gets "in the wild", apparently having issues with connectivity in general (dialling all kinds of invalid addresses), both gossip- and floodsub being unable to get anything through, and on top of all that apparently chrome has a bug where it just never deletes any closed webrtc connections so after like 3 minutes of seeing all kinds of things like ipfs nodes it simply refuses to let libp2p open more connections.

after weeks of careful tweaking it works like 50%ish reliably if i'm putting 4-6 nodes on the same non-public webrtc-star server, but any more or less than that, or if i just let them talk to each other for longer than like half an hour, and it still fails.

and with the various deprecations and stuff (and also the official webrtc-star server just crashing every few hours) i doubt i can improve my current codebase more with those dependencies, and the news of a better interoperable and specified implementation to come soon sounds definitely great!

---

Edit: i hope you forgive my frustration above (also @vasco-santos), and that you'll see why this project was total chaos for me, and then seeing a statement like "nah p2p in the browser won't happen for a while in js-libp2p" when it seemingly just never did anything (the 5% i mentioned weren't an understatement, took me weeks and careful tweaking of the "network size" to even be able to test it, and i can't rely on a fixed network size anyway, i wanna deploy that to users :P) was rather disheartening.