Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Websocket libp2p transport #314

Closed
lzrscg opened this issue Jun 28, 2024 · 20 comments
Closed

Websocket libp2p transport #314

lzrscg opened this issue Jun 28, 2024 · 20 comments

Comments

@lzrscg
Copy link

lzrscg commented Jun 28, 2024

Kudos to the Eiger team for the amazing work here! I'm from Modular Cloud and we want to run light nodes in the browser and on the backend using wasm.

Webtransport is the most performant choice when it is available, but it is not supported in Safari—which affects a large number of users.

Websockets seem like the sweet spot for supporting all platforms. Perhaps it could be added as a fallback option?
Screenshot 2024-06-27 at 5 18 59 PM

Let me know what you think :)

@zvolin
Copy link
Member

zvolin commented Jun 28, 2024

I think we could consider it, but so far celestia-node doesn't listen for websocket by default

@lzrscg
Copy link
Author

lzrscg commented Jun 28, 2024

Yes, great point. Ideally this would be coordinated with the Go client (although perhaps not strictly necessary?)

I will raise this with the Celestia team 🙏

@Wondertan
Copy link

Wondertan commented Jun 28, 2024

@lzrscg, Celestia Node team follows this :)

We could add WebSocket support, but I would go directly to WebRTC. The diagram says that the Golang side is in the prototype/unstable stage, but it's actually usable and required part of the work to make it work has been done and is accessible for us. It will be marked as non-experimental in the next release.

It needs to be enabled anyway, so warrants opening an issue.

@lzrscg
Copy link
Author

lzrscg commented Jun 28, 2024

Hey @Wondertan! I missed this message and went ahead with creating an issue in the celestia-node repo xD feel free to close it of course

I like the sound of WebRTC. But I am concerned that it cannot be run inside a Web Worker:
https://stackoverflow.com/questions/54208957/use-webrtc-with-webworker

This may be outdated and there may be some workarounds, but being able to easily run a node in a web worker is very important. I will look into this a bit more and get back with better informed thoughts.

@zvolin
Copy link
Member

zvolin commented Jun 28, 2024

I wasn't aware of the workers incompatibility. Yeah that pretty much kills it for us

@zvolin
Copy link
Member

zvolin commented Jun 28, 2024

so it is not listed in WebWorkers available API's (but to note, nor is WebTransport). Comparing WebTransport to RTCPeerConnection, the former has a 'note' on top that it's available in workers and the latter does not. Manual check tho would be the best

@zvolin
Copy link
Member

zvolin commented Jun 28, 2024

WebSocket also has this note

@zvolin
Copy link
Member

zvolin commented Jun 28, 2024

just to note Chrome implementation of WebTransport currently has issues and it would be good to have some fallback transport

@lzrscg
Copy link
Author

lzrscg commented Jun 28, 2024

Screenshot 2024-06-28 at 1 40 01 PM

Yeah just confirmed that it is not compatible.

index.html

<html>
  <body>
    <div>Hi</div>
    <script>
      new Worker("./worker.js");
    </script>
  </body>
</html>

worker.js

async function startWebRTC() {
    const localConnection = new RTCPeerConnection();
    const remoteConnection = new RTCPeerConnection();

    // Create a data channel on the local connection
    const sendChannel = localConnection.createDataChannel("sendChannel");

    // Set up handlers for the local connection
    sendChannel.onopen = () => {
      console.log("Data channel open");
      sendChannel.send("Hello, WebRTC!");
    };
    sendChannel.onclose = () => console.log("Data channel closed");

    // Set up handlers for the remote connection
    remoteConnection.ondatachannel = (event) => {
      const receiveChannel = event.channel;
      receiveChannel.onmessage = (event) => {
        console.log(`Received message: ${event.data}`);
      };
    };

    // ICE candidate exchange
    localConnection.onicecandidate = (event) => {
      if (event.candidate) {
        remoteConnection.addIceCandidate(event.candidate);
      }
    };

    remoteConnection.onicecandidate = (event) => {
      if (event.candidate) {
        localConnection.addIceCandidate(event.candidate);
      }
    };

    // Create offer
    const offer = await localConnection.createOffer();
    await localConnection.setLocalDescription(offer);
    await remoteConnection.setRemoteDescription(offer);

    // Create answer
    const answer = await remoteConnection.createAnswer();
    await remoteConnection.setLocalDescription(answer);
    await localConnection.setRemoteDescription(answer);
  }

  // Start the WebRTC connection
  startWebRTC();

you can run this with the command npx http-server (it won't load a web worker from a file:// path).

@Wondertan
Copy link

Then websocket it is

@zvolin
Copy link
Member

zvolin commented Jul 16, 2024

we've done some digging on this topic recently with @Wondertan. Here's quick summary.

Websockets:

  • browser requires wss protocol, using just ws is not allowed by browser policy
  • this means that only nodes sitting behind domain with valid tls certificates and keys can be dialed
  • unless self-signed certificates work, but it's very unlikely
  • moreover, go-libp2p removed support for wss protocol in Multiaddresses, replacing them with tls/ws, which then is not supported (yet) in rust-multiaddress, thus we cannot connect to go nodes using it

WebRTC:

  • doesn't work within web workers
  • we hit some issue when enabling webrtc on both lumina and celestia-node:
2024-07-11T11:28:11.975Z	WARN	webrtc-transport-pion	v3@v3.2.40/peerconnection.go:2277	Failed to start manager: handshake error: alert: Alert Fatal: CertificateUnknown
2024-07-11T11:28:11.975Z	WARN	webrtc-transport-pion	v2@v2.3.24/candidate_base.go:230	Failed to read from candidate udp4 host 0.0.0.0:2122: context canceled
2024-07-11T11:28:11.976Z	WARN	webrtc-transport-pion	v3@v3.2.40/peerconnection.go:1474	Failed to start SCTP: DTLS not established
  • there is a chance we could implement a libp2p Transport which uses webrtc on the gui thread forwarding all the packets to and from the worker, impact on gui thread would need to be checked tho

we'll try to see if we can get the reproduction of webrtc error and try to debug it more. To support websockets, we'd likely need to implement support in rust-multiaddress ourselves

@lzrscg
Copy link
Author

lzrscg commented Jul 16, 2024

Great research and thank you for the update! Here is my take:

this means that only nodes sitting behind domain with valid tls certificates and keys can be dialed

I agree this is not ideal, but realistically it seems like this is a good compromise for the browser in cases where webtransport doesn't work. However, I would say, in my humble opinion

there is a chance we could implement a libp2p Transport which uses webrtc on the gui thread forwarding all the packets to and from the worker, impact on gui thread would need to be checked tho

is not a good compromise. Some apps might be fine with this, but if we want light nodes embedded everywhere, I don't think this will be suitable :( I personally would not want ship this on the main thread, even with minimal impact

Do you see any fundamental issues with requiring nodes that want to accept wss traffic to get a cert? Aside from giving wss browser nodes less peer options?

@zvolin
Copy link
Member

zvolin commented Jul 16, 2024

Do you see any fundamental issues with requiring nodes that want to accept wss traffic to get a cert? Aside from giving wss browser nodes less peer options?

no, it will likely be just a very small fraction of the network

@oblique
Copy link
Member

oblique commented Jul 17, 2024

is not a good compromise. Some apps might be fine with this, but if we want light nodes embedded everywhere, I don't think this will be suitable :( I personally would not want ship this on the main thread, even with minimal impact

If we decide to ship something like this in lumina-node, we will make it configurable.

@Wondertan
Copy link

On WebRTC:

  • I shared the issue with @sukunrt; he was able to reproduce the problem and will debug it.

On WSS:

It can dial both /wss and /tls/ws Internally the parsing function you have linked to converts the /wss variant to /tls/ws variant.
See this dial matcher: https://github.com/libp2p/go-libp2p/blob/master/p2p/transport/websocket/websocket.go#L25

So if there is an implementation in the wild that doesn't support dialing /tls/ws but supports /wss you won't lose compatibility with go-libp2p nodes by advertising both.

  • Still not clear why we go-libp2p is so strict about /ws/tls instead of wss, as why wouldn't it support wss. From the server side, listening for connections should be identical. Potentially, there might be a way to solve this issue on a multiaddress level only.

Generally, we need to ask libp2p team to add interop tests for both of these scenarios.

@sukunrt
Copy link

sukunrt commented Jul 24, 2024

Still not clear why we go-libp2p is so strict about /ws/tls instead of wss, as why wouldn't it support wss.

It supports listening and dialing /wss. But it looks like we were too aggressive in deprecating /wss advertisements. We should advertise both /wss and /tls/ws for the time being.

But you will need to handle this yourself in the go-libp2p constructor anyway since /wss only makes sense with a CA signed certificate which in most cases is for a dns address. So even if we do advertise /wss you will have to provide the actual browser dialable address using AddrsFactory, something like /dns/abcd.xyz/tcp/42/wss

@oblique
Copy link
Member

oblique commented Jul 24, 2024

We added a convertion for /tls/ws to /wss in Lumina and it works. You don't need to do anything in the Go side.

@oblique
Copy link
Member

oblique commented Aug 1, 2024

We added support for WebSocket in Lumina. It currently works only for Arabica network.

We have 2 ongoing issues that still affect iOS support:

@oblique
Copy link
Member

oblique commented Aug 2, 2024

The issues were fixed but the restrictions for Safari were not yet removed from lumina.rs front end. For now you can use Chrome in iOS.

I'm closing this issue.

@oblique oblique closed this as completed Aug 2, 2024
@oblique
Copy link
Member

oblique commented Aug 7, 2024

Restrictions for Safari are now removed from lumina.rs front end. Note that WebSocket is only deployed in few Arabica nodes for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants