WIP draft

rfcs/0000-libp2p.md

@@ -0,0 +1,149 @@
+# libp2p for coda
+
+## Summary
+[summary]: #summary
+
+Coda's networking today uses an [largely-unmaintained Kademlia library written in Haskell](https://github.com/codaprotocol/kademlia). We've been intending to replace it with some other maintained S/Kademlia implementation. We use Kademlia for peer discovery: finding IP address and ports of other protocol participants. On top of this, we build a "gossip net" for broadcasting messages to all participants with a simple flooding protocol. libp2p offers these same primitives plus some useful features like NAT traversal and connection relaying. It also has a Javascript implementation that works in the browser, which will be useful for our SDK.
+
+## Detailed Design
+[detailed-design]: #detailed-design
+
+libp2p itself is a large, flexible, extensible network stack with multiple implementations and multiple options for various functionality. I propose this minimal initial configuration:
+
+
+- On the daemon side, we will use `go-libp2p`. `rust-libp2p` lacks some features (pubsub validation) that we want.
+- For the transport, secio over TCP (optionally with WebSocket). secio is libp2p's custom transport security protocol. I believe the libp2p developers intend to replace secio with TLS 1.3 in the future. Transport security is not yet essential for us, but it's nice to have and will become more important once we start using relays.
+- libp2p multiplexes several protocols over a single transport connection. For the multiplexer, we will use the libp2p mplex protocol. We can easily add more in the future, but mplex is the only one implemented by `js-libp2p` at the moment.
+- Kademlia for peer discovery and routing.
+- "floodsub" for pubsub. More on this below.
+- A custom protocol for encapsulating Coda's Jane Street RPCs (possibly temporary).
+
+This basic configuration improves on our network stack in three major ways:
+
+1. Multiplexing over a single connection should slightly reduce our connection establishment overhead (we open a separate TCP stream per RPC right now).
+2. Transport security means connections are authenticated with the NodeID. Currently our only notion of node identity is an IP address.
+3. Browsers can join the DHT and connect to daemons.
+
+It still has some limitations: no NAT traversal, no browser↔browser connections, no message relaying. 
+
+Here's a WIP signature for the new net:
+
+```ocaml
+module type Libp2p_net = sig
+  (** Handle to all network functionality. *)
+  type t
+
+  (** Essentially a hash of a public key. *)
+  type peer_id
+
+  module Keypair : sig
+    type t
+
+    (** Securely generate a new keypair. *)
+    val random : unit -> t
+
+    val id : t -> peer_id
+  end
+
+  module Multiaddr : sig
+    type t
+
+    val to_string : t -> string
+
+    val of_string : string -> t
+  end
+
+  module Pubsub : sig
+    type topic
+
+    (** A subscription to a pubsub topic. *)
+    module Subscription : sig
+      type t
+
+      (** Publish a message to this pubsub topic.
+        *
+        * Returned deferred is resolved once the publish is enqueued. *)
+      val publish : t -> bytes -> unit Deferred.Or_error.t
+
+      (** Unsubscribe from this topic, closing the write pipe.
+        *
+        * Returned deferred is resolved once the unsubscription is complete. *)
+      val unsubscribe : t -> unit Deferred.Or_error.t
+    end
+
+    (** Get the topic handle for a string.
+      *
+      * You can think of this as hashing the string.
+      *)
+    val topic_of_string : string -> topic Deferred.Or_error.t
+
+    (** Subscribe to a pubsub topic.
+      *
+      * Fails if already subscribed. If it succeeds, incoming messages for that
+      * topic will be written to the pipe.
+      *)
+    val subscribe : t -> topic -> (bytes Envelope.Incoming.t, _, _) Strict_pipe.Writer.t -> Subscription.t Deferred.Or_error.t
+
+    (** Validate messages on a topic with [f] before forwarding them. *)
+    val register_validator : t -> topic -> f:(bytes Envelope.Incoming.t -> bool Deferred.t) -> unit
+  end
+
+  (** Create a libp2p network manager.
+    *
+    * The provided [rpcs] will be used to handle calls to [rpc_peer]>
+    *)
+  val create : me:Keypair.t -> rpcs:Host_and_port.t Rpc.Implementation.t list -> t Deferred.Or_error.t
+
+  (** List of all peers we know about. *)
+  val peers : t -> Network_peer.Peer.t list Deferred.t
+
+  (** Randomly pick a few peers from all the ones we know about. *)
+  val random_peers : t -> int -> Network_peer.Peer.t list Deferred.t
+
+  (** Perform Jane Street RPC with a given peer *)
+  val rpc_peer :
+       t
+    -> Network_peer.Peer.t
+    -> (Versioned_rpc.Connection_with_menu.t -> 'q -> 'r Deferred.Or_error.t)
+    -> 'q
+    -> 'r Deferred.Or_error.t
+
+  (** Try listening on a multiaddr.
+    *
+    * If successful, returns an alternate version of the multiaddr that other
+    * nodes can use to connect. For example, if listening on
+    * ["/ip4/127.0.0.1/tcp/0"], it might return ["/ip4/127.0.0.1/tcp/35647"]
+    * after the OS selects an available listening port.
+    *)
+  val listen_on : t -> Multiaddr.t -> Multiaddr.t Deferred.Or_error.t
+
+  (** Stop listening, close all connections and subscription pipes. *)
+  val shutdown : t -> unit Deferred.Or_error.t
+end
+```
+
+Concretely, this will be implemented by spawning a child process that speaks a simple JSON protocol. This will let us use `go-libp2p` (which seems to be the most robust libp2p implementation at the moment) without figuring out how to get Go and async working in the same process.
+
+`Pubsub` will replace the `Gossip_net` functionality. Topic names will be `{genesis state hash}-{description}`: `{hash}-states`, `{hash}-transaction-pool-diffs`, `{hash}-snark-pool-diffs` . Additional topics are easy to add. The raw pubsub messages are uninterpreted bytes; it will be up to the caller to map the pipe with useful de-serialization.  Topic message validation might be a problem. There's a default timeout of 150ms waiting to validate a message. We can turn it up, but incoming messages can come with their own timeout. There are some synchronous operations that can take longer than 150ms in our daemon (I think sparse ledger creation exceeds that right now, at least when ledger_depth=30).
+
+## Drawbacks
+[drawbacks]: #drawbacks
+
+libp2p is a pretty large dependency. It isn't super mature: only recently have the three main implementations achieved DHT interoperatbility. floodsub is, algorithmically, no better than what we have today. We'll probably need to modify it to [support pubsub payloads larger than 1MB](https://github.com/libp2p/go-libp2p-pubsub/blob/master/comm.go#L31).
+
+## Rationale and alternatives
+[rationale-and-alternatives]: #rationale-and-alternatives
+
+One easy thing to do would be just replacing the `membership` implementation with a new DHT. However, this doesn't help browsers join the DHT. There aren't many browser peer to peer networking libraries. The main one seems to be PeerJS.
+
+## Prior art
+[prior-art]: #prior-art
+
+Tezos [implements their own](https://gitlab.com/tezos/tezos/tree/master/src/lib_p2p). Cardano [implements their own](https://github.com/input-output-hk/ouroboros-network/tree/master/ouroboros-network). Ethereum 2.0 [is using `libp2p`](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/networking/messaging.md). Parity created `rust-libp2p` and is using it in substrate and polkadot. Bitcoin [implements their own](https://github.com/bitcoin/bitcoin/blob/master/src/net.cpp). Stellar [implements their own](https://github.com/stellar/stellar-core/tree/master/src/overlay). Nimiq, a "browser-based blockchain", [implements their own](https://github.com/nimiq-network/core/tree/master/src/main/generic/network).
+
+## Unresolved questions
+[unresolved-questions]: #unresolved-questions
+
+1. Effective bans. [`go-libp2p` doesn't seem to support connection refusal natively](https://github.com/libp2p/go-libp2p/issues/274). We'll probably have to build a custom transport that consults the banlist. This probably won't be a big deal.
+2. Replacing IP+port with PeerID. Especially once we have relaying (which will be essential for browsers behind NATs), the IP+port of the sender isn't super meaningful: it could be relaying messages on behalf of another party. Banning by PeerID isn't very effective, new PeerIDs can be created easily.
+