[Concept] Scaling the P2P network

[hatmer]

The XUD network needs to scale beyond a few nodes.

The plan: Implementing a gossip protocol to replace the full-connectivity P2P network.

We are concerned with Consistency, Availability, and Partition tolerance:

Consistency: Ideally orders reach all nodes at the same time. Gossip protocols converge eventually, and with randomisation we can at least achieve fairness.

Availability: The P2P protocol should not slow down the network excessively. Using a gossip protocol is a good choice, as long as it is tuned properly to avoid excessive amounts of messages.

Partition tolerance: we want a protocol that is robust to malicious / faulty nodes that fail to forward orders. If we roll our own fault-tolerance we could end up getting wrecked by hostile traders or unexpected flaws in the protocol.

The design: Bitcoin’s stochastic address manager is a well-tested gossip algorithm for a system running in the same domain (high-volume financial services) and environment (public internet with potentially hostile nodes) as the XUD network. It works well for Bitcoin and has changed very little in the past 5 years. It has nothing to do with Bitcoin’s scalability/latency issues, those are due to the inefficiency of the PoW algorithm. The address manager is a modular part of the bitcoin daemon, i.e. it is involved in only the P2P aspects of the Bitcoin network. A clear high-level description: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2012-January/001100.html

[kilrau]

Goal is to first get all details straight before rushing into an implementation. We definitely should have a look at https://github.com/lightningnetwork/lightning-rfc, https://docs.bisq.network/ and https://docs.blocknet.co/

[hatmer]

Lightning’s protocol as a model for xud:
The Lightning code implementation (in Go) is clear and has some anti-abuse features. Like Bitcoin, Lightning uses a hybrid push/pull gossip protocol, i.e. every node can both push updates to peers and request updates from peers. Lightning only gossips between peers that it has an open channel with, so xud’s situation is not identical unless trading occurs in small “walled garden” cliques or member nodes are configured to trade with only a select group of nodes in some other way.

Attacks we need to worry about:
“An eclipse attack is when most (if not all) of your peers are malicious and they basically prevent you from being well-connected to the network to obtain information about transactions you're interested in. An eclipse attack is particular useful when a payer has sent some Bitcoins to you in some transaction, then decides to also doublespend the same Bitcoins. The doublespender (or payer) will use the eclipse attack to prevent you from knowing that there is also a doublespend transaction out in the open, so you get misled into believing that there's only the original transaction.

A sybil attack on the other hand is where a malicious actor is trying to spam the network with nodes that they control attempting to subvert the network's reputation system. For example, false signalling of support using version bits.” [https://bitcoin.stackexchange.com/questions/61151/eclipse-attack-vs-sybil-attack]

In order to prevent eclipse attacks we should use a stochastic address manager like Bitcoin’s. This is separate from the gossip protocol; it determines only which peers to gossip with, not how.

Information we want to gossip:
order updates (lots)
peer updates (fewer)

Unlike Bitcoin’s gossip protocol, which provides a lot of functionalities e.g. serving blocks, we only need to gossip of these two types of data.

The website says that xud provides “A global order book amongst exchanges combines and improves liquidity in the market.“ If we want to provide a global order book then we cannot employ a walled-garden system so that peers only gossip with a trusted set of trading peers.

Currently xud already gossips about orders and peers. This means that replacing xud’s fully connected broadcast network with a scalable alternative is simply a matter of preventing xud from updating all peers, i.e. only sending updates to the cryptographically-random bucket of peers selected for this round by the address manager. This is as simple as adding a helper function call to the relevant event emitter functions in Pool.ts and OrderBook.ts.

Fortunately, I have worked with the address manager in the past (network enumeration and security testing in 2014) so I am familiar with Bitcoin’s addrman.h. We would basically just have to port this algorithm to TypeScript and plug it in to the live system, it would not break backward compatibility with previous xud versions using the original full connectivity (those nodes would just still be sending to the entire network upon every update).

Bitcoin address manager:
"Design goals:

• Keep the address tables in-memory, and asynchronously dump the entire table to peers.dat.
• Make sure no (localized) attacker can fill the entire table with his nodes/addresses.

To that end:

• Addresses are organized into buckets.
  • Addresses that have not yet been tried go into 1024 "new" buckets.
  • Addresses of nodes that are known to be accessible go into 256 "tried" buckets.
  • Bucket selection is based on cryptographic hashing, using a randomly-generated 256-bit key, which should not be observable by adversaries.
  • Several indexes are kept for high performance. Defining DEBUG_ADDRMAN will introduce frequent (and expensive) consistency checks for the entire data structure.

The address manager also keeps track of when each peer was last heard from. Timestamps are only updated on an address and saved to the database when the timestamp is over 20 minutes old.“ [https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_4):_P2P_Network#Address_manager]

[kilrau]

Moved to opendexnetwork/opendex.network#23 . Closing here.