[P2P] P2P Module Knowledge Transfer

[Olshansk]

Objective

This is an umbrella ticket to capture references and discussion around knowledge transfer of the P2P module. The primary focus is to integrate Pocket V1 with LibP2P and answer outstanding questions in RainTree.

Origin Document

The results of the V1 P2P Research ultimately resulted in the development of a structured gossip algorithm called RainTree, presented here, and also summarized in the V1 P2P specification.

RainTree

A Python simulator was developed based on the RainTree specifications outlined here. Pocket is a Network with many nodes but not transmit large blobs of data (i.e. akin to smart contract blockchains), so whenever a new block is being committed or proposed, our goal is to distribute it throughout the network as quickly as possible. RainTree was designed to achieve this via a structured gossip algorithm with O(log3(n)) complexity. Though it was not formally proven, here were some preliminary stats collected:

This comment summarizes well the different types of broadcasts necessary in Pocket.

The current implementation of RainTree and the P2P module lives here.

Redundancy & Cleanup

While working on implementing the redundancy & cleanup layers in #368, we determined that the original specifications are not clear enough about the direction of these two features and require deeper thought before implementation. The next steps were outlined in this comment. The foundation for both is in place, but needs to be outlined & documented in more detail before implementation. The original PR should be referenced when this is undertaken, and the simulator needs to be updated appropriately as well.

LibP2P Integration

Though RainTree takes care of gossip (i.e. message routing), it does not take into account many other important parts of the P2P module, such as, but not limited to:

• Peer Discovery
• Peer Churn management
• Network layer security
• Message multiplexing
• Peer Routing

Our team conducted research on integrating LibP2P into our library and have decided that it is the right approach, with next steps summarized here.

Goals

• Capture the discussion & questions needed for support & knowledge transfer of the P2P specification in the future
• Integrate the V1 Pocket repository with LibP2P for network layer security, communication and multiplexing
• Leverage & integrate with peer discovery properties of LibP2P to scale a V1 LocalNet beyond the genesis configurations
• Determine the missing gaps and define a path forward on RainTree's redundancy & cleanup layers
• Identify an owner / Point of Contact (POC) for the V1 P2P module
• Outline & document follow-up work to build & scale the P2P module

Deliverable

• 1. [TECHDEBT] [P2P] Conntype as enum #268
• 2. [Python Simulator] Color code the terminal output rain-tree-sim#4
• 3. [P2P] Replace addressing via ServiceUrl/GenericParam with P2P routing #426
• 4. [P2P] Integrate LibP2P #347
• 5. [P2P][LibP2P][SPIKE] Consolidate and Refactor Node Identity #348
• 6. [TECHDEBT] [P2P] Raintree test suite improvements: Add configurations tests for dead and partially visible nodes #276
• 7. [P2P] RainTree Redundancy Layer Implementation #85
• Reviewing outstanding P2P Issues in the project and following up appropriately: closing, rescoping, create new followups, etc...
• Updating the appropriate documentation in the P2P module

Non-goals / Non-deliverables

• Updating/re-writing the P2P specification; though this will need to be done in the future
• Formally proving the RainTree algorithm; though it will be a really nice to have when we publish V1 docs on arxiv
• Delivering a complete & scalable P2P module; this is going to be a very fun, and non-trivial effort, that'll work on in the future
• Defining and outlining complete testing all edges cases and vulnerabilities of the P2P module; this will be necessary, but this ticket focuses on the development & documentation of an MVP rather than an alpha product

---

Creator: @Olshansk
Co-Owners: @deblasis

[Olshansk]

@bryanchriswhite Do you think its fiar to close this out?

[bryanchriswhite]

Deliverables 5-7 + the documentation deliverable are either still outstanding or contain outstanding components.

This issue doesn't provide much utility to me anymore so I would be fine with closing it so long as we track the documentation update with its own issue (if we haven't already).

[Olshansk]

Closing this out as completed because its fair to say that P2P knowledge was transferred.