feat: Added Peer-to-Peer networking data synchronization #177

jsimnz · 2022-02-04T08:19:16Z

This adds the first iteration of the peer-to-peer (p2p) networking stack powered by LibP2P. Additionally this starts the first iteration of the Peer API which is a gRPC API exposed over a pubsub network. Closes #69

Architecture

This creates 3 new main structures net/server, net/peer, and node/node.

net/server is the gRPC server which publishes and subcribes to RPC messages on the p2p gossibsub network.
net/peer encapsulates the net/server struct and handles DAG resolution and maintains an internal broadcast channel to synchronize the rest of the DB system.
node/node combines both the net/peer and db/db object into a unified DefraDB Node.
Embedded ipfslite host as a "DAGSyncer" as a DAG network getter/setter (currently powered by the LibP2P DHT and Bitswap protocol).

Currently needs more formal automated testing, which is notoriously difficult to test this kind of networking system. More to come.

At the moment this only syncs documents after nodes have both subscribed to the same pubsub topic. The Replicator RPC command allows nodes to pre-emptively select which nodes to push/sync documents to.

Documents are synced as their respective MerkleCRDT update graphs and ipld.Nodes.

Todo:

Add Replicator options to automatically sync new documents to a given peer

… libp2p protocol definition

…Factory

… node Options system for configuring node behavior

…g type in their publish functions. Only MerkleCompositeDAG publish their updates to the pubsub net

…d protobuf definitions. Updated core/net/Broadcaster type Implemeneted new Broadcaster type on net/peer Breaking: Updated MerkleClock interface to have AddDAGNode return the newly created node. Updated protobuf definitions for net/pb

… peer boostrapping, updated HTTP API to return errors, fixed pubsub push mechanics

… publisher now properly handling broadcast updates, updated collection create method to handle pubsub registering

…chemaID to CompositeDAGDelta objects. Update Merkle factory to include additional schema id parameter

Replicator allows a node to target another node to directly sync changes with. This is in comparison to the general pubsub network. Replicator is 'active' sync, while pubsub is 'passive' sync. Additionally, added a second tcp gRPC server for clients to interact with, and added a gRPC client utility for making RPC requests.

AndrewSisley · 2022-02-07T14:13:25Z

cli/defradb/cmd/start.go

+		var bs *broadcast.Broadcaster
+		if !config.Net.P2PDisabled {
+			bs := broadcast.NewBroadcaster(busBufferSize)
+			options = append(options, db.WithBroadcaster(bs))


:) func. options pattern makes more sense here, is nice

AndrewSisley · 2022-02-07T14:13:53Z

cli/defradb/cmd/start.go

-		if len(peers) != 0 {
-			log.Debug("Parsing boostrap peers: ", peers)
-			addrs, err := netutils.ParsePeers(strings.Split(peers, ","))
+		if !config.Net.P2PDisabled {


AndrewSisley · 2022-02-07T14:18:52Z

net/peer.go

+			// 2 is update log
+			var err error
+			if msg.Priority == 1 {
+				err = p.handleDocCreateLog(msg)


Nice cheers - this makes it quite a bit easier to tell what's what IMO, and it hosts the code in the same place

AndrewSisley

Looks good :D Looking forward to having it in and playing around with it (and figuring how to int. test it lol)

AndrewSisley · 2022-02-07T20:25:35Z

net/api/client/client.go

+
+// AddReplicator sends a request to add a target replicator to the DB peer
+func (c *Client) AddReplicator(ctx context.Context, collection string, paddr ma.Multiaddr) (peer.ID, error) {
+	var pid peer.ID


Minor, butthis var is only used when default (on err) - suggest declaring it only in those scopes

AndrewSisley · 2022-02-07T20:32:59Z

net/peer.go

+
+	// make sure were not duplicating things
+	p.mu.Lock()
+	defer p.mu.Unlock()


(Minor as it looks like this lock is not really performance sensitive, but this also impacts maintenance) It is very unclear as to how long this lock actually needs to be held - it it simple to unlock earlier? I cant tell if it is only required for the next block or so, or the whole func.

It can be moved down to line 271 after p.replicators[col.SchemaID()][pid] = struct{}{} but because there are exit cases between the Lock and that line, id rather just do it on defer. And its not like we're going to be spamming the replicator command. I find defer on a lock the most safe as it gurantees you dont accidentally deadlock yourself.

AndrewSisley · 2022-02-07T20:36:12Z

cli/defradb/cmd/start.go

+			}
+
+			server := grpc.NewServer(grpc.KeepaliveParams(keepalive.ServerParameters{
+				MaxConnectionIdle: 5 * time.Minute, // <--- This fixes it!


Minor, but suggest comment removal

AndrewSisley · 2022-02-07T20:37:42Z

cli/defradb/cmd/start.go

+			go func() {
+				log.Info("Started gRPC server, listening on %s", addr)
+				netpb.RegisterServiceServer(server, netService)
+				if err := server.Serve(tcplistener); err != nil && !errors.Is(err, grpc.ErrServerStopped) {


Quick sanity check, it is safe to not copy the tcpListener here?

yea thats fine, go will handle the copy behind the scenes, its really only an issue in loops if youre overwriting a variable.

ah nice, thanks :)

AndrewSisley · 2022-02-07T20:38:55Z

net/peer.go

+		ds:     ds,
+		bus:    bs,
+		p2pRPC: grpc.NewServer(serverOptions...),
+		// tcpRPC:         grpc.NewServer(),


Suggest removal

AndrewSisley · 2022-02-07T20:39:03Z

net/peer.go

-	if err != nil {
-		return err
-	}
+	// addr, err := utils.TCPAddrFromMultiAddr(p.tcpAddr)


Suggest removal

AndrewSisley · 2022-02-07T20:39:14Z

net/peer.go

-			log.Fatal("Fatal tcp rpc serve error:", err)
-		}
-	}()
+	// go func() {


Suggest removal

This adds the first iteration of the peer-to-peer (p2p) networking stack powered by LibP2P. Additionally this starts the first iteration of the Peer API which is a gRPC API exposed over a pubsub network. * initial peer layout * outline protobuf net package * Added basic libp2p protobuf custom types * scaffold peer/server types * Added libp2p/gostream rpc dialers * initial implementation of the p2p/net server object * implemented NewPeer with grpc serve/listen over libp2p * Added Broadcaster definition and linked into DB, MerkleCRDT, and CRDTFactory * Updated db implementations to match update core/client interfaces. Added some utility to base descriptions * BREAKING: Changed how collections are stored internally, added namespace to collection key * Implemented new client interface for AllCollections and AllDocKeys * Re-architect MerkleDAG Broadcast and DB/Peer relationship. * Add Replicator system for p2p node. * Added rpc client to CLI and hooked up AddReplicator command

…rk#177) This adds the first iteration of the peer-to-peer (p2p) networking stack powered by LibP2P. Additionally this starts the first iteration of the Peer API which is a gRPC API exposed over a pubsub network. * initial peer layout * outline protobuf net package * Added basic libp2p protobuf custom types * scaffold peer/server types * Added libp2p/gostream rpc dialers * initial implementation of the p2p/net server object * implemented NewPeer with grpc serve/listen over libp2p * Added Broadcaster definition and linked into DB, MerkleCRDT, and CRDTFactory * Updated db implementations to match update core/client interfaces. Added some utility to base descriptions * BREAKING: Changed how collections are stored internally, added namespace to collection key * Implemented new client interface for AllCollections and AllDocKeys * Re-architect MerkleDAG Broadcast and DB/Peer relationship. * Add Replicator system for p2p node. * Added rpc client to CLI and hooked up AddReplicator command

jsimnz added 16 commits July 9, 2021 17:22

peer stuff

4216430

moved net package (forgot new files xD)

241990b

outline protobuf net package

c3946ee

Added basic libp2p protobuf custom types

1603294

Added doc to net package with high high high level description. Added…

466a718

… libp2p protocol definition

scaffold peer/server types

e344f68

more outline/scaffolding. Need a design doc

b8089cc

renamed proto service to match server object

09ffdf2

Expanded protobuf definitions, added maked file for protobuf

099665b

Updated net libp2p protocol definition

fa52365

Updated server/peer objects

62457c0

Added libp2p/gostream rpc dialers

f0aaf97

Updated p2p protocolbuffer definitions

7ae19f8

rewrote libp2p/peer import to prevent name clashing

d6fc5db

initial implementation of the p2p/net server object

ee09691

implemented NewPeer with grpc serve/listen over libp2p

266faf0

jsimnz added area/network Related to internal/external network components area/p2p Related to the p2p networking system feature New feature or request labels Feb 4, 2022

jsimnz added this to the DefraDB v0.2 milestone Feb 4, 2022

jsimnz requested a review from AndrewSisley February 4, 2022 09:00

jsimnz added 9 commits February 4, 2022 04:36

Fixed issues from rebase, lots of context stuff

14e6dce

Added Broadcaster definition and linked into DB, MerkleCRDT, and CRDT…

2fe8830

…Factory

Wired in Broadcaster object into the DB and CRDT Factory. Created the…

9920bd6

… node Options system for configuring node behavior

Added core.Log type, updated MerkleCRDT implementations to use new lo…

03473aa

…g type in their publish functions. Only MerkleCompositeDAG publish their updates to the pubsub net

Wired in new doc registers on peer, added peer reference to DB, added…

4bdaa71

… peer boostrapping, updated HTTP API to return errors, fixed pubsub push mechanics

Updated CompositeDAG to include broadcaster on init, base merkle crdt…

faa5f46

… publisher now properly handling broadcast updates, updated collection create method to handle pubsub registering

Fixed custom protobuffer DocKey marshalling

226e1ae

Added cidV1 util. Implemented Collection index via schema ID. Added S…

f64bdf7

…chemaID to CompositeDAGDelta objects. Update Merkle factory to include additional schema id parameter

jsimnz added 9 commits February 6, 2022 04:11

Fixed compilation + tests after rearchitect

18c7b3c

Fixed accidental re-instance of variable

aab347e

Changed no-p2p flag name

43c30d3

Removed old Publish func arg

7e99b30

Fixed job reference wrong field(s)

99a4a21

Fixed linter issues

ac61987

Renamed net/server mutex

aa6c1bc

Added rpc client to CLI and hooked up AddReplicator command

7aa53bf

AndrewSisley reviewed Feb 7, 2022

View reviewed changes

AndrewSisley self-requested a review February 7, 2022 14:19

AndrewSisley marked this pull request as ready for review February 7, 2022 14:19

AndrewSisley approved these changes Feb 7, 2022

View reviewed changes

jsimnz added 2 commits February 7, 2022 15:18

Fixed replicator RPC issue, moved RPC server from /net/peer to /net/api

594d362

Removed old debug statements

0a3aedc

AndrewSisley reviewed Feb 7, 2022

View reviewed changes

Fixed linter issues

0bbae9a

AndrewSisley reviewed Feb 7, 2022

View reviewed changes

Addressed review comments

357fb23

jsimnz merged commit 4281ac9 into develop Feb 7, 2022

jsimnz deleted the jsimnz/feat/p2p-sync-clean-history branch February 7, 2022 20:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Added Peer-to-Peer networking data synchronization #177

feat: Added Peer-to-Peer networking data synchronization #177

jsimnz commented Feb 4, 2022 •

edited

Loading

AndrewSisley Feb 7, 2022

AndrewSisley Feb 7, 2022

AndrewSisley Feb 7, 2022

AndrewSisley left a comment

AndrewSisley Feb 7, 2022

jsimnz Feb 7, 2022

AndrewSisley Feb 7, 2022

jsimnz Feb 7, 2022

AndrewSisley Feb 7, 2022

jsimnz Feb 7, 2022

AndrewSisley Feb 7, 2022

jsimnz Feb 7, 2022

AndrewSisley Feb 7, 2022

AndrewSisley Feb 7, 2022

AndrewSisley Feb 7, 2022

jsimnz Feb 7, 2022

AndrewSisley Feb 7, 2022

jsimnz Feb 7, 2022

feat: Added Peer-to-Peer networking data synchronization #177

feat: Added Peer-to-Peer networking data synchronization #177

Conversation

jsimnz commented Feb 4, 2022 • edited Loading

Architecture

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrewSisley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsimnz commented Feb 4, 2022 •

edited

Loading