multi: implement new query based channel graph synchronization #1106

Roasbeef · 2018-04-17T02:30:50Z

In this commit, we implement the new gossip query features recently added to the specification. With this implemented, once most peers are updated, we can skip the extremely wasteful initial routing table dump on initial connect. We also add a new command line flag that allows nodes to opt out of real time channel updates all together. This is desirable, as passive routing nodes don't really need to be receiving any of the updates, and can save bandwidth by not receiving them at all.

One follow up to this PR we might want to consider (for greater savings) is to ensure that we only engage in a single channel graph sync outstanding. This would allow use to fully sync up using a single peer, then diff our state against the next peer using the new information gained, repeating until we think we're synced. This would only apply for the first few (3 or so) peers that we connect to though.

NOTE: This PR introduces a new database migration to populate the new indexes added for existing node deployments.

`lnwire`

In this PR, we add recognition of the data loss protected feature bit. We already implement the full feature set, but then never added the bit to our set of known features. Setting this will allow all the eclair mobile nodes on the network to recover their settled channel balances in the case of partial data loss.

Additionally, we both bits for the new gossip query features along with defining all the new message types. Note that at this point, we haven't yet added the zlib short channel ID compression. We'll add that in a follow up PR to not allow this PR to swell anymore.

`discovery`

We, introduce a new struct, the gossipSyncer. The role of this struct is to encapsulate the state machine required to implement the new gossip query range feature recently added to the spec. With this change, each peer that knows of this new feature will have a new goroutine that will be managed by the gossiper.

Once created and started, the gossipSyncer will start to progress through each possible state, finally ending at the chansSynced stage. In this stage, it has synchronized state with the remote peer, and is simply awaiting any new messages from the gossiper to send directly to the peer. Each message will only be sent if the remote peer actually has a set update horizon, and the message isn't before or after that horizon. A set of unit tests has been added to ensure that two state machines properly terminate and synchronize channel state.

The gossip now has complete knowledge of the current set of peers we're connected to that support the new range queries. Upon initial connect, InitSyncState will be called by the server if the new peer understands the set of gossip queries. This will then create a new spot in the peerSyncers map for the new syncer. For each new gossip query message, we'll then attempt to dispatch the message directly to the gossip syncer. When the peer has disconnected, we then expect the server to call the PruneSyncState method which will allow us to free up the resources.

Finally, when we go to broadcast messages, we'll send the messages directly to the peers that have gossipSyncer instances active, so they can properly be filtered out. For those that don't we'll broadcast directly, ensuring we skip all peers that have an active gossip syncer.

`channeldb`

We add a series of methods, and a new database index that we'll use to implement the new discovery.ChannelGraphTimeSeries interface interface. The primary change is that we now maintain two new indexes tracking the last update time for each node, and the last update time for each edge. These two indexes allow us to implement the NodeUpdatesInHorizon and ChanUpdatesInHorizon methods. The remaining methods added simply utilize the existing database indexes to allow us to respond to any peer gossip range queries.

We also add a new database migration required to update old database to the version of the database that tracks the update index for the nodes and edge policies. The migration is straight forward, we
simply need to populate the new indexes for the all the nodes, and then all the edges.

`server`

Finally, the server has been updated to signal the new feature bits that we understand, and also to decide if we should create a new gossip syncer, or just execute a full table dump upon initial connect.

Fixes #910

halseth · 2018-04-17T10:21:00Z

lnwire/query_short_chan_ids.go

+)
+
+// ErrUnknownShortChanIDEncoding is a parametrized error that indicates that we
+// came across an unkonw short channel ID encoding, and therefore were unable


nit: s/unkonw/unknown

halseth · 2018-04-17T10:25:19Z

discovery/syncer.go

+}
+
+// synchronizeChanIDs is called by the channelGraphSyncer when we need to query
+// the remote peer for its known set of channel ID"s within a particular block


halseth · 2018-04-17T10:27:54Z

discovery/syncer_test.go

+
+	case msgs := <-msgChan:
+		if len(msgs) != 3 {
+			t.Fatalf("expected 2 messages instead got %v "+


halseth · 2018-04-17T10:36:10Z

discovery/gossiper.go

@@ -1596,7 +1754,7 @@ func (d *AuthenticatedGossiper) processNetworkAnnouncement(nMsg *networkMsg) []n
 		// We'll ignore any channel announcements that target any chain
 		// other than the set of chains we know of.
 		if !bytes.Equal(msg.ChainHash[:], d.cfg.ChainHash[:]) {
-			log.Errorf("Ignoring ChannelUpdate from "+
+			log.Error("Ignoring ChannelUpdate from "+


intended change?

Nope, reverted!

merehap · 2018-04-17T21:21:20Z

channeldb/graph.go

+func (c *ChannelGraph) ChanUpdatesInHorizon(startTime, endTime time.Time) ([]ChannelEdge, error) {
+	var edgesInHorizon []ChannelEdge
+
+	err := c.db.View(func(tx *bolt.Tx) error {


Could you break this up into helper functions? IMO, ideally most functions should fit on a single screen or else it becomes difficult to hold the whole thing in one's head at the same time. Also unit testing becomes much easier.

I realize that edgesInHorizon is updated deep down, but I think that could be propagated up through the helper functions without much overhead.

I'm imagining that this entire anonymous function could be pulled out such that you end up with something like

err := c.db.View(func(tx *bolt.Tx) error { edgesInHorizon, pErr = processEdges(tx) return pErr }

and then within processEdges (or whatever you want to call it) you'd extract the inner portion of the for loop into a processEdge function.

What do you think? If this is considered useful, there are a few other places in this PR were it applies.

I'd say yes, if the processing function were used in other locations in the package/file. Atm, they aren't. The general code style in the project is to favor a clear control flow over excessive function modularization. In this case, you can read a single function and grok the logic rather than following around several other newly introduced functions.

Not sure what screen you use for viewing code, but it fits on mine ;)

merehap · 2018-04-17T21:33:36Z

channeldb/graph.go

+
+		// We'll run through the set of chanIDs and collate only the
+		// set of channel that are unable to be found within our db.
+		var cidBytes [8]byte


This constant of "8" keeps showing up. Might want to create a constant for it to make it more clear what it is.

I started to add a constant everywhere, but IMO it started to get in the way of the control flow throughout the file. 8 is the size of an unsigned integer. In each of the locations, the variable names and comments itself are enough context to understand that 8 is the size of an unsigned integer.

merehap · 2018-04-17T21:41:26Z

channeldb/graph.go

+	if edgeBytes := edges.Get(edgeKey[:]); edgeBytes != nil {
+		// In order to delete the old entry, we'll need to obtain the
+		// *prior* update time in order to delete it.
+		updateEnd := 33 + (8 * 3) + 2 + 1


I think you definitely want a constant for this value and a comment explaining what's going on. Maybe sub-constants too.

Went with adding additional detail to the constant here. Decided against a constant as it's only used in a single location in the file atm.

merehap · 2018-04-17T22:18:32Z

discovery/syncer.go

+		// query and are waiting for the final response from the remote
+		// peer before we perform a diff to see with channels they know
+		// of that we don't.
+		case waitingQueryRangeReply:


Could you use helper functions to reduce the amount of code that is in these case statements? Basically I'm imagining that for each top-level case, there would be one helper function that would contain almost all of the code this is currently contained in the case.

This would

make channelGraphSyncer considerably shorter (easier to fit into one's head)

create a clear separation between high-level details and low-level details (which would be contained in the helper functions)

make unit testing way easier, cleaner, and more granular

decrease vertical line noise by decreasing line-wrapping for over-indentation

Note that the helper functions will probably need multiple return values to allow for the "return", "continue", and standard case endings.

All of "transition" logic is already located in distinct functions. The logic in each of the case statements simply calls out to those, and handles the redundant case in each state. As is, the entire file already have a comprehensive set of unit tests for both the transition functions, and also the control flow interaction between two syncer instances.

Generally, in the codebase, we favor a clear control flow over excessive modularization. Vertical space isn't as much of an issue following this philosophy.

merehap · 2018-04-17T22:19:07Z

discovery/syncer.go

+
+				// We'll give a few hours room in our update
+				// horizon to ensure we don't miss any newer
+				// itesm.


halseth

Nits mostly, otherwise utACK 👍

halseth · 2018-04-25T07:57:24Z

discovery/syncer.go

+// this interface to determine if we're already in sync, or need to request
+// some new information from them.
+type ChannelGraphTimeSeries interface {
+	// HighestChanID should return is the channel ID of the channel we know


halseth · 2018-04-25T07:58:08Z

discovery/syncer.go

+	// HighestChanID should return is the channel ID of the channel we know
+	// of that's furthest in the target chain. This channel will have a
+	// block height that's close to the current tip of the main chain as we
+	// know it.  We'll use this to start our QueryChannelRange dance with


halseth · 2018-04-25T07:59:15Z

discovery/syncer.go

+	// remote peer's QueryChannelRange message.
+	FilterChannelRange(chain chainhash.Hash,
+		startHeight, endHeight uint32) ([]lnwire.ShortChannelID, error)
+	// FetchChanAnns returns a full set of channel announcements as well as


missing newline

halseth · 2018-04-25T08:03:10Z

discovery/syncer.go

+}
+
+// Start starts the gossipSyncer and any goroutines that it needs to carry out
+// it duties.


halseth · 2018-04-25T08:03:25Z

discovery/syncer.go

+	log.Debugf("Starting gossipSyncer(%x)", g.peerPub[:])
+
+	g.wg.Add(1)
+


nit: extra newline

halseth · 2018-04-25T08:10:47Z

discovery/syncer.go

+		"and %v blocks after", g.peerPub[:], startHeight,
+		math.MaxUint32-startHeight)
+
+	// Finally, we'll craft the channel range query, using out starting


nit: s/out/our

halseth · 2018-04-25T08:27:23Z

discovery/gossiper.go

@@ -480,6 +501,15 @@ type msgWithSenders struct {
 	senders map[routing.Vertex]struct{}
 }

+// mergeSyncerMap is used to merge the set of senders of a particular message
+// with peers that we have an active gossipSyncer with. We do this to ensure
+// that we don't broadcast messages to any peers that


unfinished comment

halseth · 2018-04-25T08:46:53Z

chan_series.go

+		return nil, err
+	}
+	for _, channel := range chansInHorizon {
+		if channel.Info.AuthProof == nil {


add a comment to this check

halseth · 2018-04-25T08:54:37Z

config.go

@@ -202,6 +202,8 @@ type config struct {
 	Color       string `long:"color" description:"The color of the node in hex format (i.e. '#3399FF'). Used to customize node appearance in intelligence services"`
 	MinChanSize int64  `long:"minchansize" description:"The smallest channel size (in satoshis) that we should accept. Incoming channels smaller than this will be rejected"`

+	NoChanUpdates bool `long:"nochanupdates" description:"If specified, lnd will not request real-time channel updates from connected peers. This option should be used by routing nodes to save bandwidth."`


Since there might be routing nodes that wish to send active payments, I think this should instead say "nodes not not wishing to pay invoices" or similar.

Well they can still send payments, they make just incur an extra routing attempt if they have a stale channel update.

cfromknecht

Have not had a chance to test, but gave a pretty thorough review and seems solid. Like halseth, just a couple nits. Fantastic work! 🔥

cfromknecht · 2018-04-19T20:56:21Z

discovery/syncer.go

+		return "waitingQueryChanReply"
+
+	case chansSynced:
+		return "syncingChans"


s/synchingChans/chansSynced/

cfromknecht · 2018-04-26T02:28:02Z

chan_series.go

+}
+
+// FilterChannelRange returns the set of channels that we created between the
+// start height and the end height. We'll use this to to a remote peer's


to to -> to respond to

cfromknecht · 2018-04-26T02:37:10Z

chan_series.go

+		if err != nil {
+			return nil, err
+		}
+		updates = append(updates, &lnwire.NodeAnnouncement{


this section looks similar to makeNodeAnn below, is there a reason this doesn't use it? maybe has to do with alias err handling?

cfromknecht · 2018-04-26T02:57:14Z

channeldb/graph.go

+		cursor := edgeIndex.Cursor()
+
+		// We'll now iterate through the database, and find each
+		// channel ID that redoes within the specified range.


cfromknecht · 2018-04-26T03:07:45Z

channeldb/graph.go

+		// channel ID that redoes within the specified range.
+		var cid uint64
+		for k, _ := cursor.Seek(chanIDStart[:]); k != nil &&
+			bytes.Compare(k, chanIDEnd[:]) <= 0; k, _ = cursor.Next() {


Is this scan intended to be inclusive or exclusive wrt. to the end block? If inclusive, should we set max values for TxIndex and TxPosition in chanIDEnd? It seems the current behavior is exclusive, though the predicate could include the end block's coinbase txn (I think?).

cfromknecht · 2018-04-26T03:31:05Z

discovery/gossiper.go

+		},
+	})
+	copy(syncer.peerPub[:], peer.SerializeCompressed())
+	d.peerSyncers[routing.NewVertex(peer)] = syncer


Do we need to check if we already have a syncer? This is spawned in a go routine, so maybe not a bad idea

cfromknecht · 2018-04-26T03:36:48Z

discovery/syncer.go

+				}
+
+				// Otherwise, it's the remote peer performing a
+				// query, which we'll attempt to deploy to.


deploy -> reply?

In this commit, we add recognition of the data loss protected feature bit. We already implement the full feature set, but then never added the bit to our set of known features.

…y msgs

…peer In this commit, introduce a new struct, the gossipSyncer. The role of this struct is to encapsulate the state machine required to implement the new gossip query range feature recently added to the spec. With this change, each peer that knows of this new feature will have a new goroutine that will be managed by the gossiper. Once created and started, the gossipSyncer will start to progress through each possible state, finally ending at the chansSynced stage. In this stage, it has synchronized state with the remote peer, and is simply awaiting any new messages from the gossiper to send directly to the peer. Each message will only be sent if the remote peer actually has a set update horizon, and the message isn't before or after that horizon. A set of unit tests has been added to ensure that two state machines properly terminate and synchronize channel state.

In this commit, we update the logic in the AuthenticatedGossiper to ensure that can properly create, manage, and dispatch messages to any gossipSyncer instances created by the server. With this set of changes, the gossip now has complete knowledge of the current set of peers we're conneted to that support the new range queries. Upon initial connect, InitSyncState will be called by the server if the new peer understands the set of gossip queries. This will then create a new spot in the peerSyncers map for the new syncer. For each new gossip query message, we'll then attempt to dispatch the message directly to the gossip syncer. When the peer has disconnected, we then expect the server to call the PruneSyncState method which will allow us to free up the resources. Finally, when we go to broadcast messages, we'll send the messages directly to the peers that have gossipSyncer instances active, so they can properly be filtered out. For those that don't we'll broadcast directly, ensuring we skip *all* peers that have an active gossip syncer.

…lGraphTimeSeries interface In this commit, we add a series of methods, and a new database index that we'll use to implement the new discovery.ChannelGraphTimeSeries interface interface. The primary change is that we now maintain two new indexes tracking the last update time for each node, and the last update time for each edge. These two indexes allow us to implement the NodeUpdatesInHorizon and ChanUpdatesInHorizon methods. The remaining methods added simply utilize the existing database indexes to allow us to respond to any peer gossip range queries. A set of new unit tests has been added to exercise the added logic.

In this commit, we add a new database migration required to update old database to the version of the database that tracks the update index for the nodes and edge policies. The migration is straight forward, we simply need to populate the new indexes for the all the nodes, and then all the edges.

…es interface In this commit, we create a new concrete implementation for the new discovery.ChannelGraphTimeSeries interface. We also export the createChannelAnnouncement method to allow the chanSeries struct to re-use the existing code for creating wire messages from the database structs.

In this commit, we add a new command line option to allow (ideally routing nodes) to disable receiving up-to-date channel updates all together. This may be desired as it'll allow routing nodes to save on bandwidth as they don't need the channel updates to passively forward HTLCs. In the scenario that they _do_ want to update their routing policies, the first failed HTLC due to policy inconsistency will then allow the routing node to propagate the new update to potential nodes trying to route through it.

Fixes lightningnetwork#1072.

…ge update index In this commit, we ensure that all indexes for a particular channel have any relevant keys deleted once a channel is removed from the database. Before this commit, if we pruned a channel due to closing, then its entry in the channel update index would ever be removed.

This new method allows outside callers to sample the current state of the gossipSyncer in a concurrent-safe manner. In order to achieve this, we now only modify the g.state variable atomically.

In this commit, we extend the AuthenticatedGossiper to take advantage of the new query features in the case that it gets a channel update w/o first receiving the full channel announcement. If this happens, we'll attempt to find a syncer that's fully synced, and request the channel announcement from it.

Roasbeef · 2018-05-31T23:33:45Z

Just pushed out a new version rebase on top of the current master (no conflicts, yay!). Will do some local testing before merging through to master.

It seems the eclair on mobile will no longer connect to nodes that don't have the data loss protect bit on. We've supported this for some time, but never actually advertised the feature bit. A commit in this PR fixes that so connectivity will be restored once it's in.

In this commit we fix an existing bug caused by a scheduling race condition. We'll now ensure that if we get a gossip message from a peer before we create an instance for it, then we create one on the spot so we can service the message. Before this commit, we would drop the first message, and therefore never sync up with the peer at all, causing them to miss channel announcements.

halseth reviewed Apr 17, 2018

View reviewed changes

merehap reviewed Apr 17, 2018

View reviewed changes

Roasbeef requested a review from aakselrod April 23, 2018 19:49

halseth previously approved these changes Apr 25, 2018

View reviewed changes

Roasbeef dismissed halseth’s stale review via 8cd17bd April 25, 2018 22:56

Roasbeef force-pushed the new-graph-sync branch 3 times, most recently from 49b9b1e to b86e621 Compare April 26, 2018 02:28

cfromknecht suggested changes Apr 26, 2018

View reviewed changes

Roasbeef force-pushed the new-graph-sync branch 2 times, most recently from 4275976 to a9981a0 Compare April 27, 2018 03:02

Roasbeef added this to the 0.5 milestone May 2, 2018

Roasbeef added 18 commits May 31, 2018 16:30

lnwire: add recognition of the data loss proected feature bit

0fcd9ce

In this commit, we add recognition of the data loss protected feature bit. We already implement the full feature set, but then never added the bit to our set of known features.

lnwire: add new gossip query required+optional feature bits

c5065fc

lnwire: add new QueryChannelRange gossip query message

9ac8c7e

lnwire: add new ReplyChannelRange gossip query message

58470dd

lnwire: add new QueryShortChanIDs gossip query msg

898d5ff

lnwire: add new ReplyShortChanIDsEnd gossip query msg

c71d3fd

lnwire: add new GossipTimestampRange gossip query msg

fa9a012

lnwire: update tests and message code definitions for new gossip quer…

62df3cb

…y msgs

peer: dispath new gossip query messages to the gossiper

40bb19e

peer: add messages summaries for the new gossip query messages

210e28a

server: instantiate the chanSeries struct in the gossiper's config

486f875

server: set the dataloss protect and gossip queries feature bits

f70f1f3

server: if remote peer knows of gossip queries, skip initial table dump

199bbe2

Roasbeef added 6 commits May 31, 2018 16:30

server: upon disconnect, prune peer query state from gossiper

d7b2185

discovery: de-duplicate node announcements with legacy chan graph sync

55dfc17

Fixes lightningnetwork#1072.

discovery: add new SyncState method to gossipSyncer

6a6700e

This new method allows outside callers to sample the current state of the gossipSyncer in a concurrent-safe manner. In order to achieve this, we now only modify the g.state variable atomically.

Roasbeef force-pushed the new-graph-sync branch from b07023f to 994d9cf Compare May 31, 2018 23:31

Roasbeef merged commit da72f8a into lightningnetwork:master Jun 1, 2018

Roasbeef mentioned this pull request Dec 7, 2022

discovery: ensure we prioritize sending out our own local announcements #7239

Merged

		log.Debugf("Starting gossipSyncer(%x)", g.peerPub[:])

		g.wg.Add(1)

multi: implement new query based channel graph synchronization #1106

multi: implement new query based channel graph synchronization #1106

Conversation

Roasbeef commented Apr 17, 2018 • edited Loading

lnwire

discovery

channeldb

server

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfromknecht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef commented May 31, 2018

Roasbeef commented Apr 17, 2018 •

edited

Loading

`lnwire`

`discovery`

`channeldb`

`server`