Some gossip never seems to sync in laboratory setup

[joostjager]

Background

I am experimenting with a pathfinding benchmark setup that spins up a bunch of nodes inside docker: https://github.com/bottlepay/pathfinding-benchmark/tree/lnd-6531

One essential step before starting the test is to make sure that all channels are opened and the latest channel policies gossiped to the test node that is going to make the test payments.

To detect whether the test node is fully synced, I set the default lnd base fee to 999 everywhere. What I'd want to see in the DescribeGraph output of the test node is every channel represented with policies that don't have a 999 base fee. (Unfortunately it seems impossible to override the policy directly when a channel is created with lnd, meaning that an extra channel_update is always needed.)

Your environment

• lnd v0.14.3-beta
• bitcoind

Steps to reproduce

Run run.sh

Note: make sure that there's sufficient memory, otherwise you may run into the death loop that is described here: #6210 (comment)

Expected behaviour

At some point during the setup stage, the test runner will wait for all channel policies to be received. In the used test graph, there should be 94 non-999-basefee policies. Then it proceeds with making the pathfinding test payments.

Actual behaviour

The test runner gets stuck on

testrunner_1               | 2022-05-12T14:24:12.807Z	DEBUG	Gossiped edges	{"count": 90, "expected": 94}

It keeps waiting for the last 4 (number varies) up to date policies, but isn't getting them.

To try to get propagation working, I'm using the following lnd.conf settings:

trickledelay=100
historicalsyncinterval=10s

Also the code waits for 2 minutes before changing the channel policies from the default, to avoid hitting the gossip rate limit.

Repro isn't 100%, sometimes it does get fully synced.

I've encountered propagation issues on mainnet too occasionally. Of course there could be many lnd and non-lnd related reasons for that. But perhaps this set up could help uncover a latent bug somewhere in the gossip chain.

[joostjager]

Some things that I've found out so far:

• The missing channel updates never even left the node that sets the policy. The problem isn't propagation on the network, but the update not even being broadcast.
• The reason why it isn't broadcast inAuthenticatedGossiper.sendBatch is because the active syncers don't have a an update horizon set (

  lnd/discovery/syncer.go

  Line 1250 in 7106ea5

  if g.remoteUpdateHorizon == nil {

  )
• Looking at the logs, GossipTimestampRange, which should set the horizon, is sent and received, but only later. In other cases it doesn't seem to be sent at all.

Is there a race condition with a peer being a syncer but without a horizon yet?

I am not familiar enough with this sub-system to determine what this means exactly, but perhaps it gives a clue?

[joostjager]

@saubyk is this issue worth prioritizing? There are quite a few open issues/bugs around gossip that may be hard to reproduce. But this one is easy to repro reliably.

I know from experience that this bug has a big impact. Without a patch applied, I couldn't get my node announcement to broadcast at all.

[saubyk]

@joostjager yes we are prioritizing the fix for the gossip propagation issue
#7186 is the pr I believe for this

[joostjager]

It seems that one is not the fix: #7186 (comment)

[yyforyongyu]

Nope it's not. Meanwhile what's the topology looking like? I'll try to run the experiment to see what went wrong.

[joostjager]

The topology is defined here. It are a lot of nodes, and part of the test setup is to verify that the designated sender node has received all channel updates.

For this test, I use a modified lnd version: https://github.com/bottlepay/lnd/commits/pathfinding-benchmark-mod-mpp. Docker-compose is automatically pulling in this branch and building it.

It contains three changes:

• Reduced scrypt parameters to improve memory footprint
• Reduced cache pre-allocation to improve memory footprint
• The gossip patch.

Without the patch, the sender node isn't going to get a full picture of the graph.

[Roasbeef]

Can you try out #7239 in your test bed?

[joostjager]

@Roasbeef ran it three times to be sure, and problem seems solved wtih #7239