Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug]: Fee policies not broadcasting, odd behavior #7073

Closed
mcsnubbs opened this issue Oct 23, 2022 · 17 comments · Fixed by #7239
Closed

[bug]: Fee policies not broadcasting, odd behavior #7073

mcsnubbs opened this issue Oct 23, 2022 · 17 comments · Fixed by #7239
Labels
bug Unintended code behaviour gossip graph

Comments

@mcsnubbs
Copy link

I recently fired up a second LND node which is demonstrating odd fee policy behavior. I opened several channels three days ago, and have attempted to change the fee policy via lncli and thunderhub. Fee policy changes are made successfully, verified via lncli. However, over the past three days, the node has not broadcasted the fee policy changes, as reported by all LN network browsers, as well as my original node which has a channel with my second node. Oddly enough, the second node appears to be receiving gossip normally, as I can adjust channel fees from my original node, which are quickly reflected in the channel policies on the second node.

I've reviewed the lnd logs and see no errors. It appears to be updating the network graph / update as expected.

I've made fee rate changes many times over the past couple of days and attempted restarts of lnd; none of which seem to effect the issue.

Your environment

LND 15.3
Ubuntu
Bitcoind

  • any other relevant environment details

Steps to reproduce

As above

Expected behaviour

Fee policy changes should be accurately updating by peers

Actual behaviour

Second node seems to be receiving fee rate updates appropriately but may not be broadcasting its own policiies?

@mcsnubbs mcsnubbs added bug Unintended code behaviour needs triage labels Oct 23, 2022
@bitromortac
Copy link
Collaborator

I've made fee rate changes many times over the past couple of days and attempted restarts of lnd; none of which seem to effect the issue.

Your fee updates could have been rate limited, if you sent out too frequently. I suggest to pause updating fees a bit (two days or so) and then test again.

@mcsnubbs
Copy link
Author

Your fee updates could have been rate limited, if you sent out too frequently. I suggest to pause updating fees a bit (two days or so) and then test again.

I'd suggest this is not applicable as the original fee change from the opening policy in lnd.conf was allowed ~24 hours to propagate without success. Subsequent changes have been allowed minimum 6-24 hours before making another attempt.

@positiveblue
Copy link
Contributor

It looks like a rate limiting thing but:

  • Does this behavior happen with all the channels, or only with one of them?
  • How many peers does your node have?
  • Did any of your changes propagate?
  • When was the last time that you updated your fees?

@mcsnubbs
Copy link
Author

  • Does this behavior happen with all the channels, or only with one of them?

All channels. Only one rate change was propagated from the node. Currently the node has a channel with my original node, and two others. After channel opening I was able to change fees once on the channel with my original node. However, no changes have since been broadcast for that channel. The other two channels have never appropriately updated fee policy changes.

  • How many peers does your node have?

Currently 7. Three are peers with channels.

  • Did any of your changes propagate?

As above, only one.

  • When was the last time that you updated your fees?

I've attempted again this morning; on average every 24 hours. This morning I switched to tor only, assuming my hydrib/VPN setup could be causing an issue, however, it does not seem to have affected the issue. I'd imagine my hybrid/VPN setup is not to blame as I've been able to successfully route payments via these channels, and the node is appropriately seeing fee policy changes from its peers with which it has active channels

@positiveblue
Copy link
Contributor

Yes, I also doubt is related to your hydrib/VPN setup.

You coud try:

  • Asking some other operator in the network (better if he is connected to your primary node) what fees does he see.
  • Updating the fee from your primary node, does that fee get propagated?
  • Updating the fee from you secondary node with another channel, do LN Browsers reflect that fee?

@mcsnubbs
Copy link
Author

I appreciate the ideas!

https://amboss.space/edge/759586x1943x2
This is one of the channels that has not updated fees... It still thinks I'm charging 3000ppm, which is specified on channel open via lnd.conf. The fees seem consistent among several LN browsers...

YES! Oddly enough, when I change fees from my primary node, it is seen by the secondary node. Here is the channel between the two: https://amboss.space/edge/759589x2173x1 ; As you can see, I was able to change fees ONCE on this channel with the secondary node, from 3000 > 1000 ppm; no others have been broadcast. Worth clarifying; the other two channels that the secondary node has with Moon and Boltz havent broadcast a single rate change.

LN browsers have not reflected the fee changes, as you can see from links provided

@mcsnubbs
Copy link
Author

Oddly enough they finally updated... Only took 3 days. Any other ideas why I could have run into the delay?

@positiveblue
Copy link
Contributor

My vote goes for rate limiting. Hard to say if it was something else without further investigation. I will close the issue, let us know if it happens again 👍

@joostjager
Copy link
Contributor

I am running into similar issues on testnet. Very hard to get policy updates propagated to explorers. Also with two independent nodes connected to the same external node (opennode in my case, I believe also running lnd), the update doesn't reach the other node.

I understand rate limiting, but if only lnd nodes are involved, isn't that supposed to be only for a minute?

@mcsnubbs
Copy link
Author

I am running into similar issues on testnet. Very hard to get policy updates propagated to explorers. Also with two independent nodes connected to the same external node (opennode in my case, I believe also running lnd), the update doesn't reach the other node.

I understand rate limiting, but if only lnd nodes are involved, isn't that supposed to be only for a minute?

It is definitely not rate limiting... It took 3 days to change from the original fee policy set on channel opening.

@joostjager
Copy link
Contributor

I have a feeling that there is a bug somewhere in the gossip code. In a different context, I did some investigation previously: #6531 (comment)

@joostjager
Copy link
Contributor

joostjager commented Oct 31, 2022

Can confirm that after applying bottlepay@41564ff, my updates propagate to mempool.space promptly. So definitely not rate limiting indeed.

@guggero guggero reopened this Oct 31, 2022
@Roasbeef
Copy link
Member

Roasbeef commented Oct 31, 2022

@joostjager how many peers does your node have? Have you modified any of the default gossip config params?

AFAICT, all your diff above does it always send the gossip messages to peers that are your active syncers. Otherwise, we'll treat them only as source of new inbound updates and use the other peers to send out outbound updates.

EDIT: Looking at the diff again, I wonder if there's a variable loop binding bug there, so a lack of:

msgChunk := msgChunk

at the very top of the loop.

@joostjager
Copy link
Contributor

At the moment it has 3 peers in ACTIVE_SYNC, but I remember that I tried adding more peers when I couldn't get my updates propagated.

I've seen the same problem in the controlled environment mentioned above. It's also reproducible there. Just need to update the docker lnd rev to a branch that doesn't include the patch.

@joostjager
Copy link
Contributor

@Roasbeef the analysis in the link above (horizon not set), does that make any sense?

@Roasbeef
Copy link
Member

Roasbeef commented Dec 7, 2022

See #7239

Roasbeef added a commit to Roasbeef/lnd that referenced this issue Dec 7, 2022
In this commit, we modify our gossip broadcast logic to ensure that we
always will send out our own gossip messages regardless of the
filtering/feature policies of the peer.

Before this commit, it was possible that when we went to broadcast an
announcement, none of our peers actually had us as a syncer peer (lnd
terminology). In this case, the FilterGossipMsg function wouldn't do
anything, as they don't have an active timestamp filter set. When we go
to them merge the syncer map, we'd add all these peers we didn't send
to, meaning we would skip them when it came to broadcast time.

In this commit, we now split things into two phases: we'll broadcast
_our_ own announcements to all our peers, but then do the normal
filtering and chunking for the announcements we got from a remote peer.

Fixes lightningnetwork#6531
Fixes lightningnetwork#7223
Fixes lightningnetwork#7073
@nullcount
Copy link

@mcsnubbs

I'd imagine my hybrid/VPN setup is not to blame as I've been able to successfully route payments via these channels

Just wanted to add that I also started experiencing this issue shortly after switching to hybrid mode using tunnelsats

Roasbeef added a commit to Roasbeef/lnd that referenced this issue Dec 15, 2022
In this commit, we modify our gossip broadcast logic to ensure that we
always will send out our own gossip messages regardless of the
filtering/feature policies of the peer.

Before this commit, it was possible that when we went to broadcast an
announcement, none of our peers actually had us as a syncer peer (lnd
terminology). In this case, the FilterGossipMsg function wouldn't do
anything, as they don't have an active timestamp filter set. When we go
to them merge the syncer map, we'd add all these peers we didn't send
to, meaning we would skip them when it came to broadcast time.

In this commit, we now split things into two phases: we'll broadcast
_our_ own announcements to all our peers, but then do the normal
filtering and chunking for the announcements we got from a remote peer.

Fixes lightningnetwork#6531
Fixes lightningnetwork#7223
Fixes lightningnetwork#7073
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Unintended code behaviour gossip graph
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants