-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[bug]: The problem with processing IP & Alias from a gossip protocol? #7223
Comments
Now I decided to look at the gossip protocol for updating nodes and there is such a field:
What happens when we describe the |
I see sources now. https://github.com/lightningnetwork/lnd/blob/master/netann/host_ann.go#L24
As I understand it, when the server starts, the AdvertisedIPs structure is filled with the current IP addresses, and only if they are subsequently changed - an announcement to the network occurs. Is that true? That is, if I run lnd, and the hostname itself does not resolve to different IP, then there is no announcement by gossip? I suspect that such logic can be simple with the |
My node sees the same as your |
@bitromortac Thanks! Another problem is that some wallets (SBW for example) stop working with the channels of the node that was transferred, because for some reason the idea that the nodes are tied to an IP address is being persistently promoted in the lightning network. Although it would be better if they were tied to the DNS name, not the IP, that was saved for the node. This is the Achilles' heel - the IP address and its replacement. :( |
May be @guggero or @Roasbeef will help to me? The problem is very serious. I'll try to describe it again in simple words here: when I move a node to another server, my IP changes. The port (9735) is open, there are no problems with the connection. But At lnd-02 now there is (The same result was a week ago. And this is after I have restarted this node many times on a new server and even manually submitted the update command
I do not know how to make the world know about the new IP! I assumed that the gossip protocol existed for this. But your buggy LND doesn't want to do that! |
Your cln peers can see your node announcements properly, so i guess the issue is with lnd not processing your announcement, while it is being generated and gossiped properly by your node. This is what i see of 02:
|
I agree that the problem may be that the problem is not in sending update nodes messages, but in their processing when received by the LND itself. Is your abbreviation |
Unfortunately I'm not very familiar with the p2p/announcement code of lnd. But it does look like there is something wrong with processing the announcements (apparently sending them out isn't the issue). We did fix something in that area recently: #7186 |
Ok, or I can try to make |
The PR is on top of |
Yes, i'm running core lightning. I just checked all the nodes you mentioned, i can see the new alias/ip address on all of them. Checking on amboss (as they run lnd) vs my data, lnd-33 is the only other i could find thats wrong there, but okay in my gossip data. |
Okay, you can check out the branch |
Thank you. I go to do it! Thanks! |
That PR also adds some additional logging. If you could set |
Question - I am currently doing this on lnd-25 (old lnd-02), but if the problem is on the receiving nodes of gossip, then it turns out that I need to run this version on the contrary - on all other nodes except lnd-25? |
You would need to run it on a node that doesn't have an accurate node announcement from the updated node. |
Ok, I will do it now... But I updated already the lnd-25. I hope this won't spoil the purity of the experiment? UPDATE: I stopped the lnd-25 now before it started completely... I can downgrade it for clean experiement, @guggero ? UPDATE2: I updated now the lnd-20, it saw lnd-02 as old IP & Alias. The lnd-25 (lnd-02 old) now stopped. I wait gossip updates from other nodes at lnd-20... UPDATE3: The first run of the patched lnd-20 was finished like this:
But I think it's my problem. I increased |
Yes, if you use the branch I created you will be able to downgrade to |
lnd-20 (patched) has old info about lnd-02, after
I don't see a packet info there. In that time I at lnd-25 patched (lnd-02 old) did a |
lnd-25 patched:
lnd-20 patched:
lnd-25 patched:
lnd-20 patched:
It's all... So at lnd-20 anounces only from one node: deezy.io ⚡✨ |
And right now the lnd-20 has right IP & Alias of last update of lnd-25 (patched)!
And last logs from
|
lnd-21, not patched, 0.15.5 right now:
Now I will do: will see how will be changing getnodeinfo of lnd-25 at lnd-20 again when will I do NodeAnnouncement After I will patch the lnd-21 and will see there result |
I was wondering why my node IP was not updating on 1ml, (when everything looked fine on my end), this all makes sense now, Good troubleshooting above! |
I have written to you about this a long time ago and exactly what you are asking.
2022-12-02 13:29:57.732 [DBG] DISC: Processing NodeAnnouncement: peer=024bfaf0cabe7f874fd33ebf7c6f4e5385971fc504ef3f492432e9e3ec77e1b5cf@52.1.72.207:9735, timestamp=2022-12-02 13:27:23 +0100 CET, node=03d37fca0656558de4fd86bbe490a38d84a46228e7ec1361801f54f9437a18d618 So peer != source |
As far as I understand, the root of the problem is the reason in the receiving nodes. That is, it doesn't matter which version you have when you change IP or Alias - it matters which version is on another node, on which much may depend (1ml, amboss or may be router node). |
Now, after a little thought, it seems to me that it is very important to apply the patch not to me, but so that it is applied in the new release as soon as possible (@Roasbeef ?)! It may be that by fixing myself, I will solve the problem of updating IP addresses in the graph only between my nodes and some that are connected to me. But this will not solve the problems about the main part of the network - they will still have the old IP addresses. Just because they have the current version of LND (1ml, amboss and other big nodes)! I still believe that the problem may be due to what I described here: |
No, they are the same node, both meaning the peer who relayed the message. The log needs to be updated a bit tho to make it more clear, and this is the code.
I'm trying to understand the topology here. So each node is connecting to the three start nodes right? And these star nodes are also connected to each other? What about the non-star nodes? |
Just created #7239 which aims to make sure we prioritize broadcasting our local announcements. |
In this commit, we modify our gossip broadcast logic to ensure that we always will send out our own gossip messages regardless of the filtering/feature policies of the peer. Before this commit, it was possible that when we went to broadcast an announcement, none of our peers actually had us as a syncer peer (lnd terminology). In this case, the FilterGossipMsg function wouldn't do anything, as they don't have an active timestamp filter set. When we go to them merge the syncer map, we'd add all these peers we didn't send to, meaning we would skip them when it came to broadcast time. In this commit, we now split things into two phases: we'll broadcast _our_ own announcements to all our peers, but then do the normal filtering and chunking for the announcements we got from a remote peer. Fixes lightningnetwork#6531 Fixes lightningnetwork#7223 Fixes lightningnetwork#7073
In this commit, we modify our gossip broadcast logic to ensure that we always will send out our own gossip messages regardless of the filtering/feature policies of the peer. Before this commit, it was possible that when we went to broadcast an announcement, none of our peers actually had us as a syncer peer (lnd terminology). In this case, the FilterGossipMsg function wouldn't do anything, as they don't have an active timestamp filter set. When we go to them merge the syncer map, we'd add all these peers we didn't send to, meaning we would skip them when it came to broadcast time. In this commit, we now split things into two phases: we'll broadcast _our_ own announcements to all our peers, but then do the normal filtering and chunking for the announcements we got from a remote peer. Fixes lightningnetwork#6531 Fixes lightningnetwork#7223 Fixes lightningnetwork#7073
Background
The problem with the announcement of Alias + IP addresses of LND servers after changing IP & Alias.
Most of the network does not see the changes made in LND 0.15.4-beta after the server was moved to a new IP and the Alias was changed (even after a week). At the same time, most of my other nodes do not see these changes (roughly speaking, 3/4 nodes or more). There is clearly some kind of bug.
Your environment
lnd
0.15.4-betaSteps to reproduce
Change the IP and Alias and do this at least 3-4 times with different nodes. I think at least on one you will see that other nodes of the network will not see changes about him in the graph even after a week.
Expected behaviour
Changes to IP and Alias should occur fairly quickly on other nodes of the network. Otherwise, it doesn't make sense to use Lightning.
Actual behaviour
Example - more than a week ago I moved
lnd-02
to another server, the IP address was changed and I changed the Alias. At first I thought the problem was that I changed theexternalip
option toexternalhosts
for simplification (+/etc/hosts
for this). But afterwards I tried to go back to theexternalip
option again (with certain IP address) and restarted the lnd server many times. But it didn't help, so I returned theexternhosts
option. Now I suppose it doesn't matter which of the two options is used.For example,
lnd-02
was moved to another server now it'slnd-25
. But even after a week, even my other nodes see it as (lncli getnodeinfo
):lnd-05, lnd-26:
lnd-20, lnd-21, lnd-27, lnd-28, lnd-39, lnd-40 (old IP & Alias!):
The
lnd-25
(oldlnd-02
) itself now is:The text was updated successfully, but these errors were encountered: