New channel with ACQIN is not active #4575
Comments
|
If you connect to the peer manually does the channel flip to active? |
|
From our point of view, we've sent a Can you check on the lnd side whether this message was correctly received, and why no |
@t-bast thanks for time point: @Roasbeef if you look closely you can see that I got 2 fundingResponse in close proximity. The first channel is stable and active, the second (With @t-bast ) is not. Thanks |
|
I don't know how to interpret the lnd logs but roasbeef or someone else from the lnd team should be able to help. I think that what roasbeef meant was to first disconnect from the ACINQ node and then reconnect, and see if that unblocked the issue (by triggering a re-send of the lost funding messages). |
|
Thanks @t-bast I already restarted the node a couple of times, but anyway I tried what you suggest, and Thanks |
|
Interesting, I don't know what these errors internally mean to lnd...let's see what the experts say ;) |
|
From what I've seen the channel funding flow succeeded as normal. However, possibly the eclair node is rejecting the channel now? Only thing I can think of is if the channel too a while to confirm initially based on the user setting a very low fee: From there possibly eclair forgot the channel if it took too long to confirm. To summarize I don't see anything out of the ordinary here other than possibly a lack of stability between the p2p node connection. |
|
@Roasbeef I don't get your reply..? It reads to me as you are saying "That's your problem, you should have used a higher fee". I must say I don't care about if need for force-closing the channel (If this is that solution), but I feel that this can come back and bite us in the ass once people will start using the lightning network. Even if this error will happen on 0.1% of the channels or even on 0.01%, it will be a lot of dead channels.
To your points, I did use 1 sat for byte, I always use it, sometimes it can take weeks until my channels are "confirmed" I never had this problem. Shouldn't this be something in the lightning-rfc ? Look again on what @t-bast stated:
It does not look as eclair forgot the channel, it is more that lnd did not send the FundingSigned... To summarize - what do you think I should do now? Is there any improvement to lnd/ecliar/lightning that will make it so that this thing will not happen again? |
|
Sorry I mistyped here, it's normal that lnd sends no I've been digging through our logs over a longer period of time, and it seems like the issue is somehow linked to It looks like we never receive a Can you check your logs around times I suspect that this could be an issue with reconnection handling somewhere... |
|
Thanks @t-bast Interestingly at the same time there is an update to the channelpoint: and then there is a new confirmation subscription, don't know what it means. it happens to two channels at the same time. I don't know how lnd mark "ChannelReestablish" in the logs @Roasbeef Thanks |
|
So turns out this is related to #4006 and might require some larger changes to properly fix. In the short term, we're looking to see if there's anything incremental we can do in this area. |
|
Also in the future, please stick to using paste bins for logs so they don't consume so much space in issues. |
|
@Roasbeef should I close the channel and try to reopen? or is it useful for you that I leave it in this Limbo state to see if the fix you'll provide is helpful? |
|
@mrmanpew you transaction was confirmed at block 644954 2020-08-23 09:07 after 5 days of submitting according your log https://mempool.space/tx/fb2b139707a14f2c1e3b91101481ed77e19cb98d3d60c4e028282b3e8abdc11a So node probably have dropped this TX and removed channels from their database. You should follow a recovery procedure in this case. Anyway, here is a quite serious question. In order to make Lightning Network efficient one should try open channels with low on-chain fee. But since commissions in Bitcoin is fluctuating highly it is quite possible to for cheap transaction to stack in mempool up to 5-7 days untill a next weekend. @Roasbeef @t-bast
|
|
You should probably not do anything until we completely diagnose the issue. |
Agree we should increase this value. The 2 days value was set 2-ish years ago, and these days you can really only have that kind of assumption on the weekends. |
@Roasbeef @guggero what am I going to do with this 4M transaction now that my node crushed #4720, I am in the process of SCB recovery, but I still get those: from ACQIN |
|
@mrmanpew it looks like Acinq indeed removed the channel from their database. Otherwise it should respond to the DLP on your request and force close it.
@t-bast can you confirm the channel isn't in your database anymore? |
|
We're not responding to DLP because lnd incorrectly sends |
|
We have indeed removed the channel from our DB (note that it has never been used, no HTLCs were exchanged, so you should be able to get your funds back easily). |
|
Note that eclair responds with an |
|
I managed to recover the channel with @guggero help, not sure what to do with this ticket as the problem still exists, and I can not help debug is anymore. |
|
Good to know you've been able to recover the funds! I think the root cause is still there, and is a small bug in lnd: lnd sometimes sends |
|
I'm experiencing an issue that may be related to this one. I recently shut down my lnd node for the first time in at least a few weeks to update to the 0.12 rc, when I started back up I noticed that almost half my channels were inactive. This persists through restarts. The logs show a bunch of connection/channel synchronization errors, here's a snippet: If I manually attempt to establish a connection to a peer with an inactive channel (using Is this a similar problem to what's described in this issue, or should I open a new issue? I'm hoping it's not anything related to data corruption, though I did enable db auto compaction right before encountering this so that could possibly be related. I can also share more logs on pastebin or reach out to someone on slack if that would be helpful in figuring out the issue. Thank you. |
|
It's hard to tell @sangaman because your logs don't show your outgoing messages. The only thing I can conclude from your logs is that your remote closes the connection, but I need more logs to understand the reason why. Or if you experience this with the ACINQ node, I can check our logs to see if that's the same issue. |
@sangaman this could be related to #4862. Of the peers that are inactive, is there any chance of you getting logs from them? If you know whether they're running lnd/eclair/clightning would also be helpful. Could you also have a look at the |
|
Possibly related if there are channel sync issues on startup: #4799 |
I'm not sure about what the peers are running but I have a few LNBIG channels and all are inactive. Most of the inactive channels show: However, I noticed two additional inactive channels (that weren't inactive as of my last message) and those both had I tried restarting lnd to see if those would reactivate, and upon restarting it looks like one of those channels was force closed. So I'm not sure what's going on here, it may be multiple issues. Apologies if I'm hijacking this one but I didn't want to open duplicate issues. |
|
@sangaman do you have any more logs from the channel where you received |
I put the relevant log lines I could find in that issue to try to keep them separate in case I am in fact dealing with unrelated issues here simultaneously. |
|
Half my channels are still inactive, is there hope to recover them or should I force close them? I'd be happy to help debug and provide logs, and would certainly rather keep the channels, but am currently in limbo and unsure how to proceed. |
|
Closing due to inactivity, we have another issue that's tracking the root cause (funding locked vs chan reest). |
Background
I tried to open a channel with ACQIN node and for some reason the channel is inactive. I do not know why, I can ping the ip address, but the channel remains inactive.
This is the command that I used to open the channel (It is in the history of the terminal:
lncli openchannel 03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f 4500000 --sat_per_byte 1Logs
Channel info
Related logs:
You can see that the channel can not be seen at 1ml
Your environment
lnd version 0.11.0-beta commit=v0.11.0-betaLinux rock64 4.4.132-1075-rockchip-ayufan-ga83beded8524 #1 SMP Thu Jul 26 08:22:22 UTC 2018 aarch64 GNU/Linux@t-bast maybe you can also have a look from your side?
The text was updated successfully, but these errors were encountered: