lnd becoming unusable after htlcs in a channel > default-remote-max-htlcs

[nicolasburtey]

Background

Since the last 24 hours, lnd is behaving weirdly.

One thing I have seen is that there is 30 htlc in one channel, despite having default-remote-max-htlcs=10 option set

Disk storage went from ~1Gb to 6Gb in 2 days. and the node is overly slow.

The log seems to show a log of error failure related to the channel that has more than 10 htlc.

2021-05-14 17:21:22.831 CEST
2021-05-14 15:21:22.829 [WRN] CRTR: Attempt 774716 for payment 6770f241f9288c4bf28698cd************************* failed: could not add downstream htlc
Info
2021-05-14 17:21:22.831 CEST
2021-05-14 15:21:22.829 [ERR] CRTR: Failed sending attempt 774716 for payment 6770f241f9288c4bf28698cd7********************* to switch: could not add downstream htlc
Info
2021-05-14 17:21:22.831 CEST
2021-05-14 15:21:22.829 [WRN] HSWC: ChannelLink(652799:1128:1): Unable to handle downstream add HTLC: commitment transaction exceed max htlc number
Info
2021-05-14 17:21:22.831 CEST
2021-05-14 15:21:22.829 [WRN] CRTR: Attempt 774717 for payment 92633c7aeb012d19da653********************* failed: could not add downstream htlc
Info
2021-05-14 17:21:22.831 CEST
2021-05-14 15:21:22.829 [ERR] CRTR: Failed sending attempt 774717 for payment 92633c7aeb012d19da653********************* to switch: could not add downstream htlc
Info
2021-05-14 17:21:22.831 CEST
2021-05-14 15:21:22.829 [WRN] HSWC: ChannelLink(652799:1128:1): Unable to handle downstream add HTLC: commitment transaction exceed max htlc number
Info
2021-05-14 17:21:22.822 CEST
2021-05-14 15:21:22.822 [WRN] CRTR: Routing failure for local channel 7177600************** occurred
Info
2021-05-14 17:21:22.821 CEST
2021-05-14 15:21:22.818 [ERR] CRTR: Channel update of ourselves received
Info
2021-05-14 17:21:22.821 CEST
2021-05-14 15:21:22.818 [WRN] CRTR: Routing failure for local channel 7177600************** occurred
Info
2021-05-14 17:21:22.816 CEST
2021-05-14 15:21:22.813 [ERR] CRTR: Channel update of ourselves received
Info
2021-05-14 17:21:22.803 CEST
2021-05-14 15:21:22.800 [WRN] CRTR: Attempt 774714 for payment 22354d6931ff4d4197a0c828e9282c4d4635a4a3b2533c45bc5802e44f385b7b failed: could not add downstream htlc
Info

Your environment

• version of lnd: 0.12.1
• which operating system (uname -a on *Nix): Linux lnd-0 4.19.167+ Fix name typo in README #1 SMP Tue Feb 2 07:59:55 PST 2021 x86_64 Linux
• version of btcd, bitcoind, or other backend: bitcoin 0.21
• any other relevant environment details

Steps to reproduce

Do not know how the node ends up in this situation

Actual behaviour

[nicolasburtey]

I have force closed the channel in question. I seems there are no longer issues in the log.

[Crypt-iQ]

The remote max htlcs only applies to one side's commitment transaction

[Crypt-iQ]

Disk storage increase is probably (I'm guessing) from failed payments that are stored. We might be mis-handling the remote max htlc's though so I'll look into it.

[nicolasburtey]

looking at the payment index (774714, several order of magnitude of the number payment this node is doing on a daily basis) with cpu maxing at 200% (instead of 10 or 20%) made me thing of a loop within lnd that was trying to do a payment and failing but continuously retrying. this issue seems to correlate very well with the time the channel suddenly reached 30 htlc.

[joostjager]

Maybe related to #4656 ?

[nicolasburtey]

Maybe related to #4656 ?

seems it's related indeed. the loop seems endless in this case.

[nicolasburtey]

Disk storage increase is probably (I'm guessing) from failed payments that are stored.

is there a way to clean this up (that doesn't involve creating a new node?)

[carlaKC]

is there a way to clean this up (that doesn't involve creating a new node?)

You can use the DeleteAllPayments API to clear the payments, 0.13 has the option to only delete the failed htlc attempts as well, if you want to keep the failed payment.

[nicolasburtey]

is there a way to clean this up (that doesn't involve creating a new node?)

You can use the DeleteAllPayments API to clear the payments, 0.13 has the option to only delete the failed htlc attempts as well, if you want to keep the failed payment.

ok thanks I'll get it a try once the .13 is released. by default lnd keeps forever trace of all payment, including failed one/and also failed htlc?

[carlaKC]

ok thanks I'll get it a try once the .13 is released. by default lnd keeps forever trace of all payment, including failed one/and also failed htlc?

Yeah, we keep everything by default. Failed payments and failed htlc attempts. If you want to just delete all failed payments, that's already available. 0.13 just adds the option to only delete the htlcs but keep the payments, so depends on how much you want to delete.

[nicolasburtey]

ok I'll try that. I'll use the api to do it, but why not providing this option to lncli as well?

seems this would be valuable for node operator when their db has grown too much already.