Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Renepay WIRE_INCORRECT_CLTV_EXPIRY #6620

Closed
Sjors opened this issue Aug 25, 2023 · 4 comments
Closed

Renepay WIRE_INCORRECT_CLTV_EXPIRY #6620

Sjors opened this issue Aug 25, 2023 · 4 comments
Assignees
@rustyrussell @Lagrang3
Labels
renepay
Milestone
v23.08 Point Rele...

Comments

@Sjors
Copy link
Contributor

Sjors commented Aug 25, 2023 •
edited
Loading

I'm not sure if WIRE_INCORRECT_CLTV_EXPIRY is perfectly normal or bad. #3862 suggests it's bad, so I figured I should open an issue.

Using v28.08 I tried to make a 41623 sat payment.

lightning-cli renepay lnbc4…

# we have computed a set of 4 flows with probability 0.000, fees 165735msat and delay 440
# disabling 7xxxxx: channel_update said it was disabled
# disabling 7xxxxx: channel_update said it was disabled
# disabling 7xxxxx: channel_update said it was disabled
# we have computed a set of 2 flows with probability 0.000, fees 191421msat and delay 440
# disabling 7xxxxx: channel_update said it was disabled
# we have computed a set of 2 flows with probability 0.000, fees 180496msat and delay 440
#   Flow 1: amount=30215000msat prob=0.997 fees=173305msat delay=74 path=-7xxxxx/1(min=max=245983423msat)->-737718x214x0/0->-7xxxxx/0->
#   Flow 2: amount=11408000msat prob=0.000 fees=7191msat delay=178 path=-7xxxxx/0(min=max=11408094msat)->-7xxxxx/0->-761331x966x1/1->
#   Flow 1: Failed at node #1 (WIRE_INCORRECT_CLTV_EXPIRY): failed: WIRE_INCORRECT_CLTV_EXPIRY (reply from remote)
# we have computed a set of 1 flows with probability 0.997, fees 173305msat and delay 440
#   Flow 3: amount=30215000msat prob=0.997 fees=173305msat delay=74 path=-7xxxxx/1(min=max=245983423msat)->-737718x214x0/0->-7xxxxx/0->
#   Flow 2: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
#   Flow 2: Failure of 11409057msat for 7xxxxx/0 capacity [0msat,1000000000msat] -> [0msat,11409056msat]
#   Flow 3: Failed at node #1 (WIRE_INCORRECT_CLTV_EXPIRY): failed: WIRE_INCORRECT_CLTV_EXPIRY (reply from remote)
# we have computed a set of 3 flows with probability 0.000, fees 181744msat and delay 440
#   Flow 4: amount=30208000msat prob=0.997 fees=173265msat delay=74 path=-7xxxxx/1(min=max=245983423msat)->-737718x214x0/0->-7xxxxx/0->
#   Flow 5: amount=576000msat prob=1.000 fees=1547msat delay=178 path=-7xxxxx/0(min=max=11415191msat)->-737352x764x1/1->-7xxxxx/0->
#   Flow 6: amount=10839000msat prob=0.050 fees=6932msat delay=178 path=-7xxxxx/0(577547msat in 1 htlcs,min=max=11415191msat)->-7xxxxx/0(max=11409056msat)->-761331x966x1/1->
#   Flow 4: Failed at node #1 (WIRE_INCORRECT_CLTV_EXPIRY): failed: WIRE_INCORRECT_CLTV_EXPIRY (reply from remote)
# we have computed a set of 1 flows with probability 0.997, fees 173265msat and delay 440
#   Flow 7: amount=30208000msat prob=0.997 fees=173265msat delay=74 path=-7xxxxx/1(min=max=245983423msat)->-737718x214x0/0->-7xxxxx/0->
#   Flow 6: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
#   Flow 6: Failure of 10840054msat for 7xxxxx/0 capacity [0msat,11409056msat] -> [0msat,10840053msat]
# we have computed a set of 1 flows with probability 0.992, fees 196msat and delay 562
# disabling 589513x1124x0: channel_update said it was disabled
# we have computed a set of 1 flows with probability 0.987, fees 196msat and delay 782
# disabling 783859x480x1: htlc above maximum
# we have computed a set of 1 flows with probability 0.977, fees 196msat and delay 584
# disabling 7xxxxx1: channel_update said it was disabled
# we have computed a set of 1 flows with probability 0.985, fees 196msat and delay 727
# disabling 7xxxxx: channel_update said it was disabled
# we have computed a set of 1 flows with probability 0.977, fees 196msat and delay 519
# disabling 7xxxxx: channel_update said it was disabled
# we have computed a set of 1 flows with probability 0.977, fees 196msat and delay 519
# disabling 775260x349x1: channel_update said it was disabled
# we have computed a set of 1 flows with probability 0.985, fees 196msat and delay 663
# disabling 7xxxxx: channel_update said it was disabled
# disabling 7xxxxx872x0: channel_update said it was disabled
# we have computed a set of 1 flows with probability 0.948, fees 196msat and delay 921
# disabling 7xxxxx395x0: channel_update said it was disabled
# disabling 7xxxxx: channel_update said it was disabled
# disabling 7xxxxx: channel_update said it was disabled
# we have computed a set of 1 flows with probability 0.969, fees 196msat and delay 767
# disabling 7xxxxx: htlc above maximum
# we have computed a set of 1 flows with probability 0.954, fees 196msat and delay 1275
# disabling 7xxxxxxxx0: htlc above maximum
# we have computed a set of 1 flows with probability 0.812, fees 196msat and delay 1215
#   Flow 8: amount=10839000msat prob=0.812 fees=196msat delay=953 path=-7xxxxx/1(min=max=17010067msat)->-7xxxxx/0->-7xxxxx/1->-7xxxxx/0->-7xxxxx/0->-7xxxxx/1->-7xxxxx/0->-7xxxxx/0->-790698x581x1/1->-7xxxxx/0->-760599x353x5/0->
#   Flow 7: Failed at node #1 (WIRE_INCORRECT_CLTV_EXPIRY): failed: WIRE_INCORRECT_CLTV_EXPIRY (reply from remote)
#   Flow 5: Failed at node #1 (WIRE_INCORRECT_CLTV_EXPIRY): failed: WIRE_INCORRECT_CLTV_EXPIRY (reply from remote)
# we have computed a set of 3 flows with probability 0.000, fees 120163msat and delay 440
#   Flow 9: amount=19361000msat prob=0.998 fees=111410msat delay=74 path=-7xxxxx/1(min=max=245983423msat)->-737718x214x0/0->-7xxxxx/0->
#   Flow 10: amount=1125000msat prob=1.000 fees=2068msat delay=178 path=-7xxxxx/0(min=max=11423479msat)->-737352x764x1/1->-7xxxxx/0->
#   Flow 11: amount=10298000msat prob=0.050 fees=6685msat delay=178 path=-7xxxxx/0(1127068msat in 1 htlcs,min=max=11423479msat)->-7xxxxx/0(max=10840053msat)->-761331x966x1/1->
#   Flow 8: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
#   Flow 8: Failure of 10839000msat for 7xxxxx/0 capacity [0msat,10000000000msat] -> [0msat,10838999msat]
# we have computed a set of 1 flows with probability 0.817, fees 196msat and delay 1556
# disabling 7xxxxx: channel_update said it was disabled
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.000002
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.000004
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.000008
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.000016
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.000032
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.000064
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.000128
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.000256
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.000512
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 2387
# delay 2387 exceeds our max 2016, so doubling delay_feefactor to 0.001024
# we have computed a set of 3 flows with probability 0.041, fees 196msat and delay 1676
#   Flow 12: amount=544000msat prob=0.990 fees=10msat delay=1414 path=-7xxxxx/1(min=max=17010067msat)->-731984x126x0/0->-730684x818x1/0->-7xxxxx/0->-7xxxxx/1->-7xxxxx/0->-771743x826x1/0->-7xxxxx/0->-796844x937x1/1->-7xxxxx/0->-7xxxxx/1->-7xxxxx/0->-7xxxxx/0->-790698x581x1/1->-7xxxxx/0->-760599x353x5/0->
#   Flow 13: amount=9751000msat prob=0.083 fees=176msat delay=953 path=-7xxxxx/1(544010msat in 1 htlcs,min=max=17010067msat)->-7xxxxx/0(max=10838999msat)->-7xxxxx/1->-7xxxxx/0->-7xxxxx/0(544000msat in 1 htlcs)->-7xxxxx/1(544000msat in 1 htlcs)->-7xxxxx/0(544000msat in 1 htlcs)->-7xxxxx/0(544000msat in 1 htlcs)->-790698x581x1/1(544000msat in 1 htlcs)->-7xxxxx/0(544000msat in 1 htlcs)->-760599x353x5/0(544000msat in 1 htlcs)->
#   Flow 14: amount=544000msat prob=0.947 fees=10msat delay=1133 path=-7xxxxx/1(10295186msat in 2 htlcs,min=max=17010067msat)->-7xxxxx/0(9751000msat in 1 htlcs,max=10838999msat)->-7xxxxx/1(9751000msat in 1 htlcs)->-7xxxxx/0(9751000msat in 1 htlcs)->-7xxxxx/0->-7xxxxx/1->-760324x644x0/0->-7xxxxx/0->-7xxxxx/1->-7xxxxx/0(10295000msat in 2 htlcs)->-790698x581x1/1(10295000msat in 2 htlcs)->-7xxxxx/0(10295000msat in 2 htlcs)->-760599x353x5/0(10295000msat in 2 htlcs)->
#   Flow 9: Failed at node #1 (WIRE_INCORRECT_CLTV_EXPIRY): failed: WIRE_INCORRECT_CLTV_EXPIRY (reply from remote)
#   Flow 11: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
#   Flow 11: Failure of 10299051msat for 7xxxxx/0 capacity [0msat,10840053msat] -> [0msat,10299050msat]
#   Flow 10: Failed at node #1 (WIRE_INCORRECT_CLTV_EXPIRY): failed: WIRE_INCORRECT_CLTV_EXPIRY (reply from remote)
# we have computed a set of 3 flows with probability 0.000, fees 120379msat and delay 440
#   Flow 15: amount=19353000msat prob=0.998 fees=111364msat delay=74 path=-7xxxxx/1(min=max=245983423msat)->-737718x214x0/0->-7xxxxx/0->
#   Flow 16: amount=1647000msat prob=1.000 fees=2564msat delay=178 path=-7xxxxx/0(min=max=11431753msat)->-737352x764x1/1->-7xxxxx/0->
#   Flow 17: amount=9784000msat prob=0.050 fees=6451msat delay=178 path=-7xxxxx/0(1649564msat in 1 htlcs,min=max=11431753msat)->-7xxxxx/0(max=10299050msat)->-761331x966x1/1->
#   Flow 14: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
#   Flow 14: Failure of 544000msat for 7xxxxx/0 capacity [0msat,10838999msat] -> [0msat,10294999msat]
#   Flow 13: Failed at node #1 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)
#   Flow 13: Failure of 9751000msat for 7xxxxx/0 capacity [0msat,10294999msat] -> [0msat,9750999msat]
#   Flow 12: Failed at node #1 (WIRE_INCORRECT_CLTV_EXPIRY): failed: WIRE_INCORRECT_CLTV_EXPIRY (reply from remote)
# we have computed a set of 3 flows with probability 0.042, fees 197msat and delay 1676
#   Flow 18: amount=1577000msat prob=0.971 fees=29msat delay=1414 path=-7xxxxx/1(min=max=17010067msat)->-731984x126x0/0->-730684x818x1/0->-7xxxxx/0->-7xxxxx/1->-7xxxxx/0->-771743x826x1/0->-7xxxxx/0->-796844x937x1/1->-7xxxxx/0->-7xxxxx/1->-7xxxxx/0->-7xxxxx/0->-790698x581x1/1->-7xxxxx/0->-760599x353x5/0->
#   Flow 19: amount=7685000msat prob=0.183 fees=139msat delay=953 path=-7xxxxx/1(1577029msat in 1 htlcs,min=max=17010067msat)->-7xxxxx/0(max=9750999msat)->-7xxxxx/1->-7xxxxx/0->-7xxxxx/0(1577000msat in 1 htlcs)->-7xxxxx/1(1577000msat in 1 htlcs)->-7xxxxx/0(1577000msat in 1 htlcs)->-7xxxxx/0(1577000msat in 1 htlcs)->-790698x581x1/1(1577000msat in 1 htlcs)->-7xxxxx/0(1577000msat in 1 htlcs)->-760599x353x5/0(1577000msat in 1 htlcs)->
#   Flow 20: amount=1577000msat prob=0.831 fees=29msat delay=1133 path=-7xxxxx/1(9262168msat in 2 htlcs,min=max=17010067msat)->-7xxxxx/0(7685000msat in 1 htlcs,max=9750999msat)->-7xxxxx/1(7685000msat in 1 htlcs)->-7xxxxx/0(7685000msat in 1 htlcs)->-7xxxxx/0->-7xxxxx/1->-760324x644x0/0->-7xxxxx/0->-7xxxxx/1->-7xxxxx/0(9262000msat in 2 htlcs)->-790698x581x1/1(9262000msat in 2 htlcs)->-7xxxxx/0(9262000msat in 2 htlcs)->-760599x353x5/0(9262000msat in 2 htlcs)->
#   Flow 15: Failed at node #1 (WIRE_INCORRECT_CLTV_EXPIRY): failed: WIRE_INCORRECT_CLTV_EXPIRY (reply from remote)
{
   "code": 210,
   "message": "Timed out"
}

Happy to send the uncensored log via email if needed.

I tried a regular pay a few minutes later which succeeded.
@Sjors
Copy link
Contributor Author

Sjors commented Aug 25, 2023

Also, I think logs with sat amounts would be more readable, e.g. fees 181.744sat instead of fees 181744msat

@cdecker
Copy link
Member

cdecker commented Aug 26, 2023

It's not bad for your funds' safety, but it is bad in terms of getting a payment through. It sounds like we either have a channel that recently changed its CLTV delta (time to take an upstream HTLC resolution, turn around and settle the downstream HTLC), or we are picking a wrong value from our local view (stale value, uninitialized value, ...). The good news is that the values are rather constrained due to what nodes will accept.

If it is a node updating their delta, then we should be seeing this from time to time and see the update propagating through the network (they should technically accept the old value for a while until the network has seen the new value exactly to prevent this kind of issue).

Let's monitor the situation and see if it happens and if we can see an update changing the value that we can correlate :-)

@rustyrussell
Copy link
Contributor

Hmm, we apply the channel_update it sent (and it did send one, since you'd see a complaint here if we didn't).

So let's assume renepay is getting it wrong, we just have to figure out how...

@rustyrussell rustyrussell self-assigned this Aug 29, 2023
@rustyrussell rustyrussell added this to the v23.08 Point Releases milestone Aug 29, 2023
rustyrussell added a commit to rustyrussell/lightning that referenced this issue Aug 29, 2023
@rustyrussell
renepay: calculate fee and delay accumulation correctly.
7d84d5b 
1. When we add a shadow amount, we were using the wrong channel for
   the fee calculation.
2. Similarly, when calculating the delay amount.

The result is that we can get WIRE_INCORRECT_CLTV_EXPIRY repeatedly
from nodes.

Reported-by: https://github.com/Sjors
Fixes: ElementsProject#6620
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changlog-Experimental: Fixed: `renepay` handles ctlv correctly when it varies along a path.
@rustyrussell
Copy link
Contributor

Thanks @Sjors .... I hope you're keeping count of how many beers/equiv I owe you for such great bug reports!

@Amperstrand Amperstrand mentioned this issue Feb 6, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rustyrussell rustyrussell

@Lagrang3 Lagrang3

Labels
renepay
Projects
None yet
Milestone
v23.08 Point Releases
Development

No branches or pull requests
4 participants
@Sjors @cdecker @rustyrussell @Lagrang3