No routing through LND

[evd0kim]

Issue and Steps to Reproduce

Given:
-- LND node, clightning (CLG later) node.
-- CLG has at least one channel with LND. The channel is "balanced". I mean the payment is so small that there is enough capacity anyway.
-- single hop payments processed unilaterally in both ends, LND and CLG

Problem:
-- multihop payments aren't processed in CLG->LND directions but the CLG<-LND is okay.

Very common error:
{ "code" : 205, "message" : "Could not find a route" }

Problem is reproducible in any situation involving LND-CLG interaction, channel any size, fresh or old, I mean also long-living channels which have been opened during several updates on both ends of the channel.

[evd0kim]

Disclaimer: I'm an LND administrator and have no CLG node accessible (now). So the whole situation was reproduced thanks to @fiatjaf

Model situation.

1. I generated invoice in WalletOfSatoshis. It had receiver:
   { "node": { "last_update": 1556645135, "pub_key": "026c7d28784791a4b31a64eb34d9ab01552055b795919165e6ae886de637632efb", "alias": "LivingRoomOfSatoshi.com_LND_1", "addresses": [ { "network": "tcp", "addr": "13.75.179.99:9735" } ], "color": "#3399ff" }, "num_channels": 216, "total_capacity": "847350894" }

2. My node has direct channels with node from which I tried to send payment and LivingRoomOfSatoshi.com_LND_1. So there would be not more than 2 hop.

3. Sender node (CLG) failed to send first payment and I had success only next time. Logs from clightning node: http://sprunge.us/RLVF88 .As you can see from logs first payment failed. By the way what means WIRE_FEE_INSUFFICIENT here?

4. In step 2 I mentioned my node which have one channel with enough liquidity. It is not 03fb822818be083e0a954db85257a2911a3d55458b8c1ea4124b157e865a836d12 which is in logs in failed payment. But it is LNBIG node which has LND client onboard. The situation is very similar to that if payment would go through my LND node.

I can't figure out what really happens. Mostly clightning client cant find a route even if next node is well connected (more than 40 active channels, 100-200 in rank) for a payments really fitting well to capacity of the channel.

[evd0kim]

Clightning lispeers output concerning my (LND) node:

         "channels" : [
            {
               "state" : "CHANNELD_NORMAL",
               "scratch_txid" : "9ac240ba9a3be3b284fa4bde3cc508bc8464740298afdd646b318f1ac3a81e9a",
               "owner" : "lightning_channeld",
               "short_channel_id" : "573762x1393x1",
               "direction" : 0,
               "channel_id" : "af1030c6c0bc0419bc482635bd8f73f3671c6a4d0c15f61f437ea70121c91cdb",
               "funding_txid" : "da1cc92101a77e431ff6150c4d6a1c67f3738fbd352648bc1904bcc0c63010af",
               "private" : false,
               "funding_allocation_msat" : {
                  "02a5340317463d36d6d2ff9c8fd2e4fad87e4a948ea34dcefdb8e9518bffe2975f" : 0,
                  "02a54deb8d0f11d47c6f55cec5e673063c9fad2619559e8d87ae3eb4c381668449" : 1000000000
               },
               "funding_msat" : {
                  "02a5340317463d36d6d2ff9c8fd2e4fad87e4a948ea34dcefdb8e9518bffe2975f" : "0msat",
                  "02a54deb8d0f11d47c6f55cec5e673063c9fad2619559e8d87ae3eb4c381668449" : "1000000000msat"
               },
               "msatoshi_to_us" : 103461000,
               "to_us_msat" : "103461000msat",
               "msatoshi_to_us_min" : 0,
               "min_to_us_msat" : "0msat",
               "msatoshi_to_us_max" : 503467000,
               "max_to_us_msat" : "503467000msat",
               "msatoshi_total" : 1000000000,
               "total_msat" : "1000000000msat",
               "dust_limit_satoshis" : 546,
               "dust_limit_msat" : "546000msat",
               "max_htlc_value_in_flight_msat" : 18446744073709551615,
               "max_total_htlc_in_msat" : "18446744073709551615msat",
               "their_channel_reserve_satoshis" : 10000,
               "their_reserve_msat" : "10000000msat",
               "our_channel_reserve_satoshis" : 10000,
               "our_reserve_msat" : "10000000msat",
               "spendable_msatoshi" : 93461000,
               "spendable_msat" : "93461000msat",
               "htlc_minimum_msat" : 0,
               "minimum_htlc_in_msat" : "0msat",
               "their_to_self_delay" : 144,
               "our_to_self_delay" : 144,
               "max_accepted_htlcs" : 483,
               "status" : [
                  "CHANNELD_NORMAL:Reconnected, and reestablished.",
                  "CHANNELD_NORMAL:Funding transaction locked. Waiting for their announcement signatures."
               ],
               "in_payments_offered" : 20,
               "in_msatoshi_offered" : 503637000,
               "in_offered_msat" : "503637000msat",
               "in_payments_fulfilled" : 18,
               "in_msatoshi_fulfilled" : 503482000,
               "in_fulfilled_msat" : "503482000msat",
               "out_payments_offered" : 87,
               "out_msatoshi_offered" : 401027935,
               "out_offered_msat" : "401027935msat",
               "out_payments_fulfilled" : 2,
               "out_msatoshi_fulfilled" : 400021000,
               "out_fulfilled_msat" : "400021000msat",
               "htlcs" : []
            }
         ]

As you can see, there are significant spendable amounts. However, 10 sat payment cant be forwarder through the channel.

[SimonVrouwe]

Not sure about this, but maybe this has to do with this status?
CHANNELD_NORMAL:Funding transaction locked. Waiting for their announcement signatures.
same as #1778, see also #2409

[cdecker]

Sorry could you explain the topology a bit better? From what I gather you have the following two setups:

C -> L -> T

and

L -> C -> T

L being the LND node, C being the c-lightning node, and T being your intended destination for the payment.

What you are saying is that the first scenario fails with c-lightning unable to route the payment, while te second works?

In that case there are a couple of possible causes:

• The L -> T failed to forward the payment, e.g., channel was temporarily unavailable, and C attempted to find an alternative route but failed (which also causes the Could not find route issue). You can check that in the logs
• C was not aware of the L -> T link (yet), meaning it really can't find a route. Is the L -> T channel buried under sufficient confirmations to be public? You can check by calling lightning-cli listchannels 1x2x3 with 1x2x3 being the short channel id of that channel. Notice that the channel may be usable for the endpoints before it becomes available to the rest of the network.
• The L -> T link is not public, again check with listchannels, but still usable by its endpoints.

Generally speaking the logs would be really helpful in this case to figure out what C knows about, how L behaves during the payment attempt and what happens during the payment attempt in C.

[evd0kim]

@cdecker

I described the case when:

C -> L -> T(L)

and also tested potential route when

L -> C -> T(L)

So yes, two setups. In the latter setup everything worked good. So payment settled only in first case.

I acknowledge your recommendation and will keep an eye on them for a while. I will test both cases again using my own clightning node in according to specified cases and report later.

[cdecker]

Ok, I connected from my node (c-lightning with ID 03b31e5bbf2cdbe115b485a2b480e70a1ef3951a0dc6df4b1232e0e56f3dce18d6) to the node in the above listpeers output (02a54deb8d0f11d47c6f55cec5e673063c9fad2619559e8d87ae3eb4c381668449). I am able to successfully route payments through that node using the probe plugin in about 50% of cases (notice that this is doing random payments to random destinations which are offline very often so this is a very good success rate).

One thing that I notice in the output above is that 02a54deb (the node that I just tested) owns all the funds in the channel. IIRC that should be the lnd node. Since 02a53403 doesn't have any capacity it will obviously not be able to route through that channel. It seems lnd was used to open a channel to c-lightning, not the other way around. Did I misunderstand the roles in this setup, or is there simply a misunderstanding in how capacities work here?

[evd0kim]

You are right about roles and not quite right about capacity: it is first thing I usually check when looking at potential route or trying to figure out what happened.

Usually I'm testing amounts around 1000 sats. My peer who has routing problems tried 1 sat payments.

I also look at probe plugin, thank you.

[evd0kim]

I think we shall close an issue. Now I checked routing using two custodial wallets with clients of known type. The payment was 1000 sats and passed from clightning node to target LND via another LND (famous LNBIG).

Payment log:

[                                                                                                        
  {                                                                                                      
    "id": "02c91d6aa51aa940608b497b6beebcb1aec05be3c47704b682b3889424679ca490",                          
    "channel": "562892x1021x0",                                                                          
    "direction": 0,                                                                                      
    "msatoshi": 1000002,                                                                                 
    "amount_msat": "1000002msat",                                                                        
    "delay": 318                                                                                         
  },                                                                                                     
  {                                                                                                      
    "id": "026c7d28784791a4b31a64eb34d9ab01552055b795919165e6ae886de637632efb",                          
    "channel": "567255x1518x0",                                                                          
    "direction": 1,                                                                                      
    "msatoshi": 1000000,                                                                                 
    "amount_msat": "1000000msat",                                                                        
    "delay": 288                                                                                         
  }                                                                                                      
]

I'm pretty sure now that I missed something in my initial tests and my peer probably too.
I will test routing with my leaf clightning node later to be 100% sure that everything works fine.

[cdecker]

Thanks for the update @engenegr, I'll close this for now, but feel free to open a new ticket should the issue return.