pay: most payments fail

[kilrau]

Issue and Steps to Reproduce

Most payments (as small as 10000 sats) we are trying to send via CLN's pay gRPC are failing with Reason: Ran out of routes to try after 18 attempts: see paystatus which is why we are not using CLN for sending payments in production right now, only receiving. Only opening this issue now because we wanted to make sure it's not sth incredibly stupid on our side. Couldn't find anything obvious.

Observations:

• Quite often we see a large amount of mpp shards (like 15) where 1 or 2 fail that make the entire payment fail. This is the most evident thing we are observing - why so many shards?
• Via gRPC CLN gives up trying very fast. Like within seconds.
• The number of attempts sending the same amount to the same destination varies greatly:

PaymentPending
PaymentFailed

Reason: Ran out of routes to try after 48 attempts: see `paystatus`
---
PaymentPending
PaymentFailed
Reason: Ran out of routes to try after 61 attempts: see `paystatus`
---
PaymentPending
PaymentFailed
Reason: Ran out of routes to try after 18 attempts: see `paystatus`

• Smaller manual payments via lightning-cli sometimes work fine, whereas they mostly fail on gRPC (just a hunch)
• Not all gRPC payments fail, some go through, but it's sth like >90% failure rate
• Routing and all other things work fine
• Our CLN node definitely is large and well connected enough by now to reliably send the payments we are trying to send
• Nothing helpful in the logs

We are absolutely clueless on what's going on and if it's simply the amount of mpp shards that make things fail. Any hints appreciated.

getinfo output

{
   "id": "02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018",
   "alias": "Boltz|CLN",
   "color": "ff9800",
   "num_peers": 88,
   "num_pending_channels": 0,
   "num_active_channels": 80,
   "num_inactive_channels": 1,
   "address": [
      {
         "type": "ipv4",
         "address": "45.86.229.190",
         "port": 9736
      },
      {
         "type": "ipv6",
         "address": "2a10:1fc0:3::270:a9dc",
         "port": 9736
      },
      {
         "type": "torv3",
         "address": "oo5tkbbpgnqjopdjxepyfavx3yemtylgzul67s7zzzxfeeqpde6yr7yd.onion",
         "port": 9736
      }
   ],
   "binding": [
      {
         "type": "ipv4",
         "address": "0.0.0.0",
         "port": 9736
      },
      {
         "type": "ipv6",
         "address": "::1",
         "port": 9736
      }
   ],
   "version": "v23.08.1",
   "blockheight": 812772,
   "network": "bitcoin",
   "fees_collected_msat": 80795360,
   "lightning-dir": "/home/bitcoin/.lightning/bitcoin",
   "our_features": {
      "init": "08a0000a8a69a2",
      "node": "88a0000a8a69a2",
      "channel": "",
      "invoice": "02000002024100"
   }
}

https://amboss.space/node/02d96eadea3d780104449aca5c93461ce67c1564e2e1d73225fa67dd3b997a6018

[kilrau]

Adding questions @michael1011 previously asked on discord:

• is there a way to force it to make it split less aggressively? Like eg only to at most 2 parts
• I am seeing pay terminate and fail wayyy before the limit set by retry_for. Ran out of routes to try after 54 attempts: see paystatus would that error mean that all possible routes have been tried and failed? I can't imagine that to be true, because different attempts of paying the same invoice show different numbers

[kingonly]

Are you sending to nodes with private channels?

[kilrau]

It's pretty big nodes we tried sending to so far so high probability that these have private channels, yes.

[kingonly]

Does the invoice has routing hints is basically what I'm asking.

[kilrau]

Yes almost all do. Will do some test using invoices without routing hints.

[kilrau]

Update: no difference when paying 100k sats invoices without routing hints.

Again the destination is a big node and we are definitely well connected enough to pay 100k sats invoices.

[kilrau]

So this is very interesting:

• paying 100k sats invoice - failed
• paying 50k sats invoice - failed
• paying 10k sats invoices worked flawlessly for 50x in a row = 500k sats. It always used the exact same route. So there was a route that could have easily handled the larger payments - how can it be that the first two payments didn't use this route? Also CLN didn't have to try much - it knew the route already and used it at first shot for the 10k sats payments.

Btw for these tests I am always using (for my standards) a very high routing fee limit of 0.5% / 5000ppm to make sure this is not causing any failures and obviously always the same destination.

[kilrau]

Another test that simply confirms that there is no difference between lightning-cli and grpc (as expected):

• lightning-cli pay <100k sats invoice> - failed after 4 attempts.

{
   "code": 210,
   "message": "Ran out of routes to try after 4 attempts: see `paystatus`",
   "attempts": [
      {
         "status": "failed",
         "failreason": "failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)",
         "partid": 0,
         "amount_msat": 100000000
      },
      {
         "status": "failed",
         "failreason": "failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)",
         "partid": 1,
         "amount_msat": 100000000,
         "parent_partid": 0
      },
      {
         "status": "failed",
         "failreason": "failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)",
         "partid": 2,
         "amount_msat": 100000000,
         "parent_partid": 1
      },
      {
         "status": "failed",
         "failreason": "failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)",
         "partid": 3,
         "amount_msat": 100000000,
         "parent_partid": 2
      }
   ]
}

paystatus also reveals that CLN wanted to try 10 times but it only tried 4 times. I don't know why.

• paying 10x 10k sats invoices via grpc worked flawlessly again.

Again same destination, same 5000 ppm routing fee limit for both.

[kilrau]

Test if routing hints play a role: they don't

Same behavior with invoices that contain routing hints:

• paying 100k sats invoice - failed
• paying 10x10k sats invoice - success and again used the same route for all 10 payments

Again same destination, same 5000 ppm routing fee limit for both.

[kilrau]

Ran similar tests today and .. it doesn't reproduce as reliably as before. Sometimes the first 100k payment or even larger goes through :/

Same destination as before, same 5000 ppm routing fee limit as before.

Any ideas what else I could try?

[vincenzopalazzo]

Any ideas what else I could try?

When the payment fails it is possible that we choose the wrong path and we keep choosing the wrong one over and over again.

Or is it possible that some channels as disabled? e.g: On chain fee disagreement?

From your messages, I am not able to track down what is happening, but I think it is also difficult from the pay command to track what is going for you due that we hide the complexity under the hood

[kilrau]

choose the wrong path and we keep choosing the wrong one over and over again

I would love to follow exactly which routes are being tried and see with which errors they fail but I simply couldn't find how. How can I?