Payment through an open channel fails

[Illidanek]

Issue and Steps to Reproduce

I have an open channel with Node X. Here is the state of this channel (some values are hidden for privacy reasons):

 {
  "id": "{NODE X ID}", 
  "connected": true, 
  "netaddr": [
    "NODEX IP:58572"
  ], 
  "global_features": "", 
  "local_features": "82", 
  "globalfeatures": "", 
  "localfeatures": "82", 
  "channels": [
    {
      "state": "CHANNELD_NORMAL", 
      "scratch_txid": "{SCRATCH_TXID}", 
      "owner": "lightning_channeld", 
      "short_channel_id": "{CHANNEL_X_ID}", 
      "channel_id": "{CHANNEL ID}", 
      "funding_txid": "{FUNDINGTX_ID}", 
      "msatoshi_to_us": 3325920000, 
      "msatoshi_to_us_min": 3325920000, 
      "msatoshi_to_us_max": 3325920000, 
      "msatoshi_total": 3325920000, 
      "dust_limit_satoshis": 546, 
      "max_htlc_value_in_flight_msat": 18446744073709551615, 
      "their_channel_reserve_satoshis": 33260, 
      "our_channel_reserve_satoshis": 33259, 
      "spendable_msatoshi": 3292661000, 
      "htlc_minimum_msat": 0, 
      "their_to_self_delay": 144, 
      "our_to_self_delay": 399, 
      "max_accepted_htlcs": 483, 
      "status": [
        "CHANNELD_NORMAL:Reconnected, and reestablished.", 
        "CHANNELD_NORMAL:Funding transaction locked. Waiting for their announcement signatures."
      ], 
      "in_payments_offered": 0, 
      "in_msatoshi_offered": 0, 
      "in_payments_fulfilled": 0, 
      "in_msatoshi_fulfilled": 0, 
      "out_payments_offered": 0, 
      "out_msatoshi_offered": 0, 
      "out_payments_fulfilled": 0, 
      "out_msatoshi_fulfilled": 0, 
      "htlcs": [
      ]
    }
  ]
}

The announcement signature has not been sent yet, but from what I was told, this is not a problem and I should be able to use the channel. However, when I try to send to this node, the payment through this channel fails (and ends up using a different route with a few hops). Here is the error I get during send:

failures": [
{
"message": "Outgoing subdaemon died",
"type": "FAIL_PAYMENT_REPLY",
"erring_index": 0,
"failcode": 4103,
"erring_node": "MY NODE ID",
"erring_channel": "{CHANNEL_X_ID}",
"route": [
{
"id": "NODE X ID",
"channel": "{CHANNEL_X_ID}",
"msatoshi": 1238254,
"delay": 144
}
]
}

Do note that I am trying to send 1234 satoshis, so there is plenty of capacity.

getinfo output

"version": "v0.6.2-121-gda465f0"

[wythe]

First thing would be to upgrade to the latest version, v0.6.3, or rebuild from master. Then we can jump into further debugging.

[Illidanek]

Ok, I updated, here's the current version number:

"version": "v0.6.3rc1-104-g287af7b"

And I'm still getting the same problem. Here are the details of another payment not being able to be routed through a directly connected channel:

attempts": [{ "strategy": "Initial attempt", "start_time": "2019-01-25T08:36:54.777Z", "age_in_seconds": 105, "end_time": "2019-01-25T08:37:25.698Z", "duration_in_seconds": 30, "excluded_channels": [{SOME_OTHER_CHANNELS}], "route": [
{
"id": "{NODE_X}",
"channel": "${CHANNEL_X}",
"direction": 1,
"msatoshi": 1234000,
"delay": 144
}
], "failure": { "code" : 204, "message" : "Call to waitsendpay:failed: WIRE_TEMPORARY_CHANNEL_FAILURE (Outgoing subdaemon died)", "data" :
{
"erring_index": 0,
"failcode": 4103,
"failcodename": "WIRE_TEMPORARY_CHANNEL_FAILURE",
"erring_node": "{MY_NODE}",
"erring_channel": "${CHANNEL_X}",
"erring_direction": 1
} }}

[cdecker]

Do you have any logs with for that node? Since the first hop fails it appears that the connection gets lost and we only learn about the failure when attempting to send an update through it. Is node X stable otherwise, or do you get timeouts from time to time?

[rustyrussell]

listpeers {NODE_X} debug should give you way-too-much information, but might be helpful here?

[mloop1]

Got a similar error today (WIRE_TEMPORARY_CHANNEL_FAILURE)
Let me know if I can provide any log or other useful info?

2019-04-09 19:01:21: RPC call failed: method: waitsendpay, payload: {'payment_hash': '765425c0b756434f3164b797e88694d65f7ea815785927d7d8fd9072edf83810'}, error: {'data': {'erring_channel': '570523x648x1', 'failcode': 4103, 'failcodename': 'WIRE_TEMPORARY_CHANNEL_FAILURE', 'erring_index': 1, 'erring_direction': 0, 'erring_node': '0331f80652fb840239df8dc99205792bba2e559a05469915804c08420230e23c7c'}, 'code': 204, 'message': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)'}

[evd0kim]

I'm trying to figure out what is happening with one of my peer node (c-lightning). I've created an issue about routing.

Now I have same error as described here and actually a routing failure described below.

1. We tried to route 5896 sat payment.
2. I know one of supposed routes could be via LND with 2 hops, so C->LND->C
3. We got error "Could not find a route"

Paystatus logs from payment origin node (C):
https://pastebin.com/KUApq7JS

Listpeers debug logs with lnd node from payment origin node (C):
https://pastebin.com/wdDWxLHF

About potential route:
Channel state how it sees LND node (my):
First hop (C->LND)

[
  {
    "active": true,
    "remote_pubkey": "02a5340317463d36d6d2ff9c8fd2e4fad87e4a948ea34dcefdb8e9518bffe2975f",
    "channel_point": "da1cc92101a77e431ff6150c4d6a1c67f3738fbd352648bc1904bcc0c63010af:1",
    "chan_id": "630857990667304961",
    "capacity": "1000000",
    "local_balance": "621783",
    "remote_balance": "366987",
    "commit_fee": "11230",
    "commit_weight": "724",
    "fee_per_kw": "15510",
    "unsettled_balance": "0",
    "total_satoshis_sent": "1014010",
    "total_satoshis_received": "647022",
    "num_updates": "344",
    "pending_htlcs": [],
    "csv_delay": 144,
    "private": false,
    "initiator": true,
    "chan_status_flags": "ChanStatusDefault"
  }
]

Second hop:

[
  {
    "active": true,
    "remote_pubkey": "02c16cca44562b590dd279c942200bdccfd4f990c3a69fad620c10ef2f8228eaff",
    "channel_point": "1a113cb7cadaaf827e7e06ab6b3d40c3a120ae4c9001236f70b95eb160b25f5f:0",
    "chan_id": "616301556290420736",
    "capacity": "1089964",
    "local_balance": "391577",
    "remote_balance": "687099",
    "commit_fee": "11288",
    "commit_weight": "724",
    "fee_per_kw": "15591",
    "unsettled_balance": "0",
    "total_satoshis_sent": "1100766",
    "total_satoshis_received": "1492344",
    "num_updates": "6972",
    "pending_htlcs": [],
    "csv_delay": 144,
    "private": false,
    "initiator": false,
    "chan_status_flags": "ChanStatusDefault"
  }
]

As you can see both channels are active and have enough funds to route payment. Not even at 2% channel capacity boundary. Direct payment 5896 to LND was successful.

The very question for me now, what means WIRE_TEMPORARY_CHANNEL_FAILURE and how to deal with it?

[ZmnSCPxj]

@Illidanek is this still an issue?

@mloop1 your error is from remote ("(reply from remote)"), and is thus different from this issue.

[ZmnSCPxj]

Closing due to no response from OP.