Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug]: Unknown desc = edge not found #6987

Closed
dannydeezy opened this issue Oct 5, 2022 · 24 comments
Closed

[bug]: Unknown desc = edge not found #6987

dannydeezy opened this issue Oct 5, 2022 · 24 comments
Labels
bug Unintended code behaviour channel management The management of the nodes channels graph

Comments

@dannydeezy
Copy link

dannydeezy commented Oct 5, 2022

Background

Seems like my LND will randomly not be able to find active channels it has. I do lncli listchannels and the channel shows up as active, but then when I do lncli getchaninfo with the id, it says edge not found.

This has happened a lot, especially when the other node is a Tor node with only 1 total channel.

lncli version 0.15.1-beta commit=v0.15.1-beta

Your environment

  • lncli version 0.15.1-beta commit=v0.15.1-beta
  • Linux Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux
  • bitcoind backend Bitcoin Core version v22.0.0

Steps to reproduce

I don't know how to reproduce, it's flaky.

Example:

lncli listchannels and i see the following entry (among my 800+ other channels)

{
            "active": true,
            "remote_pubkey": "027d92cde72c29ce5a5ae11d91c7ba84245e7917dfd20f1f96ad7f3ea54836a3ad",
            "channel_point": "f7debbfd729a234f4e37235a495e2103d9c1636cabb9e960223b450bd9359954:1",
            "chan_id": "832475437786923009",
            "capacity": "25750000",
            "local_balance": "25011680",
            "remote_balance": "736777",
            "commit_fee": "883",
            "commit_weight": "1116",
            "fee_per_kw": "785",
            "unsettled_balance": "0",
            "total_satoshis_sent": "0",
            "total_satoshis_received": "25011680",
            "num_updates": "373",
            "pending_htlcs": [
            ],
            "csv_delay": 2016,
            "private": false,
            "initiator": false,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "257500",
            "remote_chan_reserve_sat": "257500",
            "static_remote_key": false,
            "commitment_type": "ANCHORS",
            "lifetime": "43603",
            "uptime": "43048",
            "close_address": "",
            "push_amount_sat": "0",
            "thaw_height": 0,
            "local_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "257500",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "25492500000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 2016,
                "chan_reserve_sat": "257500",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "25492500000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "alias_scids": [
            ],
            "zero_conf": false,
            "zero_conf_confirmed_scid": "0"
        },

but then doing lncli getchaninfo 832475437786923009 immediately alfter, i get:

[lncli] rpc error: code = Unknown desc = edge not found

logs from lnd:

[ERR] RPCS: [/lnrpc.Lightning/GetChanInfo]: edge not found

If i restart the node, it fixes it. But I can't be restarting the node all the time.

@dannydeezy dannydeezy added bug Unintended code behaviour needs triage labels Oct 5, 2022
@GeorgeTsagk
Copy link
Collaborator

Hmm.. I was also able to reproduce that a couple of times locally on a regtest setup.. Will see if I can get that error again and share some context. Although I was experimenting with 0-conf channels so could be same bug but different origin.

@crywolf
Copy link

crywolf commented Oct 21, 2022

I had the same problem with newly created channel today. Fixed by restarting lnd.

LND version: 0.15.2-beta
Bitcoin Core version v23.0.0

@guggero
Copy link
Collaborator

guggero commented Oct 24, 2022

@dannydeezy and @crywolf: this was fixed by #6892.
Can you please upgrade to lnd v0.15.3-beta and confirm this doesn't happen anymore?

Closing this issue in the meantime, please re-open it if v0.15.3-beta does not fix.

@guggero guggero closed this as completed Oct 24, 2022
@Wired4ncer
Copy link

Hello. Same issue just happened to me in 0.15.5

@Wired4ncer
Copy link

Wired4ncer commented Mar 4, 2023

Closed a channel with peer and reopened new one, that one got confirmed, but strangely fees and other channel attributes were not set at all and immediately it went inactive. Any attempt to change the parameters fail with similar

"reason": "UPDATE_FAILURE_UNKNOWN",
"update_error": "could not update policies"

Logs are showing channel trying to re-establish, but failing due to edge not found
I'm unable to coop-close or to update the fee policy.
Any advice would be great. Thank you

@guggero guggero reopened this Mar 6, 2023
@sputn1ck
Copy link
Collaborator

sputn1ck commented Mar 6, 2023

Hey @Wired4ncer you could maybe try an update to 0.16 rc1.

For some context, smiliar to OP, the mentioned channel is shown as active in listchannels, but getchaninfo fails?

@Wired4ncer
Copy link

Wired4ncer commented Mar 6, 2023

Yes that exactly was happening. Channel didn't receive any params when opened (fees, htlc size, ctv etc.) and was failing in logs due to not having it when trying to reestablish, but I was not able to update it. Ended up FC'ing it and reopening. Second time it all went fine.

I don't have it open anymore, so updating to 0.16 at the moment won't give us any results.

@gradientskier
Copy link

gradientskier commented Mar 26, 2023

This just happened to me too. In particular, the opening transaction was paying a very low fee (1sat/vb), so the channel has been in state pending for a very long time, for about three weeks. Then I managed to CPFP fee bump the transactions (and I had to broadcast again the opening transaction via external block explorer). The opening transaction eventually got confirmed, but the channel is now in "inactive" state in listchannels and getchaninfo returns error "edge not found". Is FC'ing and reopening the only solution?

@guggero
Copy link
Collaborator

guggero commented Mar 27, 2023

The remote peer will forget the channel after 2 weeks (2016 blocks). So yes, unfortunately force closing is the only option. We'll try to make this more clear with this: #7434

@bota87
Copy link
Contributor

bota87 commented Apr 15, 2023

Just had the same issue on a batch open, all the channel have the same problem.
Transaction was broadcasted and confirmed on the same day.
There is something that I can do other closing? Restart didn't fix the issue.
Running LND 0.16

[ERR] NANN: Unable to retrieve chan status for Channel(xxxxxxxx:2): edge not found
[ERR] NANN: Unable to retrieve chan status for Channel(xxxxxxxx:1): edge not found
[ERR] NANN: Unable to retrieve chan status for Channel(xxxxxxxx:3): edge not found

@bota87
Copy link
Contributor

bota87 commented Apr 16, 2023

I can add other info about this: I chatted with one of the peers, on his side all looks fine, he tried to change the fees and worked. The channel isn't announced but isn't marked as private.
On my side updatechanpolicy, updatechanstatus and payinvoice fails

$ lncli updatechanpolicy 0 1000 144 xxxxxxxxx:3
{
   "failed_updates":[
      {
         "outpoint":"xxxxxxxx:3",
         "reason":"UPDATE_FAILURE_UNKNOWN",
         "update_error":"could not update policies"
      }
   ]
}

$ lncli updatechanstatus --chan_point xxxxxxxx:3 --action enable
[lncli] rpc error: code = Unknown desc = edge not found

$ lncli payinvoice --outgoing_chan_id xxxx 
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   xxxxx
Payment status: FAILED, reason: FAILURE_REASON_INSUFFICIENT_BALANCE
[lncli] FAILED

on the logs I found this error

[ERR] FNDG: Unable to advance state(xxxxxxxxx:3): failed adding to router graph: error generating channel announcement: unable to generate channel update announcement: channel not found

Logs

[INF] BRAR: Breach arbiter starting
[INF] NTFN: New block epoch subscription
[INF] FNDG: Funding manager starting
[INF] BRAR: Starting contract observer, watching for breaches.
[INF] HSWC: HTLC Switch starting
[INF] NTFN: New confirmation subscription: conf_id=2, txid=xxxxxxxxx, num_confs=3 height_hint=yyyyyy
[INF] NTFN: New confirmation subscription: conf_id=3, txid=xxxxxxxxx, num_confs=2 height_hint=yyyyyy
[INF] NTFN: New confirmation subscription: conf_id=4, txid=xxxxxxxxx, num_confs=3 height_hint=yyyyyy
[INF] NTFN: New block epoch subscription
[INF] FNDG: Waiting for funding tx (xxxxxxxxx) to reach 2 confirmations
[INF] FNDG: Waiting for funding tx (xxxxxxxxx) to reach 3 confirmations
[INF] FNDG: Waiting for funding tx (xxxxxxxxx) to reach 3 confirmations
[INF] NTFN: New block epoch subscription
[INF] CNCT: ChainArbitrator starting
[INF] CNCT: Creating ChannelArbitrators for 19 active channels

[INF] NTFN: New spend subscription: spend_id=16, outpoint=xxxxxxxxx:1, script=0 zzzzzzz, height_hint=yyyyyy
[INF] NTFN: Dispatching historical spend rescan for outpoint=xxxxxxxxx:1, script=0 zzzzzzz, start=yyyyyy, end=yyyyyy
[INF] NTFN: New spend subscription: spend_id=18, outpoint=xxxxxxxxx:3, script=0 zzzzzzz, height_hint=yyyyyy
[INF] NTFN: Dispatching historical spend rescan for outpoint=xxxxxxxxx:3, script=0 zzzzzzz, start=yyyyyy, end=yyyyyy
[INF] NTFN: New spend subscription: spend_id=4, outpoint=xxxxxxxxx:2, script=0 zzzzzzz, height_hint=yyyyyy
[INF] NTFN: Dispatching historical spend rescan for outpoint=xxxxxxxxx:2, script=0 zzzzzzz, start=yyyyyy, end=yyyyyy

[INF] CNCT: Close observer for ChannelPoint(xxxxxxxxx:1) active
[INF] CNCT: Close observer for ChannelPoint(xxxxxxxxx:3) active
[INF] CNCT: Close observer for ChannelPoint(xxxxxxxxx:2) active

[INF] CNCT: ChannelArbitrator(xxxxxxxxx:1): starting state=StateDefault, trigger=chainTrigger, triggerHeight=yyyyyy
[INF] CNCT: ChannelArbitrator(xxxxxxxxx:3): starting state=StateDefault, trigger=chainTrigger, triggerHeight=yyyyyy
[INF] CNCT: ChannelArbitrator(xxxxxxxxx:2): starting state=StateDefault, trigger=chainTrigger, triggerHeight=yyyyyy

[WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:3) w/ newer version
[WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:1) w/ newer version
[WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:2) w/ newer version

[INF] HSWC: Trimming open circuits for chan_id=x:y:z, start_htlc_id=695
[INF] PEER: Peer(ppppppppp): loading ChannelPoint(xxxxxxxxx:3)
[WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:3, using default values
[INF] PEER: Peer(ppppppppp): loading ChannelPoint(xxxxxxxxx:2)
[WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:2, using default values
[INF] PEER: Peer(ppppppppp): loading ChannelPoint(xxxxxxxxx:1)
[WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:1, using default values
[INF] PEER: Peer(ppppppppp): loading ChannelPoint(xxxxxxxxx:3)
[WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:3, using default values

[INF] LNWL: Marking unconfirmed transaction xxxxxxxxx mined in block block111
[INF] CRTR: Pruning channel graph using block block111 (height=block111)
[INF] CRTR: Block block111 (height=block111) closed 0 channels
[INF] NTFN: New block: height=block111, sha=block111
[INF] UTXN: Attempting to graduate height=block111: num_kids=0, num_babies=0
[INF] HSWC: Garbage collected 5 shared secret hashes at height=block111

[INF] CRTR: Block block222 (height=block222) closed 2 channels
[INF] NTFN: New block: height=block222, sha=block222
[INF] NTFN: Dispatching 2 confirmation notification for txid=xxxxxxxxx
[INF] FNDG: ChannelPoint(xxxxxxxxx:3) is now active: ChannelID(chanid1)
[INF] UTXN: Attempting to graduate height=block222: num_kids=0, num_babies=0
[INF] HSWC: Garbage collected 4 shared secret hashes at height=block222
[INF] CHBU: Updating on-disk multi SCB backup: num_old_chans=19, num_new_chans=19
[INF] FNDG: Peer(ppppp) is online, sending FundingLocked for ChannelID(chanid1)
[WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:3) w/ newer version
[INF] CHBU: Updating backup file at /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
[INF] CHBU: Swapping old multi backup file from /root/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.backup to /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
[ERR] NANN: Unable to retrieve chan status for Channel(xxxxxxxxx:3): edge not found
[INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=18 ineligible=0) sessions(acquired=0 exhausted=106)
[ERR] RPCS: [/lnrpc.Lightning/GetChanInfo]: edge not found
[INF] PEER: Peer(ppppp): New channel active ChannelPoint(xxxxxxxxx:3) with peer
[INF] HSWC: ChannelLink(xxxxxxxxx:3): starting
[ERR] FNDG: Unable to advance state(xxxxxxxxx:3): failed adding to router graph: error generating channel announcement: unable to generate channel update announcement: channel not found
[INF] HSWC: Trimming open circuits for chan_id=block111:y:3, start_htlc_id=0
[INF] HSWC: Adding live link chan_id=chanid1, short_chan_id=block111:y:3
[INF] CNCT: Attempting to update ContractSignals for ChannelPoint(xxxxxxxxx:3)
[INF] HSWC: ChannelLink(xxxxxxxxx:3): HTLC manager started, bandwidth=4564566000 mSAT
[INF] HSWC: ChannelLink(xxxxxxxxx:3): attempting to re-synchronize
[INF] HSWC: ChannelLink(xxxxxxxxx:3): resending FundingLocked message to peer
[INF] HSWC: ChannelLink(xxxxxxxxx:3): received re-establishment message from remote side
[INF] FNDG: Received duplicate fundingLocked for ChannelID(chanid1), ignoring.

[INF] CRTR: Block block333 (height=block333) closed 0 channels
[INF] NTFN: New block: height=block333, sha=block333
[INF] UTXN: Attempting to graduate height=block333: num_kids=0, num_babies=0
[INF] NTFN: Dispatching 3 confirmation notification for txid=xxxxxxxxx
[INF] NTFN: Dispatching 3 confirmation notification for txid=xxxxxxxxx
[INF] FNDG: ChannelPoint(xxxxxxxxx:1) is now active: ChannelID(chanid2)
[INF] FNDG: ChannelPoint(xxxxxxxxx:2) is now active: ChannelID(chanid3)
[INF] HSWC: Garbage collected 3 shared secret hashes at height=block333
[INF] CHBU: Updating on-disk multi SCB backup: num_old_chans=19, num_new_chans=19
[INF] RPCS: channel: xxxxxxxxx:1 not found by channel event store
[INF] RPCS: channel: xxxxxxxxx:2 not found by channel event store
[INF] FNDG: Peer(pppppppppp) is online, sending FundingLocked for ChannelID(chanid2)
[WRN] CHBU: Replacing disk backup for ChannelPoint(xxxxxxxxx:3) w/ newer version
[INF] CHBU: Updating backup file at /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
[INF] CHBU: Swapping old multi backup file from /root/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.backup to /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
[INF] FNDG: Peer(pppppppppp) is online, sending FundingLocked for ChannelID(chanid3)
[INF] CHBU: Updating on-disk multi SCB backup: num_old_chans=19, num_new_chans=19
[ERR] FNDG: Unable to advance state(xxxxxxxxx:2): failed adding to router graph: error generating channel announcement: unable to generate channel update announcement: channel not found
[INF] PEER: Peer(pppppppppp): New channel active ChannelPoint(xxxxxxxxx:2) with peer
[INF] PEER: Peer(pppppppppp): New channel active ChannelPoint(xxxxxxxxx:1) with peer
[INF] HSWC: ChannelLink(xxxxxxxxx:2): starting
[INF] HSWC: Trimming open circuits for chan_id=block111:y:2, start_htlc_id=0
[INF] HSWC: Adding live link chan_id=chanid3, short_chan_id=block111:y:2
[INF] CNCT: Attempting to update ContractSignals for ChannelPoint(xxxxxxxxx:2)
[INF] HSWC: ChannelLink(xxxxxxxxx:2): HTLC manager started, bandwidth=4564566000 mSAT
[INF] HSWC: ChannelLink(xxxxxxxxx:2): attempting to re-synchronize
[INF] HSWC: ChannelLink(xxxxxxxxx:1): starting
[INF] CHBU: Updating backup file at /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
[INF] CHBU: Swapping old multi backup file from /root/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.backup to /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
[INF] CHBU: Updating on-disk multi SCB backup: num_old_chans=19, num_new_chans=19
[INF] HSWC: Trimming open circuits for chan_id=block111:y:1, start_htlc_id=0
[INF] HSWC: Adding live link chan_id=chanid2, short_chan_id=block111:y:1
[INF] CNCT: Attempting to update ContractSignals for ChannelPoint(xxxxxxxxx:1)
[INF] HSWC: ChannelLink(xxxxxxxxx:1): HTLC manager started, bandwidth=4564566000 mSAT
[INF] HSWC: ChannelLink(xxxxxxxxx:1): attempting to re-synchronize
[INF] HSWC: ChannelLink(xxxxxxxxx:1): resending FundingLocked message to peer
[INF] HSWC: ChannelLink(xxxxxxxxx:1): received re-establishment message from remote side
[INF] HSWC: ChannelLink(xxxxxxxxx:2): resending FundingLocked message to peer
[INF] HSWC: ChannelLink(xxxxxxxxx:2): received re-establishment message from remote side
[INF] FNDG: Received duplicate fundingLocked for ChannelID(chanid3), ignoring.
[INF] CHBU: Updating backup file at /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
[INF] CHBU: Swapping old multi backup file from /root/.lnd/data/chain/bitcoin/mainnet/temp-dont-use.backup to /root/.lnd/data/chain/bitcoin/mainnet/channel.backup
[INF] FNDG: Received duplicate fundingLocked for ChannelID(chanid2), ignoring.
[ERR] RPCS: [/lnrpc.Lightning/GetChanInfo]: edge not found
[ERR] FNDG: Unable to advance state(xxxxxxxxx:1): failed adding to router graph: error generating channel announcement: unable to generate channel update announcement: channel not found
[INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=41 ineligible=0) sessions(acquired=0 exhausted=910)

@ellemouton
Copy link
Collaborator

@bota87 - So I think we only add the channel to our local graph after receiving the ChannelReady message from our peer - if this all happened for you on the same day (ie, your peer has not timed out the channels) then my hunch says that either they are not sending the ChannelReady/FundingLocked or your node is not handling the message properly. Do you see anything in the logs re FundingLocked? might be good idea to ask your peer the same question to see if the issue is with sending the message

@bota87
Copy link
Contributor

bota87 commented Apr 17, 2023

Hi @ellemouton, all it's happened in the same day (batch channels open request and transaction confirmation).
On the logs that I've posted there are many FundingLocked messages, I've checked again for ChannelReady but didn't find.
I've asked to the peer his logs, waiting for the reply.

@bota87
Copy link
Contributor

bota87 commented Apr 18, 2023

Here the logs from the peer, he said that didn't find any ChannelReady message

@ellemouton
Copy link
Collaborator

further up in your logs, do you at all see a log starting with Unable to store the forwarding policy?

(also, don't worry about ChannelReady, in 0.16 it will be FundingLocked)

@ellemouton
Copy link
Collaborator

another question: when exactly did you upgrade to 0.16? did you initiate the funding and update to 0.16 afterwards or was this all done after upgrading to 0.16?

@bota87
Copy link
Contributor

bota87 commented Apr 19, 2023

further up in your logs, do you at all see a log starting with Unable to store the forwarding policy?

no the only messages talking about policy is this one
[WRN] PEER: Peer(ppppppppp): Unable to find our forwarding policy for channel xxxxxxxxx:3, using default values

another question: when exactly did you upgrade to 0.16? did you initiate the funding and update to 0.16 afterwards or was this all done after upgrading to 0.16?

I'm not sure, I migrated from 0.15.4 to 0.16 on the same day but didn't remember if I start the batch channel open before or after the update

@ellemouton
Copy link
Collaborator

ok cool - I managed to reproduce your scenario. Thanks for reporting! Working on a fix asap

@bota87
Copy link
Contributor

bota87 commented Apr 19, 2023

That's great! The fix will let me use this channels or I have to close and reopen? I have to wait for the fix before opening other channels?

@ellemouton
Copy link
Collaborator

I think the fix will let you use the channels as long as the peer doesnt force close before you apply the fix.

It should be safe to open other channels in the mean time - the error resulted from the channel being opened before upgrading to 0.16, then upgrading to 0.16 before channel confirmation.

@bota87
Copy link
Contributor

bota87 commented Apr 19, 2023

Updated to 0.16.1.rc1, the channels works! 🥳
Thank's @ellemouton

@saubyk saubyk added channel management The management of the nodes channels graph and removed needs triage labels Aug 30, 2023
@saubyk
Copy link
Collaborator

saubyk commented Aug 30, 2023

Closing this issue. Feel free to reopen if the reported problem has not been addressed.

@saubyk saubyk closed this as completed Aug 30, 2023
@dskvr
Copy link

dskvr commented Nov 2, 2023

Still happens on most recent release.

2023-11-02 14:13:24.024 [ERR] RPCS: [/lnrpc.Lightning/GetChanInfo]: edge not found

@ralyodio
Copy link

ralyodio commented Nov 6, 2023

i get this opening fee modal in RTL.

500 - {"code":2,"message":"edge Not Found","details":[]}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Unintended code behaviour channel management The management of the nodes channels graph
Projects
None yet
Development

No branches or pull requests