LND v0.14.2-beta fails to startup completely

[juscamarena]

Background

Describe your issue here.

LND fails to start up completely and deadlocks at some point without having any active channels.

I can't stop LND cleanly with stop unless I force kill it as well.

Your environment

• version of lnd
  0.14.2-beta commit=v0.14.2-beta

• which operating system (uname -a on *Nix)
  x86_64 GNU/Linux

• version of btcd, bitcoind, or other backend
  bitcoind v22.0.0

• any other relevant environment details

Steps to reproduce

Tell us how to reproduce this issue. Please provide stacktraces and links to code in question.

goroutine.out.txt
lnd.log

Expected behaviour

Tell us what should happen

LND startups fully.

Actual behaviour

Tell us what happens instead

[juscamarena]

Here's the heap dump as well.
heap.out.txt

[juscamarena]

I updated to 0.15 rc6 and the node fully starts up now. ¯_(ツ)_/¯

[guggero]

The goroutine dump, is that from after you issue the shutdown command? It looks like it to me since there are only very few goroutines active. It would've been useful to get a goroutine dump from the moment when you suspected it to have deadlocked. But since you updated already we won't know which bugfix in 0.15 fixed the issue for you...

[juscamarena]

The goroutine dump, is that from after you issue the shutdown command? It looks like it to me since there are only very few goroutines active. It would've been useful to get a goroutine dump from the moment when you suspected it to have deadlocked. But since you updated already we won't know which bugfix in 0.15 fixed the issue for you...

hmm. I'm not sure. I don't remember.

Yeah, well I had ~900 channels active and users were asking us when we'd we be back online as all that liquidity went offline. I assume it's something fixed in the new update in any case?

[guggero]

Yes, I totally understand that you needed the problem to be solved ASAP.
It just would've been interesting to see where the deadlock actually happened to make sure it isn't just a fluke that it now works with 0.15. But I guess we have to assume that one of the many bug fixes in 0.15 also took care of this specific problem.

[Crypt-iQ]

did you have any breach closes

[Crypt-iQ]

This is fixed #6214 , logs show channel arbitrator starting in state StateContractClosed and then no log outputs after.