Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can not open database, lnd keeps restarting #4720

Closed
manreo opened this issue Oct 25, 2020 · 17 comments
Closed

Can not open database, lnd keeps restarting #4720

manreo opened this issue Oct 25, 2020 · 17 comments

Comments

@manreo
Copy link
Contributor

manreo commented Oct 25, 2020 •
edited
Loading

Background

I restarted the node and now I can not open the database, lnd just keeps restarting:

2020-10-25 04:53:41.089 [DBG] SRVR: removing peer 03bb88ccc444534da7b5b64b4f7b15e1eccb18e102db0e400d4b9cfe93763aa26d@138.68.14.104:9735
2020-10-25 04:53:41.102 [INF] RPCS: Stopping RPC Server
2020-10-25 04:53:41.109 [INF] RPCS: Stopping SignRPC Sub-RPC Server
2020-10-25 04:53:41.116 [INF] RPCS: Stopping VersionRPC Sub-RPC Server
2020-10-25 04:53:41.122 [INF] RPCS: Stopping RouterRPC Sub-RPC Server
2020-10-25 04:53:41.122 [INF] RPCS: Stopping AutopilotRPC Sub-RPC Server
2020-10-25 04:53:41.126 [INF] RPCS: Stopping ChainRPC Sub-RPC Server
2020-10-25 04:53:41.126 [INF] RPCS: Stopping InvoicesRPC Sub-RPC Server
2020-10-25 04:53:41.126 [INF] RPCS: Stopping WalletKitRPC Sub-RPC Server
2020-10-25 04:53:41.126 [INF] RPCS: Stopping WatchtowerRPC Sub-RPC Server
2020-10-25 04:53:41.126 [INF] RPCS: Stopping WatchtowerClientRPC Sub-RPC Server
2020-10-25 04:53:41.236 [INF] LTND: Shutdown complete

2020-10-25 04:55:05.017 [INF] LTND: Version: 0.11.1-beta commit=v0.11.1-beta, build=production, logging=default
2020-10-25 04:55:05.017 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2020-10-25 04:55:05.033 [INF] LTND: Opening the main database, this might take a few minutes...
2020-10-25 04:55:05.034 [INF] LTND: Opening bbolt database, sync_freelist=false


2020-10-25 16:57:34.736 [INF] LTND: Version: 0.11.1-beta commit=v0.11.1-beta, build=production, logging=default
2020-10-25 16:57:34.736 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2020-10-25 16:57:34.737 [INF] LTND: Opening the main database, this might take a few minutes...
2020-10-25 16:57:34.737 [INF] LTND: Opening bbolt database, sync_freelist=false
2020-10-25 16:58:35.246 [INF] LTND: Version: 0.11.1-beta commit=v0.11.1-beta, build=production, logging=default
2020-10-25 16:58:35.247 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2020-10-25 16:58:35.247 [INF] LTND: Opening the main database, this might take a few minutes...
2020-10-25 16:58:35.247 [INF] LTND: Opening bbolt database, sync_freelist=false
2020-10-25 16:59:35.740 [INF] LTND: Version: 0.11.1-beta commit=v0.11.1-beta, build=production, logging=default
2020-10-25 16:59:35.741 [INF] LTND: Active chain: Bitcoin (network=mainnet)

The timing are really strange, at 4:53 there was a shutdown and then it tried to open again at 4:55 ?

This is what I see in the jpurnalctl:

admin@rock64:~$ sudo journalctl -f -u lnd
-- Logs begin at Sun 2020-10-25 04:54:23 UTC. --
Oct 25 17:58:04 rock64 systemd[1]: Started LND Lightning Daemon.
Oct 25 17:58:05 rock64 lnd[3247]: Attempting automatic RPC configuration to bitcoind
Oct 25 17:58:05 rock64 lnd[3247]: Automatically obtained bitcoind's RPC credentials
Oct 25 17:58:05 rock64 lnd[3247]: 2020-10-25 17:58:05.026 [INF] LTND: Version: 0.11.1-beta commit=v0.11.1-beta, build=production, logging=default
Oct 25 17:58:05 rock64 lnd[3247]: 2020-10-25 17:58:05.026 [INF] LTND: Active chain: Bitcoin (network=mainnet)
Oct 25 17:58:05 rock64 lnd[3247]: 2020-10-25 17:58:05.026 [INF] LTND: Opening the main database, this might take a few minutes...
Oct 25 17:58:05 rock64 lnd[3247]: 2020-10-25 17:58:05.026 [INF] LTND: Opening bbolt database, sync_freelist=false
Oct 25 17:58:05 rock64 systemd[1]: lnd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 25 17:58:05 rock64 systemd[1]: lnd.service: Unit entered failed state.
Oct 25 17:58:05 rock64 systemd[1]: lnd.service: Failed with result 'exit-code'.
Oct 25 17:59:05 rock64 systemd[1]: lnd.service: Service hold-off time over, scheduling restart.
Oct 25 17:59:05 rock64 systemd[1]: Stopped LND Lightning Daemon.
Oct 25 17:59:05 rock64 systemd[1]: Started LND Lightning Daemon.
Oct 25 17:59:05 rock64 lnd[3266]: Attempting automatic RPC configuration to bitcoind
Oct 25 17:59:05 rock64 lnd[3266]: Automatically obtained bitcoind's RPC credentials
Oct 25 17:59:05 rock64 lnd[3266]: 2020-10-25 17:59:05.482 [INF] LTND: Version: 0.11.1-beta commit=v0.11.1-beta, build=production, logging=default
Oct 25 17:59:05 rock64 lnd[3266]: 2020-10-25 17:59:05.483 [INF] LTND: Active chain: Bitcoin (network=mainnet)
Oct 25 17:59:05 rock64 lnd[3266]: 2020-10-25 17:59:05.483 [INF] LTND: Opening the main database, this might take a few minutes...
Oct 25 17:59:05 rock64 lnd[3266]: 2020-10-25 17:59:05.483 [INF] LTND: Opening bbolt database, sync_freelist=false
Oct 25 17:59:05 rock64 lnd[3266]: panic: freepages: failed to get all reachable pages (page 43068: multiple references)
Oct 25 17:59:05 rock64 lnd[3266]: goroutine 20 [running]:
Oct 25 17:59:05 rock64 systemd[1]: lnd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 25 17:59:05 rock64 systemd[1]: lnd.service: Unit entered failed state.
Oct 25 17:59:05 rock64 systemd[1]: lnd.service: Failed with result 'exit-code'.

Your environment

v0.11.1-beta
Linux rock64 4.4.132-1075-rockchip-ayufan-ga83beded8524 #1 SMP Thu Jul 26 08:22:22 UTC 2018 aarch64 GNU/Linux
Bitcoin Core version v0.20.1

Thanks
@manreo
Copy link
Contributor Author

manreo commented Oct 26, 2020 •
edited
Loading

Hi @guggero
I've tried the following:
~/download/gopath/bin/chantools compactdb --sourcedb /home/admin/download/db_restore/channel.db --destdb /home/admin/download/db_restore/res2/compacted.db

and this is what I get:

admin@rock64:~/download/db_restore$ ~/download/gopath/bin/chantools compactdb --sourcedb /home/admin/download/db_restore/channel.db --destdb /home/admin/download/db_restore/res2/compacted.db
2020-10-26 00:16:42.366 [INF] CHAN: chantools version v0.5.0 commit v0.5.0
2020-10-26 00:16:50.434 [WRN] CHAN: Could not read bucket 'chan-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:50.434 [WRN] CHAN: Could not read bucket 'disabled-edge-policy-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:50.435 [WRN] CHAN: Could not read bucket 'edge-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:50.435 [WRN] CHAN: Could not read bucket 'edge-update-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:50.435 [WRN] CHAN: Could not read bucket 'zombie-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:50.925 [WRN] CHAN: Could not read bucket 'chan-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:50.926 [WRN] CHAN: Could not read bucket 'disabled-edge-policy-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:50.926 [WRN] CHAN: Could not read bucket 'edge-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:50.926 [WRN] CHAN: Could not read bucket 'edge-update-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:50.926 [WRN] CHAN: Could not read bucket 'zombie-index' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.523 [WRN] CHAN: Could not read bucket 'message-store' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.524 [WRN] CHAN: Could not read bucket 'metadata' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.524 [WRN] CHAN: Could not read bucket 'missioncontrol-results' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.524 [WRN] CHAN: Could not read bucket 'network-result-store-bucket' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.524 [WRN] CHAN: Could not read bucket 'next-payment-id-key' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.524 [WRN] CHAN: Could not read bucket 'nib' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.524 [WRN] CHAN: Could not read bucket 'open-chan-bucket' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.524 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h���W(�`l�Q�{f��j����Q��$�J�' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.524 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�
                  4��/���������gh�|�eNO%' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
�@�����<J8�T���n�9���er�j' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h��;��
                      �2Gf���u�WӀ�D�wf�i8ܠo�C�' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�d��׺*�ŧ�&�֤�' �C��:�AfW��P�' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�-;��AZ�c�
�1�i՜�p�Di`Kh�����' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�1F�ڢʮ�Gʄ�3���O���i����/' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�3�Xb�R؁���rzp!(ʗ.�
�*:��B�' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�5e�Zf�������y�pj�/���[���9r' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�5��M�6�L�n �t��d0P�LJ6V_�@*d�' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�6�Tq�k�
�����3��x]Aǩ�ȓ�gnh' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�7���j�]�*0}.����g�-X{k`B�O�<]q' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�7�d��<~YC��Gq��h���K�W�(]�$�' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.525 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�?/@G�L�����q��f
�V�t'8Q�G' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�@�~�Z�2Οm�ͩVM��7y�I�'�{K#1' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�A�l����{ݔ���b�th���;��< [���' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��@i�}�L�!�O�' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�]�7r�������3�.��6	"򑯜g0�

                                             ��' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�`��0��qN?Ѱ��`���o����Wc��
                                           w��' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�e̝�߳L�e�),4�nF9�X��F�,� >$�' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�qT�X'D�
                         "L�F8J2f��Z;��7�0v���' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�u��z�t�D܀�P�a�!cH;�e�U' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h�����
                      �pU��Z`��ݞe��K�/��' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h��؛��*�A�-"��5� %�h��c��n`��{
                                              ' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h����^��Ԥ����-���A�-��4W�F�D��' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.526 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h���h��*�4����k�Y;�Flִ�S �6� ' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.527 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h��o)��ޱB�#�̗�����vA?����ݐ' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.527 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h���8�r�~2��t�H�,]��^h���' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.527 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h���4mv��Ѓw�ݑ��Gj����(s��'' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.527 [WRN] CHAN: Could not read bucket 'o�
�{�$�ֆH�<ّ��Q˚�\��w��9' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.527 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h���fM���UY�跂�ɱ�'�/���c`�{�' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.527 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h���]vLy�
U�����Ӊ,���}���}�l%�d' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.527 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h��uk)�Xu����o�����_��' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.527 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h��~�'~�!d?���NVm�'�W1t�P�
ojY6��' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
2020-10-26 00:16:52.527 [WRN] CHAN: Could not read bucket 'o�
��r���F�c�O��e��h��]�N���L�[�z���؋�w���m�WNJ' (full path '[graph-edge disabled-edge-policy-index]') database is likely corrupted. Continuing anyway but skipping corrupt bucket.
error compacting DB: bucket already exists
Error running chantools: error compacting DB: bucket already exists
admin@rock64:~/download/db_restore$

admin@rock64:~/download/db_restore$ ls -la channel.db
-rw-rw-rw- 1 root root 349,319,168 Oct 26 00:12 channel.db
admin@rock64:~/download/db_restore$ ls -la res2/compacted.db
-rw------- 1 admin admin 42,086,400 Oct 26 00:17 res2/compacted.db
admin@rock64:~/download/db_restore$

What do you think?

@guggero
Copy link
Collaborator

guggero commented Oct 26, 2020

That sounds like your channel database got corrupted pretty badly. It looks like the damage is mostly contained to the graph-edge bucket. But I'd still recommend to not continue with this DB.
I'm afraid the safest course of action is probably to restore from the channel backup (closes all channels!).

What kind of storage medium are you using?
It's very weird that a normal shutdown would cause this. Can you post more of the log from before 2020-10-25 04:53:41.089? There should be a critical error message that caused the shutdown.

@manreo
Copy link
Contributor Author

manreo commented Oct 26, 2020 •
edited
Loading

Ho man. That is bad.. I had ~1 BTC of channels in this node. Dam it.
Maybe I can use the etcd database to recover, it is functional? or did I had to turn it on?
Any other viable option?

I'm (was) running on Rock64 with a Toshiba 1TB

Full logfile is here (I've sent you the password in Slack):
lnd.zip

@Roasbeef
Copy link
Member

How did you typically shutdown this node? Are you aware of any recent power failures?

@manreo
Copy link
Contributor Author

manreo commented Oct 27, 2020 •
edited
Loading

I almost never shut it down.
I am not aware of power failures, but the external hard-drive made some noises at ~3am, so I (half asleep) did sudo shutdown (this is what I usually do) I thought that it sends a message to the services that a shutdown is imminent (1 min) and they shut themselves down. It does seems that this is what happen, but then it went back up.

@guggero
Copy link
Collaborator

guggero commented Oct 27, 2020

Thanks for the logs!

It looks like a shutdown was issued at 04:53:29.
And I do see that it was completed some seconds later:
2020-10-25 04:53:41.236 [INF] LTND: Shutdown complete

Am I correct in assuming that lnd's channel database was also stored on the external hard disk?
Could be there was some silent data corruption (the disk making strange noises is never a good sign!) and it was only detected on startup.

The etcd feature is still experimental and not turned on by default.

You do have a channel.backup file, right?

@manreo
Copy link
Contributor Author

manreo commented Oct 28, 2020

I have some backup. How can I access the wallet when I can not start the lnd service?

@guggero
Copy link
Collaborator

guggero commented Oct 28, 2020

You re-create your node from the seed, then apply the channel backup which will ask all your peers to force-close the channels.
It's all described here: https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md

Make sure you keep a copy of the old files around, just in case there are some zombie channels that can't be recovered with this procedure.

This was referenced Oct 28, 2020
Closed
Closed
@manreo
Copy link
Contributor Author

manreo commented Oct 28, 2020

I don't get it I need a another node? with a synchronized bitcoind? a bitcoin wallet is a small file, why it is even in the channel.db ?

@guggero
Copy link
Collaborator

guggero commented Oct 28, 2020 •
edited
Loading

See my comment in #4728:

This is already implemented. The on-chain wallet is a separate file (by default located in .lnd/data/chain/bitcoin/mainnet/wallet.db.
My comment in #4720 was imprecise. If you only need to recover the channels because of a database corruption in the channel.db file, you can just move or delete that file and access the on-chain wallet individually (but all channels will be gone until recovered from SCB).

I recommended doing a full restore from the seed because the data corruption from a failed hard drive could also affect the wallet.db file.

@manreo
Copy link
Contributor Author

manreo commented Oct 28, 2020 •
edited
Loading

Thanks, How can I check if the wallet.db is corrupted ?
By the way I also had this situation:
#4689

I assume that #4689 + me setting logging to DEBUG (#4613 (comment)) was too much for the hardware

@manreo
Copy link
Contributor Author

manreo commented Oct 29, 2020

@guggero
I get this when I try to unlock the wallet (And deleting channel.db):

admin@rock64:~$ lncli unlock 
Input wallet password: 
[lncli] rpc error: code = Unknown desc = missing address manager namespace

does it mean that the wallet was also corrupted

@guggero
Copy link
Collaborator

guggero commented Oct 29, 2020

Yes, the error missing address manager namespace comes from the wallet and means some data isn't there where it should.

@manreo manreo mentioned this issue Nov 3, 2020
Closed
@manreo
Copy link
Contributor Author

manreo commented Nov 3, 2020

Hi, I've started the recovery process but most of the channels are in this state:

    "waiting_close_channels": [
        {
            "channel": {
                "remote_node_pub": "0227d5b940cba21be92244953475ccdd3cefbed8f397be03e3155a5f41f304fc93",
                "channel_point": "b4dd90fc2978114e2485b0b78393bb4f5c450714115842447b457664d9d9bcf3:1",
                "capacity": "191804",
                "local_balance": "0",
                "remote_balance": "0",
                "local_chan_reserve_sat": "0",
                "remote_chan_reserve_sat": "0",
                "initiator": "INITIATOR_REMOTE",
                "commitment_type": "STATIC_REMOTE_KEY"
            },
            "limbo_balance": "0",
            "commitments": {
                "local_txid": "",
                "remote_txid": "",
                "remote_pending_txid": "",
                "local_commit_fee_sat": "0",
                "remote_commit_fee_sat": "0",
                "remote_pending_commit_fee_sat": "0"
            }
        },
        {
        {
            "channel": {
                "remote_node_pub": "0241e0f3e97edb6b3e80b5f1e779eba7808247af1a8c9324e592ef066e168ff833",
                "channel_point": "91457711330e7fe1c9669e84fe4a90ed08facc1f53412aad3596d747f27b304d:0",
                "capacity": "550000",
                "local_balance": "0",
                "remote_balance": "0",
                "local_chan_reserve_sat": "0",
                "remote_chan_reserve_sat": "0",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "STATIC_REMOTE_KEY"
            },
            "limbo_balance": "0",
            "commitments": {
                "local_txid": "",
                "remote_txid": "",
                "remote_pending_txid": "",
                "local_commit_fee_sat": "0",
                "remote_commit_fee_sat": "0",
                "remote_pending_commit_fee_sat": "0"
            }
        },
        {
            "channel": {
                "remote_node_pub": "024f8c0d5a49fedffd249ea1ce5050b500bd2d86286f5ee0acc0e9e2ed62aa6a67",
                "channel_point": "6e1d85000e1ebf1301ea9de9f0bf429b7baf75b1d79b9d7e0a9a3cd22450d9a8:0",
                "capacity": "5000000",
                "local_balance": "0",
                "remote_balance": "0",
                "local_chan_reserve_sat": "0",
                "remote_chan_reserve_sat": "0",
                "initiator": "INITIATOR_REMOTE",
                "commitment_type": "STATIC_REMOTE_KEY"
            },
            "limbo_balance": "0",
            "commitments": {
                "local_txid": "",
                "remote_txid": "",
                "remote_pending_txid": "",
                "local_commit_fee_sat": "0",
                "remote_commit_fee_sat": "0",
                "remote_pending_commit_fee_sat": "0"
            }
        },

Is that ok? is there anything else I should do?
Thanks

@manreo manreo mentioned this issue Nov 5, 2020
Closed
@guggero
Copy link
Collaborator

guggero commented Nov 30, 2020

@mrmanpew what's the status on this issue? I don't remember if you reached out to me on Slack or whether this is still unresolved?

@manreo
Copy link
Contributor Author

manreo commented Nov 30, 2020 •
edited
Loading

I managed to save some of the channels.. Still working on it. Probably will know better in a week or two.
Thanks.

@manreo
Copy link
Contributor Author

manreo commented Dec 15, 2020

The amazing @guggero helped me to recover all the funds. Thank you very much!

@manreo manreo closed this as completed Dec 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Roasbeef @guggero @manreo