channel.db is over 1.5gb after DeleteAllPayments and recompacting! Can I do anything else? #5705
Comments
|
This PR should help to clear out a lot of state: #4364 How old is your node? Number of channels as well? Do you have automated GC of invoices activated? If you have a lot of invoices, then these flags can be used to clear them out either on start up, or on an on going basis. Another flag you can use is: With that your node will keep a more compact graph that is more live. |
|
@Roasbeef thanks for the response! The node is 7 months old and has over 100 channels. The PR appears (at a casual glance) to apply to closed channels, which will certainly help, but most of my channels are both old and still open. I do already have both of the above invoice GC options enabled via config file. I have not enabled strictgraphpruning. Do you expect that will have a very significant impact? What is it that eats up so much space that can't be reclaimed? More importantly, does it scale with the number of payments? With the number of channels? It would be helpful to know whether I should discourage or reduce either to extend the lifetime of the node. Thanks again for your time and patience! |
|
My node is a few months old, few channels... I have been monitoring the channels.db file and it had been around 300MB. A couple days ago I noticed that it started to grow. In the last few days I haven't open any new channels, but I did get one force closed by me (not on purpose, it just closed and now it shows that I did the force close). In the last days I have also activated the watchtower client with one tower and updated to 0.13.3. But I cant't tell for sure if the size increases were before or after any of these changes... I am at 512MB now, it is growing ~50MB per day. |
|
Mine is 16GB for like 100 routings, 60 channels, 60 days uptime. V 0.13.3 on Ryzen7, Ubuntu 20. btc@xxx:~/.lnd$ ls -al ./data/graph/mainnet/channel.db Using settings from Alex' rundown here: https://github.com/alexbosworth/run-lnd What needs to be done here? Is this a problem for my funds...? |
|
Ok, today I have This is 1 GB per two days. I don't think that's healthy. Any explanation? Thanks. |
|
One thing you can do is look at the number of state updates on your channels and reduce those by closing/reopening |
Monitoring the channel.db size is critical as a very large DB can lead to node crashes (and potentially DB corruption & SCB recovery) on low-resource devices like Raspberry Pis (e.g. lightningnetwork/lnd#5705). This update adds a line to the LND section of the welcome script to show the channel.db size For now the size is obtained by: channeldb=$(du -h /mnt/ext/lnd/data/graph/mainnet/channel.db | head -c4) It would be better to get a numerical value and then add some simple logical tests to colour the size green, orange or red when it reaches a certain large size (e.g. 500MN orange, 1GB red). What would be the best way to do this? (i.e. get a numerical value for the db size rather than a string like now)
Why type of activity is your node even doing? is it all just failed rebalance attempts? |
|
Today I have Growing is up to 1 GB per day... Thanks for your suggestions! @Roasbeef No, I am not rebalancing, tried it about 10 times some weeks ago but stopped then. @alexbosworth Can you give me some search terms for the log to look for? Grepping for ERR gives just Looks like problems with TOR I think Grepping for WRN gives just I have a home-made python script that queries (read-only macaroon) some data every 20 mins. lnd logs this for it What I really have a lot is log like this: Any help appreciated! Thanks! |
|
@feikede did you ever compact your DB (add |
|
@guggero Thanks, I'll try it an report the result. btw, whats also in the log is this (removed the channel points - all different) |
|
@guggero Hm, I read about the compression. Maybe I do it another time because it's more of a workaround not a solution.... |
|
It's not a compression. Not sure what you exactly read, but it's the only way the underlying database format (bbolt) actually returns disk space. Otherwise your DB will never shrink, only grow. |
|
@alexbosworth is this the requested state update count? Are these high numbers or is it just normal? |
|
@guggero You're the man! It's down from 25 GB to 140 MB... That's cool! Now I'll look if it still grows by 1 GB per day. Big Thanks! |
|
So... it all comes back to the fact that channels are meant to be "short lived" as it accrues num_updates ? |
Oh wow, that's a lot of updates! I would say with the current way the forwarding log is stored, it basically comes down to the number of updates, yes. What #4364 did was remove that forwarding log for closed channels. |
|
yeah, well... I have quite a good routing flow and I'm still at 10M inbound deficit. thank you @guggero for starting this discussion, I think we just got some drag here... to be honest, i foresaw this as many peers got down gradually. One concern I have is that out of 8 peers which i am in direct contact, only 3 of them came back after the channel.db issues, most of them quit running node... as they realised the risks involved. So, if I understand correctly, we might have to introduce some similar "taproot" technology to forwardings of active channels to try and then a similar "purge" for data, perhaps like bitcoind purge does? |
Don't look too high so the cause of larger size is probably past payment failure states that are logged to the database |
you would probably notice a difference in db size by closing and reopening the largest num update channels |
|
Looks like my channel.db problem is solved for now. It grows by about 5 MB per day now - far below the 1 GB two days before. Lessons learned for me: Add db.bolt.auto-compact=true to lnd.config and don't have 50 days uptime. |
|
Mine is also under control with the compaction. It went from ~500MB to ~100MB and then stabilized around 150MB. |
That would be great, @guggero! My channel.db is at 9.3GB after compaction (and after |
* Add channel.db size to Welcome script Monitoring the channel.db size is critical as a very large DB can lead to node crashes (and potentially DB corruption & SCB recovery) on low-resource devices like Raspberry Pis (e.g. lightningnetwork/lnd#5705). This update adds a line to the LND section of the welcome script to show the channel.db size For now the size is obtained by: channeldb=$(du -h /mnt/ext/lnd/data/graph/mainnet/channel.db | head -c4) It would be better to get a numerical value and then add some simple logical tests to colour the size green, orange or red when it reaches a certain large size (e.g. 500MN orange, 1GB red). What would be the best way to do this? (i.e. get a numerical value for the db size rather than a string like now) * updated variable name I changed the variable name from 'channeldb' to channel_db_size' to make it more meaningful. * Update 20-raspibolt-welcome Replaced "head -c4" with "awk '{print $1}'" as advised by Pablo [here](https://t.me/raspibolt/3521) as a most robust way of getting the relevant section of the output (thanks to him!) * Changed title to 'Channel.db size:'
* Add channel.db size to Welcome script Monitoring the channel.db size is critical as a very large DB can lead to node crashes (and potentially DB corruption & SCB recovery) on low-resource devices like Raspberry Pis (e.g. lightningnetwork/lnd#5705). This update adds a line to the LND section of the welcome script to show the channel.db size For now the size is obtained by: channeldb=$(du -h /mnt/ext/lnd/data/graph/mainnet/channel.db | head -c4) It would be better to get a numerical value and then add some simple logical tests to colour the size green, orange or red when it reaches a certain large size (e.g. 500MN orange, 1GB red). What would be the best way to do this? (i.e. get a numerical value for the db size rather than a string like now) * updated variable name I changed the variable name from 'channeldb' to channel_db_size' to make it more meaningful. * Update 20-raspibolt-welcome Replaced "head -c4" with "awk '{print $1}'" as advised by Pablo [here](https://t.me/raspibolt/3521) as a most robust way of getting the relevant section of the output (thanks to him!) * Changed title to 'Channel.db size:'
* Add channel.db size to Welcome script Monitoring the channel.db size is critical as a very large DB can lead to node crashes (and potentially DB corruption & SCB recovery) on low-resource devices like Raspberry Pis (e.g. lightningnetwork/lnd#5705). This update adds a line to the LND section of the welcome script to show the channel.db size For now the size is obtained by: channeldb=$(du -h /mnt/ext/lnd/data/graph/mainnet/channel.db | head -c4) It would be better to get a numerical value and then add some simple logical tests to colour the size green, orange or red when it reaches a certain large size (e.g. 500MN orange, 1GB red). What would be the best way to do this? (i.e. get a numerical value for the db size rather than a string like now) * updated variable name I changed the variable name from 'channeldb' to channel_db_size' to make it more meaningful. * Update 20-raspibolt-welcome Replaced "head -c4" with "awk '{print $1}'" as advised by Pablo [here](https://t.me/raspibolt/3521) as a most robust way of getting the relevant section of the output (thanks to him!) * Changed title to 'Channel.db size:'
* Add channel.db size to Welcome script Monitoring the channel.db size is critical as a very large DB can lead to node crashes (and potentially DB corruption & SCB recovery) on low-resource devices like Raspberry Pis (e.g. lightningnetwork/lnd#5705). This update adds a line to the LND section of the welcome script to show the channel.db size For now the size is obtained by: channeldb=$(du -h /mnt/ext/lnd/data/graph/mainnet/channel.db | head -c4) It would be better to get a numerical value and then add some simple logical tests to colour the size green, orange or red when it reaches a certain large size (e.g. 500MN orange, 1GB red). What would be the best way to do this? (i.e. get a numerical value for the db size rather than a string like now) * updated variable name I changed the variable name from 'channeldb' to channel_db_size' to make it more meaningful. * Update 20-raspibolt-welcome Replaced "head -c4" with "awk '{print $1}'" as advised by Pablo [here](https://t.me/raspibolt/3521) as a most robust way of getting the relevant section of the output (thanks to him!) * Changed title to 'Channel.db size:'
* Monitoring channel.db size in welcome/motd script (#809) * Add channel.db size to Welcome script Monitoring the channel.db size is critical as a very large DB can lead to node crashes (and potentially DB corruption & SCB recovery) on low-resource devices like Raspberry Pis (e.g. lightningnetwork/lnd#5705). This update adds a line to the LND section of the welcome script to show the channel.db size For now the size is obtained by: channeldb=$(du -h /mnt/ext/lnd/data/graph/mainnet/channel.db | head -c4) It would be better to get a numerical value and then add some simple logical tests to colour the size green, orange or red when it reaches a certain large size (e.g. 500MN orange, 1GB red). What would be the best way to do this? (i.e. get a numerical value for the db size rather than a string like now) * updated variable name I changed the variable name from 'channeldb' to channel_db_size' to make it more meaningful. * Update 20-raspibolt-welcome Replaced "head -c4" with "awk '{print $1}'" as advised by Pablo [here](https://t.me/raspibolt/3521) as a most robust way of getting the relevant section of the output (thanks to him!) * Changed title to 'Channel.db size:' * update MOTD welcome script for v3 The MOTD welcome script grew significantly over time and does not follow best shell script practices. This change updates it to reflect the setup of RaspiBolt v3 and improves overall script quality. * system overview: trap Ctrl-C for clean exit When the user presses Ctrl-C, the color encoding might disrupt the regular command line. This change traps the Ctrl-C abort commands SIGINT and SIGTERM, and exits the script cleanly. * system overview: address pr feedback Co-authored-by: Indra <86242283+VajraOfIndra@users.noreply.github.com>
I am trying the same,by adding that line in lnd.conf. Update: worth it! I obtained 500MB from 2GB size. In exactly 2h30m waiting. |
|
@Darth-Coin Well waited! When you Ctrl-C that process you'd have to emergency restore your channels - force close all ... never ctrl-C that :-) |
|
@feikede Nooo, I didn't even touch it. Just looking to the logs normally. Running an Umbrel node (dockers). I was tempted once to stop the starting process, but move along immediately, I knew that if nothing other things appear wrong, I should just wait. I had 2 channels in force close after that, but not sure what could cause that. Maybe the long time waiting to start, I don't know. |
* Monitoring channel.db size in welcome/motd script (#809) * Add channel.db size to Welcome script Monitoring the channel.db size is critical as a very large DB can lead to node crashes (and potentially DB corruption & SCB recovery) on low-resource devices like Raspberry Pis (e.g. lightningnetwork/lnd#5705). This update adds a line to the LND section of the welcome script to show the channel.db size For now the size is obtained by: channeldb=$(du -h /mnt/ext/lnd/data/graph/mainnet/channel.db | head -c4) It would be better to get a numerical value and then add some simple logical tests to colour the size green, orange or red when it reaches a certain large size (e.g. 500MN orange, 1GB red). What would be the best way to do this? (i.e. get a numerical value for the db size rather than a string like now) * updated variable name I changed the variable name from 'channeldb' to channel_db_size' to make it more meaningful. * Update 20-raspibolt-welcome Replaced "head -c4" with "awk '{print $1}'" as advised by Pablo [here](https://t.me/raspibolt/3521) as a most robust way of getting the relevant section of the output (thanks to him!) * Changed title to 'Channel.db size:' * update MOTD welcome script for v3 The MOTD welcome script grew significantly over time and does not follow best shell script practices. This change updates it to reflect the setup of RaspiBolt v3 and improves overall script quality. * system overview: trap Ctrl-C for clean exit When the user presses Ctrl-C, the color encoding might disrupt the regular command line. This change traps the Ctrl-C abort commands SIGINT and SIGTERM, and exits the script cleanly. * system overview: address pr feedback Co-authored-by: Indra <86242283+VajraOfIndra@users.noreply.github.com>
|
This should be solved now after the update to 0.15.x that includes the optional revocation log migration. |
|
@guggero I confirm, from 27GB I am running with 780MB now... |
|
101G -> 130M!!!! thank you!!!! |
Background
After running DeleteAllPayments and compacting on my workhorse machine with chantools and moving back to the node, my channel.db lists as follows:
ls -lAF ~/.lnd/data/graph/mainnet/
-rw------- 1 bitcoin bitcoin 1564844032 Sep 9 17:58 channel.db
My channel.db is too large to compact on my node and must be moved to a stronger machine to run chantools. Ultimately this is not sustainable if the database cannot ever be reduced in size enough to compact it on the node itself. When it swells over about 2.1 gb, lnd will crash with a paging error.
Is there anything else I can do? The only option to clear data through the API seems to be DeleteAllPayments, which I use liberally, but it doesn't seem to slow the growth or allow significant gains when compacting.
Is lnd simply not intended for light hardware? Do I need to migrate to C-Lightning if I intend to continue to run an RPi node, and are they are any tools for such a migration?
Thanks in advance for any help!
Your environment
Raspberry Pi 4 running DietPi
lnd version 0.13.1-beta commit=v0.13.1-beta
Linux dietnode 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 GNU/Linux
Bitcoin Core version v0.21.1
The text was updated successfully, but these errors were encountered: