Backup is out of date, we cannot continue safely. Emergency shutdown.

[manelio]

I'm using lightningd via raspiblitz.

After restarting after a power failure lightningd is not running anymore.

When I try to start the service manually I receive an error message Backup is out of date, we cannot continue safely. Emergency shutdown.:

sudo -u bitcoin /usr/local/bin/lightningd --conf=/home/bitcoin/.lightning/config
Plugin '/home/bitcoin/cl-plugins-enabled/backup.py' returned an invalid response to the db_write hook: 

{"jsonrpc": "2.0", "id": 11, "error": {"code": -32600, "message": "Error while processing db_write: 'Process' object has no attribute 'parents'", "traceback": "Traceback (most recent call last):\n  File \"/home/bitcoin/.local/lib/python3.7/site-packages/pyln/client/plugin.py\", line 621, in _dispatch_request\n    result = self._exec_func(method.func, request)\n  File \"/home/bitcoin/.local/lib/python3.7/site-packages/pyln/client/plugin.py\", line 606, in _exec_func\n    return func(*ba.args, **ba.kwargs)\n  File \"/home/bitcoin/cl-plugins-enabled/backup.py\", line 63, in on_db_write\n    assert(check_first_write(plugin, change.version))\n  File \"/home/bitcoin/cl-plugins-enabled/backup.py\", line 56, in check_first_write\n    kill(\"Backup is out of date, we cannot continue safely. Emergency shutdown.\")\n  File \"/home/bitcoin/cl-plugins-enabled/backup.py\", line 110, in kill\n    procs = [p for p in psutil.Process(os.getpid()).parents()]\nAttributeError: 'Process' object has no attribute 'parents'\n"}}

Is there any way to fix this problem?

[cdecker]

There are two things here:

1. the psutils version seems to not have the .parents property (old?) so we can't locate the root process and kill it.
2. the outdated backup, which seems to have lagged behind the live database. Notice that the crash is intentional, since the DB being out of sync with the backup is not automatically recoverable. This can happen if you run without the backup plugin and then re-enable it, resulting in missing writes in the backup for the time it wasn't configured.

The latter can be addressed by removing the backup and re-initializing it. Can you report the pip freeze output to check what versions are installed?

[SimonVrouwe]

@manelio Please could you also report lightningd (or raspiblitz) version you are running?

[Coinomatron]

Hi, I just did a clean shutdown and have the same issue after a reboot, Ubuntu 20. Not sure what to do.
pip freeze output here: https://pastebin.com/b5MSKCzR

[Coinomatron]

Was thinking of just moving the file.bkp and writing to a new backup file, but I'm not sure if that won't mess with the .lightning datadir...

Edit, missed the part where you said "The latter can be addressed by removing the backup and re-initializing it".

Edit2, worked, although I'm really fuzzy on python, unfortunately.

[cdecker]

Ok, seems it was indeed an older version of psutils and an intermittent start that didn't update the backup accordingly. Thanks for confirming that the fix works 👍

[Coinomatron]

Not to reopen this, but I only yesterday read that lightningd has a built in backup functionality that can write to a second database. Is that the preferred way? Using the plugin now and plan to rewrite the backup file occasionally when the file becomes, lets say 3 or 4 times the .lightning dir size.

[cdecker]

Indeed there now is a litestream based option, but I have no idea how it works under the covers and I personally still prefer my backup plugin. I'll add the replay-to-secondary-db as an option too.