Online backup

[IngoMeyer441]

We have a dedicated machine only for AiiDA-based workflows. Currently, there is a file system backup job based on BorgBackup to create nightly backup snapshots. However, these snapshots are not guaranteed to be consistent, since our users have long-running jobs which can take several days, so the AiiDA daemon is not stopped during the backup process. Is there any recommended way to handle this situation? What we need is some kind of online backup. I already thought about using file system snapshots (like LVM can do) but in this case, the database and file repository must also be synchronized before the snapshot is created.

[ltalirz]

Good question. Here are a few thoughts:

• File system snapshots sound like a good solution, and database dumps are often comparatively fast (use the "custom" format -Fc with pg_dump).
  Edit: just in case it was unclear, using the snapshot of the database directory is not a solution
• It probably makes sense to start with the database dump and do the repository backup afterwards.
  Having a few "orphaned" files in the repository (created after the database dump completed) should be unproblematic, and AiiDA's provenance model means that it is unlikely for files associated with a node to be edited/removed (notable exceptions are manual operations like verdi node delete, which could lead to an orphaned database record in the above approach).

I would expect that, for practical purposes, this approach works well enough - however, to my knowledge there is currently no solution that guarantees consistency between file repository and database in a "live" backup.

If that turns out to be important in practice, you or others may want to open an enhancement proposal for it.

[IngoMeyer441]

Thanks for your thoughts and the AiiDA storage details. Probably, creating a dump of the database and then freezing the file system state with a snapshot is sufficient for us if old data in the file repository is unlikely to change. IMHO, it is most important, that the backup can be used to restore an useable state. Some additional data shouldn't be problematic in this use case.

[IngoMeyer441]

For pg_dump it is probably best to combine -Fc with gzip or pigz and the --rsyncable flag:

pg_dump -Z0 -Fc <database_name> | pigz --rsyncable > pg.dump.gz

With this, Backup tools like Borg can apply data deduplication more efficiently. See https://www.aktau.be/2014/10/23/pg-dump-and-pigz-easy-rsyncable-backups-with-postgresql/ for a reference.

[ltalirz]

Also, on second thought I guess one actually does not need to wait for the database dump to complete - I believe (?) the database dump should reflect the content in the database at the time when the dump was initiated.

I.e. if both the (practically instantaneous) file system snapshot and the dump are initiated simultaneously, that should reduce the time window for inconsistencies to a minimum.

[zhubonan]

I use restic which I think is similar to Borg (also does deduplication). I use the pg_dump approach to for the database and back it up with the repository together.

Is there any problem stopping the daemon? Even if you have long-running jobs stopping the daemon should not be an issue. Just for the peace of mind I always do my backups with the daemon stopped.

[IngoMeyer441]

Thanks for your input @zhubonan. I didn't know that the daemon can be stopped while calculations are running. So in this case, I think a good backup strategy with a minimal daemon downtime would be

• Stop the AiiDA daemon
• Create (an instant) file system snapshot (e.g. with LVM)
• Start a database dump (and do not wait for completion)
• Start the daemon again
• Wait for the database dump to complete
• Backup the file system snapshot and the database dump
• Delete the file system snapshot and the dump

[zhubonan]

yes this would work I think. I usually do not bother using LVM and typicallying dumping the database is very fast. AiiDA usually do not delete files from the repository, so you should not have to worry about any inconsistency. For example, an "inconsistent" repository may only contain some extra (maybe incomplete) files that belongs to the nodes not yet in the dumped database, but this does not render the repository errorous.

What I do is just:

• Stop the AiiDA daemon
• dump the database
• do the backup
• Start the AiiDA daemon

This course has a bit more downtime.

In theory, you should be able to do just dump the database and run backup without even stopping the daemon.