Secure backup setup

[Yannik]

Hello everyone,

first, I want to state that I have read various issues (like #900) which were somehow related to this.

A secure backup setup should IMO have the following properties:

No single point of failure

If the device being backup up is compromised, the backup needs to stay in tact. If the backup destination is compromised, the host being backup needs to stay in tact.

The device being backed up should not be able to delete its own backup

Otherwise an attacker could encrypt your backups (ransomware), or just delete them.
As a result, any type of "push" backup is not viable, for the exception of the target being append-only. I know borg does this, however, it leads to practically keeping any amount of old backups, as disk space is never freed again. Additionally, periodically running borg in non-append-only mode would lead to issues if a client is compromised, but we don't know about it yet. (because during these periodic runs, the compromised clients backup deletion would be executed, see #2251, #1772).
Therefore, I'll now focus on pull-based backup setups.

The host pulling the backup should only have read-only access, restricted to the directories that are being backed up

Obviously, if the host pulling the backups has access to the host being backed up, a compromised backup-pulling host can just overtake all the hosts that it backs up. Therefore, it needs to have strict read-only access. However, this needs to be restricted to the directories that are being backed up. Otherwise, the backed up ssh key will grant immediate access. The often-proposed solutions with ssh -R take care of neither of this.

I'm currently rolling a backup solution based on rsnapshot which takes care of these issues (see https://github.com/Yannik/ansible-role-rsnapshot-backup-host, https://github.com/Yannik/ansible-role-rsnapshot-remote-host), but I would like to move to a solution which allows for differential file backups.

Is there any setup with borg backup which is actually secure?

[ThomasWaldmann]

Borg has push mode, but no built-in pull mode. Assumptions about pull mode being more safe than push are based on high security assumptions about the backup repo server, which might be correct or just a nice illusion.

Borg does have append-only mode, which avoids that a client can physically delete data.
It is a rather low level hack and not very comfortable to use (in case of a hack), but it does what it says.
The problem of actually having to know when you got hacked and not kill your backups when you still need them is a very fundamental one and not related to borg. Same goes for knowing whether your backups (still) contain what you expect them to contain.

Borg client being able to delete backups does not imply ransomware being able to encrypt your backup repo. This is only (as always) the case if you allow fs access to the repo (e.g. local directory, samba, nfs, sshfs, ...), but does not apply to borg client/server mode (ssh: repo url).

Good idea about the pull-mode that is restricted to read-only and only to the to-be-backed-up directories.
But as long as we have no pull mode, this is just a enhancement request for a non-existing feature.

rsnapshot has its own issues btw. due to its low tech approach, guess you can find them on the internet yourself.

A relatively safe borg setup using the currently existing features would:

• use client/server, authorized_keys on repo server with restrictions (ssh-key only, append-only, only access to own repo, ...)
• alternatively to borg append-only you could also do snapshots on fs level on the repo server, e.g. with zfs
• either never prune (safe, but space consuming) or make sure your repo/archives are in good state, use separate ssh key and no append-only for prune
• as it is push, you don't have the issue of the repo server being root on any backup client

[Yannik]

Actually I thought about the snapshot approach. Even though I don't have experience with zfs, my previous experiences with snapshots is, that it is a right mess to free up space as well. IMO snapshots duplicate the features that should be provided by differential backup software. Otherwise (this is a simplification!) I could just rsync and create a snapshot every night.

The approach you suggest, using append-only, has the issues we both already mentioned:

• Either never prune (not really viable)
• or ensure that repo/archives are in a good state, which, depending on the level of sophistication of the attacker (we are talking about penetrating a specific user account on a linux system, which should not be too easy to begin with), can be very hard, and will be, in any case, time consuming (do this for X repos)

Neither the former nor the latter issue really make borg a viable backup solution for production. This is not a rant, I would really like to use it, but it appears to me, that it is not usable with security in mind right now.

Is this a concious decision? I see that there a quite a few bountys posted by borgbackup on various (imo) not-really-important things, but there seems to be no traction on this topic (and related bugs #2251, #1772) for a long time.

[ThomasWaldmann]

It's a matter of priorities:

• 1.0: fixes and essential, small features
• 1.1: security improvements, more fixes, a few bigger features, tuning
• 1.2: see hydrogen milestone (master branch currently)

The client/server code is complex (esp. its async parts), hard to test and issues with it hard to debug, so the prio with that was also rather to stabilize/fix it and make it easier to add new rpc methods / params without breaking old clients than to vastly extend or even duplicate it to implement pull mode.

[Yannik]

Hey @ThomasWaldmann, do you see any improvements with regards to these issues in the coming months?

[ThomasWaldmann]

The coming months will focus on 1.2, thus: no.

[mlbarrow]

@Yannik I think your characterization is a bit harsh ("...really make borg a viable backup solution for production"). There are many of us out here in the wild Internet that are successfully using it in production, as it's not difficult to amend borg with tools (e.g., snapshots) to satisfy the stated goals.

It sounds like you are looking for a turn-key solution with a single tool that meets your needs; there are numerous offerings on the market that would be viable for production, per your definition.

[mcfaddendavid]

@ThomasWaldmann

Assumptions about pull mode being more safe than push are based on high security assumptions about the backup repo server, which might be correct or just a nice illusion.

I don't think this is an illusion, but a basic imperative for designing any backup setup. Of course, it's up to the administrator to ensure that this is indeed the case. But if your servers are more likely to be hacked that your clients, that's a big issue.

The offline backup is also just another level of security, requiring manual intervention. That way you have three escelating levels of security:

• clients: protected by (often insecure) passwords that users frequently type every day (vulnerable to phishing and logging)
• backup-server: protected by the admin's (hopefully secure) password and any additional keys
• offline backup: protected by physical access to the storage and any additional encryption keys.

[Yannik]

@ThomasWaldmann I would like to bump this issue, as unfortunately, rsnapshot, which I've so far used for secure pull-backups, is being removed from debian in the next debian release (rsnapshot/rsnapshot#279)
I would very much like to switch to borg, but I'm really hard pressed about doing that, with the basic backup security parameters outlined in my OP not being possible :(

[ThomasWaldmann]

IIRC we have pull mode in our docs meanwhile and #2125 might help with the read-only part.

https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html#ssh-agent

[jonathanmmm]

Hi,
just thinking how to make borgbackup secure. Would it work, if the backup server has LVM/BTRFS file system an I do regular snapshots of the folder where all clients save their data? That way I could just keep append-only off and let clients do pruning, but LVM/BTRFS are actually backing up.

So in an event, that a client is corrupted and deletes all his backups or gets his file encrypted and pushes them into the repo and prunes old backups. I could just mount/look at an old snapshot, before this corruption and be able to restore from there, right?

This is a way, to ensure the backups are safe, at this stage of borg backups release. Of course I have to trust the backup server not to delete the snapshots, but that also is the same with the regular backups. I would just have to create a script that just has like a retention policy, like deleting LVM/BTRFS snapshots older than a month/year whatever would seem a good compromise.
Question is of course, how much and how fast could the disk of the backup server fill up, as every file that changes a bit, creates complete new blocks on the server (as they are encrypted).

But would that be still a viable and safe solution, until append-only or so forbids prune/delete commands?

Or what about local borg backup on the server to another folder without encryption (as the files are encrypted already), that way I could like a snapshotting structure even on disks that don't have LVM/BTRFS?