Option To Exclude Certain Directories Or Features From Backup

[DrShivang]

Hi,
Consider An Option To Exclude Certain Directories & / Or Features From Backing Up.
Specifically The Nextcloud Data Directory.

As In A Scenario Where I Am Using External HDD as Data Directory For Nextcloud
So In the Chances of Some Container or System Crash, It would Generally Be Safe.
And It's Supposed to Grow Up To A Few TBs in the coming months.
So Backing Up A Few Hundred GBs or a Couple of TBs which is already on some external medium will just unnecessarily occupy double space.

If we can have some option to exclude the data directory from borg backup, the backups will still have everything like settings databases containers etc and will not occupy huge spaces for externally mounted mediums as such those can be mounted again using external mount setting.
Presently we have a setting to externally mount the nextcloud directory and it's being backed up by Borg.

We can add an option in AIO Backup Settings or Even some workarounds will work, it will be beneficial for anybody who wants to skip the externally mounted directory to be skipped from backups.

[szaimen]

But what if the external HDD dies?

[DrShivang]

Any system can crash.. Thats why we had an option to create backup on external HDD.. to create offline backups.. But for regular backup, like weekly or before system updates .. where chances of externally mounted directory crashing are least.. we can avoid lengthy huge backups by giving an option of ignoring externally mounted data directory or data directory itself. What will be achieving is faster backups of all containers with all the settings which can be restored in a click in cases of system crash. And as such as data directory is externally mounted it will be available till the time its mounted. As system updates don't touch data directory as far as am aware.. So, why backup user data directory when its not required? Giving it as an option before initiating backup so that user can decide whether to backup data directory as well or skip it.. incase user feels its safe, backed up or already duplicated.. Makes sense ?

…

On Sat, 2 Apr, 2022, 05:08 Simon L., ***@***.***> wrote: But what if the external HDD dies? — Reply to this email directly, view it on GitHub <#428 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWOTAJNQ5QGBNDN3DIEVDZLVC6CHPANCNFSM5R2JUBTQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[szaimen]

Makes sense ?

No, it doesn't. You will cry if the hdd someday breaks or you get a virus or any other unexpected thing happens when you haven't made a backup that includes your files as well.

[DrShivang]

Thanks for your reply, Will try that tagging option.. I understand what you mean, i just didn't wanted the backup app to backup whole of my data directory everytime before an update. As i mentioned before, the data directory is few TBs and I can't afford to have multiple / frequent backups of the same. I do keep one master backup for sure, just for the sake of reason as you mentioned above - not crying one day in event of hdd crash etc.. 😀

…

On Mon, 4 Apr, 2022, 00:21 Simon L., ***@***.***> wrote: If you really want to exclude your or some data, you can already do so by using e.g. the external storage app or put a specific file into directories that shall not be backed up. This file must be conform to https://bford.info/cachedir/ But as I said, it is not recommended to exclude you data from those backups. At least as long as you don't have a backup of this data. — Reply to this email directly, view it on GitHub <#428 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWOTAJP7ISJMQGUKHMCJ5YDVDHSCJANCNFSM5R2JUBTQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[szaimen]

As i mentioned before, the data directory is few TBs and I can't afford to have multiple / frequent backups of the same.

Why is that? Did you know that the program that we use for backups is very efficient in storing multiple backups? In my testing and although I run daily backups, the whole backup archive rarely uses more than the originals data size. So you should be save creating subsequent backups. You can read more about this here: https://github.com/borgbackup/borg#what-is-borgbackup and https://github.com/borgbackup/borg#main-features

[szaimen]

So if you already have one backup you will be able to simply create a new backup without it needing additional space (as long as the files don't change)
Only the changes to files will be stored in new backups from then on.

[szaimen]

@DrShivang feel free to try that out :)

[szaimen]

And now I see where this feature request comes from: you didn't know how good borg really is :)

[DrShivang]

And now I see where this feature request comes from: you didn't know how good borg really is :)

You caught me red-handed

So if you already have one backup you will be able to simply create a new backup without it needing additional space (as long as the files don't change) Only the changes to files will be stored in new backups from then on.

This just completely skipped off my head.
I need to read about the borg first..

[szaimen]

Just to add an answer and mark this discussion as answered:

We will not add this feature since the backup tool that we use is very efficient in storing multiple backups.

[gadric]

I have a rsync backup for all my data running so I was also looking for a way to exclude my data dir from backup.

I was wondering, how can I effectively restore my Nextcloud AIO instance (only the instance, not the data dir)? would it be enough to copy back the docker volumes and shoot up the master container?

[szaimen]

I was wondering, how can I effectively restore my Nextcloud AIO instance (only the instance, not the data dir)? would it be enough to copy back the docker volumes and shoot up the master container?

Principally, yes

Here is what we do internally:

all-in-one/Containers/borgbackup/backupscript.sh

Lines 158 to 238 in bba2675

	# Do the restore
	if [ "$BORG_MODE" = restore ]; then
	get_start_time
	# Perform the restore
	if [ -n "$SELECTED_RESTORE_TIME" ]; then
	SELECTED_ARCHIVE="$(borg list "$BORG_BACKUP_DIRECTORY" | grep "nextcloud-aio" | grep "$SELECTED_RESTORE_TIME" | awk -F " " '{print $1}' | head -1)"
	else
	SELECTED_ARCHIVE="$(borg list "$BORG_BACKUP_DIRECTORY" | grep "nextcloud-aio" | awk -F " " '{print $1}' | sort -r | head -1)"
	fi
	echo "Restoring '$SELECTED_ARCHIVE'..."
	mkdir -p /tmp/borg
	if ! borg mount "$BORG_BACKUP_DIRECTORY::$SELECTED_ARCHIVE" /tmp/borg; then
	echo "Could not mount the backup!"
	exit 1
	fi
	# Restore everything except the configuration file
	if ! rsync --stats --archive --human-readable -vv --delete \
	--exclude "nextcloud_aio_mastercontainer/session/"** \
	--exclude "nextcloud_aio_mastercontainer/certs/"** \
	--exclude "nextcloud_aio_mastercontainer/data/daily_backup_running" \
	--exclude "nextcloud_aio_mastercontainer/data/configuration.json" \
	/tmp/borg/nextcloud_aio_volumes/ /nextcloud_aio_volumes; then
	echo "Something failed while restoring from backup."
	umount /tmp/borg
	exit 1
	fi
	# Save current aio password
	AIO_PASSWORD="$(jq '.password' /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json)"
	# Save current path
	BORG_LOCATION="$(jq '.borg_backup_host_location' /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json)"
	# Save current nextcloud datadir
	if grep -q '"nextcloud_datadir":' /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json; then
	NEXTCLOUD_DATADIR="$(jq '.nextcloud_datadir' /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json)"
	else
	NEXTCLOUD_DATADIR='""'
	fi
	# Restore the configuration file
	if ! rsync --archive --human-readable -vv \
	/tmp/borg/nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json \
	/nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json; then
	echo "Something failed while restoring the configuration.json."
	umount /tmp/borg
	exit 1
	fi
	# Set backup-mode to restore since it was a restore
	CONTENTS="$(jq '."backup-mode" = "restore"' /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json)"
	echo -E "${CONTENTS}" > /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json
	# Reset the backup path to the currently used one
	CONTENTS="$(jq ".borg_backup_host_location = $BORG_LOCATION" /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json)"
	echo -E "${CONTENTS}" > /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json
	# Reset the AIO password to the currently used one
	CONTENTS="$(jq ".password = $AIO_PASSWORD" /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json)"
	echo -E "${CONTENTS}" > /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json
	# Reset the datadir to the one that was used for the restore
	CONTENTS="$(jq ".nextcloud_datadir = $NEXTCLOUD_DATADIR" /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json)"
	echo -E "${CONTENTS}" > /nextcloud_aio_volumes/nextcloud_aio_mastercontainer/data/configuration.json
	umount /tmp/borg
	# Inform user
	get_expiration_time
	echo "Restore finished successfully on $END_DATE_READABLE ($DURATION_READABLE)"
	# Add file to Nextcloud container so that it skips any update the next time
	touch "/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/skip.update"
	chmod 777 "/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/skip.update"
	# Add file to Nextcloud container so that it performs a fingerprint update the next time
	touch "/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/fingerprint.update"
	chmod 777 "/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data/fingerprint.update"
	fi

[gadric]

Hmm, I might have overlook something, but are you restoring only the nextcloud docker volume? What are DB etc.?

[szaimen]

we are restoring everything. See

all-in-one/php/containers.json

Lines 237 to 266 in bba2675

	{
	"name": "nextcloud_aio_nextcloud",
	"location": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud",
	"writeable": true
	},
	{
	"name": "%NEXTCLOUD_DATADIR%",
	"location": "/nextcloud_aio_volumes/nextcloud_aio_nextcloud_data",
	"writeable": true
	},
	{
	"name": "nextcloud_aio_database",
	"location": "/nextcloud_aio_volumes/nextcloud_aio_database",
	"writeable": true
	},
	{
	"name": "nextcloud_aio_database_dump",
	"location": "/nextcloud_aio_volumes/nextcloud_aio_database_dump",
	"writeable": true
	},
	{
	"name": "nextcloud_aio_apache",
	"location": "/nextcloud_aio_volumes/nextcloud_aio_apache",
	"writeable": true
	},
	{
	"name": "nextcloud_aio_mastercontainer",
	"location": "/nextcloud_aio_volumes/nextcloud_aio_mastercontainer",
	"writeable": true
	},

[gadric]

my bad.

[gadric]

I just saw #695 and #1046.

Since it is possible to backup more directories, I wanted to raise the question if it makes sense also to disallow the data directory it self.

The reason I am requesting this again is that I am rethinking my backup strategy and with borg doing all the backup, some common scenarios are hard to cover. One of which is actually that sometimes one directory is messed by one person and I have to restore it from the backup. With borg, I have to restore the whole instance, which takes long time with over 1TB.

From my point of view, it would make sense to backup the DB and configs and everything else, but exclude the data directory. If you have to restore one directory, then the admin can restore it by copy back everything and do a scan. If the admin has to restore the whole instance, then the admin has to restore the borg backup and then copy backup all files, and then start everything.

This would provide much more flexibility but of course would be for more advanced users. I would contribute to this solution. However, I need some support to understand where to start.

[gadric]

And I would also like to start the backup from the backup server via this command: sudo docker exec -it nextcloud-aio-mastercontainer DAILY_BACKUP=1 /daily-backup.sh

This looks good but why not run the command from the server that runs AIO itself? Otherwise you need to connect from the smb server to the server that runs AIO using SSH and trigger the command then but this is more complicated then simply triggerring it via a cronjob or shell script from the server that runs AIO.

Well this is because of the setup I have here. I have 2 servers, on the first one, I have an Nextcloud and SmartHome etc. running. This server runs 24/7. The second one is more or less a backup server with a couple of services which I do not need allday. It is normally in sleep mode.

So everything night at 11pm, it wakes up and start the backup via rsync. Server 2 connects to Server 1 and pulls all data that needs to be backed up and then sync it to an external disk and also do an encrypted bacup in the cloud. All this runs in a defined sequence, otherwise the sync to external disk or cloud might be broken. That is why I run everything from a script.

Last question: how to disable the automatic backup via GUI to start it on a specfic time if it has been already set up?

Simply disable the option again from the AIO interface?
So just leave it empty and click on submit?

[szaimen]

So just leave it empty and click on submit?

If the field is empty, is the option already disabled. You enable automatic backup and updates by typing in a time and clicking on submit.

[gadric]

So just leave it empty and click on submit?

If the field is empty, is the option already disabled. You enable automatic backup and updates by typing in a time and clicking on submit.

I meant actually the other way around :) It is set but I want to disable it :)

[szaimen]

Then simply click on the Disable daily backups button?

[gadric]

I am running currently into the following issue: #1053

[giovacode]

Sorry to raise this ticket again, but this is crutial for me to exclude NEXTCLOUD DATADIR from borg backups, because this folder takes 2TB (this is a mounted iscsi) and is allready persisted on 2 Synology.

Because Synology sucks (no freedom), nextcloud-aio-borgbackup is not compatible with my architecture constraint.

My Goal is to backup config + bdd only, then migrate this on a second server witch has allready Nextcloud Datadir clone.
So if my primary server crash, i can switch to the spair server.
Is there a doc on how to migrate config + bdd only?

I tried to modify backupscript.sh with no luck :

VOLUME_DIRS="$(find /nextcloud_aio_volumes -mindepth 1 -maxdepth 1 -type d -not \( -path /nextcloud_aio_volumes/nextcloud_aio_nextcloud_data -prune \))"

[szaimen]

If you already have an external backup solution in place, simply use that one for backup and restore...

You should make sure to stop the container correctly then. See https://github.com/nextcloud/all-in-one#how-to-stopstartupdate-containers-or-trigger-the-daily-backup-from-a-script-externally

When using your own backup solution you can also choose what to back up and not to back up.

[archont00]

Another use case: users may have files, which are huge and there is no need for a backup (like Android ROM images). So it is not about great Borg Backup deduplication feature, it is about no need for backup at all.

Before migrating from my own Nexctloud docker instance to Nextcloud AIO, I used borgmatic which was configured to exclude any dirs with .nobackup file. Worked well across all users.

It would be nice to see something like that in AIO.

EDIT: As nextcloud-aio-borgbackup image uses --exclude-caches option for borg create, it is possible to put a file CACHEDIR.TAG in directory, which should be ignored for backup purposes. A bit ugly workaround to the --exclude-if-present borg option, however usable.

https://bford.info/cachedir/

[marmeladapk]

This has an unfortunate side effect that if you're restoring the backup after a botched update then borg will delete all files in folders marked by CACHEDIR.TAG file. In my case it was fine since I just restored a zfs snapshot, but without that I would loose my files needlesly.