-
-
Notifications
You must be signed in to change notification settings - Fork 753
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Very slow backup speed when creating archive from sshfs to external hdd #7374
Comments
hetzner may use a rate limit, so you need to run the benchmark with local network where you do not have the rate limit. |
No, hetzner does not use a rate limit, but I can try to test it with a local server when I have the opportunity |
Not sure if we can do anything here, just too much variables / unknowns to point to any specific issue:
|
AIUI you're mounting a remote internet server using sshfs and then creating a backup of the files in that sshfs mount to a local disk. This can perform somewhat okay if the average file size is big enough and the BDP is <<2 MB (because ssh), but sshfs makes no attempt at latency hiding and couldn't do much if it tried - so this is very dependent on the latency of the sshfs mount. This setup will always perform badly with many files. |
is there anything preventing the installation/usage of borg on the remote server? at first glance it seems hetzner storage boxes have borg 1.2 avaliable - so maybe something cna be done there to run brok directly rather than doing the remote filesystem dance (which is costly) |
Hello :) Your benchmarks seem a little off:
Especially the Laptop->local hdd benchmarks seem like you are just writing into your RAM and dont do fsync, so you are not really testing your hdd write speeds. |
@someone-somenet-org See the code about what these numbers mean. |
@RonnyPfannschmidt the problem is that the borg running on the storage box can only be used for backing up to the box, not the other way around. |
In borg 1.x the files cache uses the absolute path as the cache key. Maybe that was the problem (if the rsyned stuff was not at the same place as the mounted sshfs)? |
just chipping in that i recently also got a hetzner storage box, and am using it to back up data using borg. i don't do an sshfs mount, and my command is: borg create --progress --stats ssh://storagebox:23/./borg::date /backups my upload speed is 30Mbps, so assuming borg uploaded the entire 121GB, it should have been done witin 9 hours... let me be clear: i'm not sure if the issue is ssh, borg, or hetzner at this point, i'm just chipping in that i have the same issue, sidenote: i've started to open the archive with vorta, which is currently still fetching and building the archive index, |
I've used Borg for years to backup snapshots from my Hetzner server to my home server using Borg over ssh. These are large files, but the advantage has always been Borgs fantastic ability to pick out what's change and only back that up. At least that's what I assume happens as my logs show "Duration" of between a minute and 3 hours, never more than that. (If it's possible to pull backups over ssh without sshd please let me know! |
@Waldorf3 please check the FAQ about the files cache. |
I had a look, but it seems files cache is discussed in several FAQ entries. Most of it is beyond my paygrade, but I think the basic takeaway is that as long as the path is the same, it shouldn't need to redo the files cache - or do I misunderstand something? Are you suggesting it is going to be a matter of time before performance will become as before? |
@Waldorf3 Many network filesystems (like sshfs) do not have stable inodes, but the default for detection of "this file was not changed" is So, try |
Thanks. I've updated the script with this setting and will check over the next days if it changes anything. |
Have you checked borgbackup docs, FAQ, and open GitHub issues?
Yes
Is this a BUG / ISSUE report or a QUESTION?
QUESTION
System information. For client/server mode post info for both machines.
Your borg version (borg -V)
borg 1.2.3
Operating system (distribution) and version.
fedora 37
Hardware / network configuration, and filesystems used.
hetzner storage box, (xfs?)/mounted via sshfs as
fuse.sshfs
linux laptop (
AMD Ryzen 7 4800H
,32 GB RAM
),btrfs + luks
external hdd (
WDC WD20SDRW
),xfs + luks
How much data is handled by borg?
300GB
Full borg commandline that lead to the problem (leave away excludes and passwords)
borg create -v --stats --progress --files-cache=ctime,size --timestamp=2023-02-20 /backup/borg/nextcloud_datbkp::2023-02-20 .
Describe the problem you're observing.
To backup the content of the remote storage box, I used
rsync
until last year to save the data on an external hdd semiregularly.Since I wanted to use incremential backups, I decided to switch to borgbackup, created a new repo on the hdd, made a first archive from the rsync backup (also with
--files-cache=ctime,size
) which took about 2 hours, and then started to back up the content of the storage box, which turned out to be very slow (~10GB per hour), even when just deduplication of existing files was done, and the network speed would allow for much faster download times.Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
Yes, it always happens
Include any warning/errors/backtraces from the system logs
None
CRUD benchmark results
From the storage box to the hdd:
I did an additional test from the laptop to the hdd:
I can provide additional benchmarks (like storage box -> laptop) if needed
The text was updated successfully, but these errors were encountered: