Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very slow backup speed when creating archive from sshfs to external hdd #7374

Open
AntonOellerer opened this issue Feb 23, 2023 · 15 comments
Open

Comments

@AntonOellerer
Copy link

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

QUESTION

System information. For client/server mode post info for both machines.

Your borg version (borg -V)

borg 1.2.3

Operating system (distribution) and version.

fedora 37

Hardware / network configuration, and filesystems used.

hetzner storage box, (xfs?)/mounted via sshfs as fuse.sshfs
linux laptop (AMD Ryzen 7 4800H, 32 GB RAM), btrfs + luks
external hdd (WDC WD20SDRW), xfs + luks

How much data is handled by borg?

300GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg create -v --stats --progress --files-cache=ctime,size --timestamp=2023-02-20 /backup/borg/nextcloud_datbkp::2023-02-20 .

Describe the problem you're observing.

To backup the content of the remote storage box, I used rsync until last year to save the data on an external hdd semiregularly.
Since I wanted to use incremential backups, I decided to switch to borgbackup, created a new repo on the hdd, made a first archive from the rsync backup (also with --files-cache=ctime,size) which took about 2 hours, and then started to back up the content of the storage box, which turned out to be very slow (~10GB per hour), even when just deduplication of existing files was done, and the network speed would allow for much faster download times.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes, it always happens

Include any warning/errors/backtraces from the system logs

None

CRUD benchmark results

From the storage box to the hdd:

C-Z-BIG           4.06 MB/s (10 * 100.00 MB all-zero files: 246.47s)
R-Z-BIG         490.09 MB/s (10 * 100.00 MB all-zero files: 2.04s)
U-Z-BIG          37.79 MB/s (10 * 100.00 MB all-zero files: 26.46s)
D-Z-BIG        6327.17 MB/s (10 * 100.00 MB all-zero files: 0.16s)
C-R-BIG           3.89 MB/s (10 * 100.00 MB random files: 256.86s)
R-R-BIG         105.67 MB/s (10 * 100.00 MB random files: 9.46s)
U-R-BIG          36.07 MB/s (10 * 100.00 MB random files: 27.72s)
D-R-BIG         307.78 MB/s (10 * 100.00 MB random files: 3.25s)
C-Z-MEDIUM        2.12 MB/s (1000 * 1.00 MB all-zero files: 471.65s)
R-Z-MEDIUM      655.21 MB/s (1000 * 1.00 MB all-zero files: 1.53s)
U-Z-MEDIUM       14.95 MB/s (1000 * 1.00 MB all-zero files: 66.89s)
D-Z-MEDIUM     5369.50 MB/s (1000 * 1.00 MB all-zero files: 0.19s)
C-R-MEDIUM        1.76 MB/s (1000 * 1.00 MB random files: 567.90s)
R-R-MEDIUM      111.18 MB/s (1000 * 1.00 MB random files: 8.99s)
U-R-MEDIUM       15.75 MB/s (1000 * 1.00 MB random files: 63.50s)
D-R-MEDIUM      113.18 MB/s (1000 * 1.00 MB random files: 8.84s)
C-Z-SMALL         0.10 MB/s (10000 * 10.00 kB all-zero files: 976.49s)
R-Z-SMALL       188.45 MB/s (10000 * 10.00 kB all-zero files: 0.53s)
U-Z-SMALL         0.14 MB/s (10000 * 10.00 kB all-zero files: 727.17s)
D-Z-SMALL       395.59 MB/s (10000 * 10.00 kB all-zero files: 0.25s)
C-R-SMALL         0.09 MB/s (10000 * 10.00 kB random files: 1093.80s)
R-R-SMALL        81.39 MB/s (10000 * 10.00 kB random files: 1.23s)
U-R-SMALL         0.14 MB/s (10000 * 10.00 kB random files: 706.49s)
D-R-SMALL        80.20 MB/s (10000 * 10.00 kB random files: 1.25s)

I did an additional test from the laptop to the hdd:

C-Z-BIG         420.36 MB/s (10 * 100.00 MB all-zero files: 2.38s)
R-Z-BIG         453.48 MB/s (10 * 100.00 MB all-zero files: 2.21s)
U-Z-BIG        2637.48 MB/s (10 * 100.00 MB all-zero files: 0.38s)
D-Z-BIG        3449.62 MB/s (10 * 100.00 MB all-zero files: 0.29s)
C-R-BIG          76.27 MB/s (10 * 100.00 MB random files: 13.11s)
R-R-BIG         101.26 MB/s (10 * 100.00 MB random files: 9.88s)
U-R-BIG        2287.09 MB/s (10 * 100.00 MB random files: 0.44s)
D-R-BIG         318.11 MB/s (10 * 100.00 MB random files: 3.14s)
C-Z-MEDIUM      478.53 MB/s (1000 * 1.00 MB all-zero files: 2.09s)
R-Z-MEDIUM      653.02 MB/s (1000 * 1.00 MB all-zero files: 1.53s)
U-Z-MEDIUM     2893.54 MB/s (1000 * 1.00 MB all-zero files: 0.35s)
D-Z-MEDIUM     3734.19 MB/s (1000 * 1.00 MB all-zero files: 0.27s)
C-R-MEDIUM       69.29 MB/s (1000 * 1.00 MB random files: 14.43s)
R-R-MEDIUM      106.06 MB/s (1000 * 1.00 MB random files: 9.43s)
U-R-MEDIUM     2837.86 MB/s (1000 * 1.00 MB random files: 0.35s)
D-R-MEDIUM      107.29 MB/s (1000 * 1.00 MB random files: 9.32s)
C-Z-SMALL        32.45 MB/s (10000 * 10.00 kB all-zero files: 3.08s)
R-Z-SMALL       196.30 MB/s (10000 * 10.00 kB all-zero files: 0.51s)
U-Z-SMALL        90.66 MB/s (10000 * 10.00 kB all-zero files: 1.10s)
D-Z-SMALL       370.42 MB/s (10000 * 10.00 kB all-zero files: 0.27s)
C-R-SMALL        17.68 MB/s (10000 * 10.00 kB random files: 5.66s)
R-R-SMALL       102.73 MB/s (10000 * 10.00 kB random files: 0.97s)
U-R-SMALL        85.75 MB/s (10000 * 10.00 kB random files: 1.17s)
D-R-SMALL        93.16 MB/s (10000 * 10.00 kB random files: 1.07s)

I can provide additional benchmarks (like storage box -> laptop) if needed

@infectormp
Copy link
Contributor

hetzner may use a rate limit, so you need to run the benchmark with local network where you do not have the rate limit.

@AntonOellerer
Copy link
Author

No, hetzner does not use a rate limit, but I can try to test it with a local server when I have the opportunity

@ThomasWaldmann
Copy link
Member

Not sure if we can do anything here, just too much variables / unknowns to point to any specific issue:

  • internet connection speed / latency
  • sshfs speed / issues
  • linux storing lots of data to USB devices sometimes can be also rather slow and even slow down the whole system
  • hetzner server speed / load
  • LUKS

@enkore
Copy link
Contributor

enkore commented Aug 29, 2023

AIUI you're mounting a remote internet server using sshfs and then creating a backup of the files in that sshfs mount to a local disk. This can perform somewhat okay if the average file size is big enough and the BDP is <<2 MB (because ssh), but sshfs makes no attempt at latency hiding and couldn't do much if it tried - so this is very dependent on the latency of the sshfs mount. This setup will always perform badly with many files.

@RonnyPfannschmidt
Copy link
Contributor

is there anything preventing the installation/usage of borg on the remote server?

at first glance it seems hetzner storage boxes have borg 1.2 avaliable - so maybe something cna be done there to run brok directly rather than doing the remote filesystem dance (which is costly)

@someone-somenet-org
Copy link

Hello :)

Your benchmarks seem a little off:

  • HDDs usually dont exceed 200 MB/s.
  • More that 125 MB/s exceeds 1 GBit/s.

Especially the Laptop->local hdd benchmarks seem like you are just writing into your RAM and dont do fsync, so you are not really testing your hdd write speeds.

@ThomasWaldmann
Copy link
Member

@someone-somenet-org See the code about what these numbers mean.

@AntonOellerer
Copy link
Author

@RonnyPfannschmidt the problem is that the borg running on the storage box can only be used for backing up to the box, not the other way around.

@ThomasWaldmann
Copy link
Member

In borg 1.x the files cache uses the absolute path as the cache key. Maybe that was the problem (if the rsyned stuff was not at the same place as the mounted sshfs)?

@thutex
Copy link

thutex commented Oct 12, 2024

just chipping in that i recently also got a hetzner storage box, and am using it to back up data using borg.
the speeds (both up and down) seem to be terrible, while my connection can perform much better.

i don't do an sshfs mount, and my command is: borg create --progress --stats ssh://storagebox:23/./borg::date /backups
last upload (5263438 files, compressed 121.36GB, deduplicated 7.36GB) took over 24 hours:
Duration: 1 days 1 hours 8 minutes 5.14 seconds

my upload speed is 30Mbps, so assuming borg uploaded the entire 121GB, it should have been done witin 9 hours...
say for some reason my speed was only half of that, it still should have finished in 18 hours.
and, ofcourse, as it only uploads the deduplicated data.... well... something, somewhere is off.

let me be clear: i'm not sure if the issue is ssh, borg, or hetzner at this point, i'm just chipping in that i have the same issue,
so we can exclude things like the local system or internet connection, meaning it looks like either borg or hetzner (or ssh, ofcourse) being the issue

sidenote: i've started to open the archive with vorta, which is currently still fetching and building the archive index,
while it was started almost 3 hours ago (download speeds are 100Mbps), so it's an issue in both directions.

@Waldorf3
Copy link

I've used Borg for years to backup snapshots from my Hetzner server to my home server using Borg over ssh. These are large files, but the advantage has always been Borgs fantastic ability to pick out what's change and only back that up. At least that's what I assume happens as my logs show "Duration" of between a minute and 3 hours, never more than that.
Now I decided to turn things around, so instead of pushing the backups, I pull them from my home server, and for that I had to use sshfs instead, and now I find backup times of no less than 9 hours, and up to more than a day. The backups are similar content and size, so is the bandwidth. There must be something odd about using sshfs for this.

(If it's possible to pull backups over ssh without sshd please let me know!

@ThomasWaldmann
Copy link
Member

@Waldorf3 please check the FAQ about the files cache.

@Waldorf3
Copy link

@Waldorf3 please check the FAQ about the files cache.

I had a look, but it seems files cache is discussed in several FAQ entries. Most of it is beyond my paygrade, but I think the basic takeaway is that as long as the path is the same, it shouldn't need to redo the files cache - or do I misunderstand something? Are you suggesting it is going to be a matter of time before performance will become as before?

@ThomasWaldmann
Copy link
Member

@Waldorf3 Many network filesystems (like sshfs) do not have stable inodes, but the default for detection of "this file was not changed" is --files-cache=inode,ctime,size. The files cache is what makes borg really fast.

So, try --files-cache=ctime,size as the inode numbers likely will change between backups.

@Waldorf3
Copy link

Thanks. I've updated the script with this setting and will check over the next days if it changes anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants