1.7.0 Extremely Slow Upload for small files

[BrettMayson]

I have created a brand new ownCloud setup running on Ubuntu 14.04
I am using the client version 1.7.0 also on Ubuntu 14.04
The upload speed from the client is extremely slow saying my 6GB upload will take 3.5 days

I am on the same network as the server with a 100Mpbs ethernet connect and nothing else on the network.

[BrettMayson]

I have noticed that while syncing large files it jumps up to about 600kb/s but small files take seconds each

[guruz]

This might unfortunately be a known issue with the current server and syncing algorithm.
Are you uploading into a shared directory?
Do you have a bandwidth limit enabled in the client?

@ogoffart @dragotin @DeepDiver1975 @PVince81 Let's increase the current job concurrency from 3 to 4 jobs?

[BrettMayson]

@guruz
It is not a shared directory
Bandwidth is set to unlimited for up and down.

[BrettMayson]

I have just noticed it also deleted a folder with about 80MB of all small files.
I was able to restore the folder from the webUI but then things acted all weird before the folder disappeared without a trace from both the webUI and my computer.

[guruz]

I have just noticed it also deleted a folder with about 80MB of all small files.

OK that is a bad bug. Do you have any more info about this issue? Any logs? Does the folder have anything special (characters, links etc..)

[BrettMayson]

Couldn't find it in any logs. The folder was named AndroidStudioProjects. Should this maybe be moved to a seperate bug? Although I don't really have much information to provide.

[guruz]

@BmanDesignsCanada it sounds odd, yes, a seperate bug would be nice.

[8h2a]

I have almost the same problem. (with Client 1.7.1)
I just added about 1000 files that are smaller than 10KB each (8MB in sum) and it takes about an hour to upload them. (Server is in same LAN with 1000MBit/s network connection and the only thing i noticed is that the mysql server uses around 10% CPU)

So i am guessing the file upload algorithm is very inefficient.
It would be nice if the client would upload such small files in bulk to speed things up.
maybe something like this: #548
Related core issues: owncloud/core#7072 and owncloud/core#12001

Please tell me if you need more information.
But the behaviour is really easy to reproduce: Just add a bunch of small files.

[PVince81]

There's research in progress to find a way to batch upload small files: owncloud/core#12001

[seiferflo]

Hi,
Just saw that post as I have to wait almost 2 days to upload 25GB with 25000 files via owncloud 1.7.1 on windows 8.1 to a Synology DS415+ & owncloud 7.0.3.5
I'm thinking of abandoning and turn back to Cloudstation which I think is not as good as owncloud but I can't work with that now.
I've read PVince81 post above, but there is no activity for a month now. Any idea of the timescale for a fix?
Thanks in advance for your reply.

[mkymikky]

Hi, I can reproduce very slow uploads containing a huge amount of files, both small and huge.

Scenario:
Client Version 1.7.1 (build 4382) at Windows 7 64bit, Core i7, 8GB Ram, 219GB in 46.556 files, Compressed on NTFS
Server Version 7.0.4 at Ubuntu 14.04 LTS, Cubietruck (ARM), 2GB Ram, Apache2 2.4.7, PHP 5.5.9, MySQL 5.5.35
User without Admin-Right, Unlimited Space
Network with one gigabit via two gigabit-switches

Currently it runs with few interruptions for 3 days and managed to upload 107,6GB to the Serverspace of 941,9GB. I edited the configuration (.htaccess and php.ini) to accept files up to 16GB of size and set the upload directory to the 1TB hard disk containing ownclouds data files. On bigger files I recognize transfer speed to be between 1.0 and 1.3MB per second. Smaller files take very long to be stored.

As I'm a programmer, I would be glad to support you fixing this issue.
Is it of any help installing a daily build on a Windows 2008 R2, 64bit, 32GB, Core i3 with XAMPPlight and retry syncing from scratch?

[netzwerch]

Hello,
I can reproduce that problem when having lots of files too.

Scenario:
Clients: OC-Client Version 1.7.1 (build 4382) at Windows 7 64bit, Core i7, >=8GB Ram
Server Version 7.0.4 at Ubuntu 14.04 LTS, Core i3 (4 Cores), 6GB Ram, Nginx, PHP 5.5, MySQL 5.5
Local Network
Shared Folder with 580GB in ~500.000 files.

We desperately uploaded all the files from one single client within 1-2 weeks via the oc-client. Now that all the files are in place, we experience that every sync we start, stays basically only in "Discovering '...foldernames...'" stage. The only way to bypass that, is to only select smaller subfolders from inside the huge shared folder in the client as a folder pair.
Folder sizes from about 20GB seem to be acceptable, since they really start to sync after a while and also upload files changes in a reasonable time. Bigger sized (>100-200 GB) folders can be considered as never ever synced, since they are stuck on "Discovering '...foldername...' ".

Does anyone else also tried to use ownCloud with that amount of files? Whats your file amount limit where a good user experience could be expected?

[guruz]

Could you try a 1.8 testing version?
Things have changed there with regards to the discovery phase ("preparing to sync")

On 15.02.2015, at 02:59, Christian W. notifications@github.com wrote:

Hello,
I can reproduce that problem when having lots of files too.

Scenario:
Clients: OC-Client Version 1.7.1 (build 4382) at Windows 7 64bit, Core i7, >=8GB Ram
Server Version 7.0.4 at Ubuntu 14.04 LTS, Core i3 (4 Cores), 6GB Ram, Nginx, PHP 5.5, MySQL 5.5
Local Network
Shared Folder with 580GB in ~500.000 files.

We desperately uploaded all the files from one single client within 1-2 weeks via the oc-client. Now that all the files are in place, we experience that every sync we start, stays basically only in "preparing sync" stage. The only way to bypass that, is to only select smaller subfolders from inside the huge shared folder in the client as a folder pair.
Folder sizes from about 20GB seem to be acceptable, since they really start to sync after a while and also upload files changes in a reasonable time. Bigger sized (>100-200 GB) folders can be considered as never ever synced, since they are stuck on "preparing sync".

Does anyone else also tried to use ownCloud with that amount of files? Whats your file amount limit where a good user experience could be expected?

—
Reply to this email directly or view it on GitHub.

[mkymikky]

My test showed no acceleration while "preparing to sync". In contrary running it on similar hardware with version 1.71 vs. 1.8beta indicated the new version to be slightly slower until synchronizing files*.

*the discrepancy is not significant in my eyes

Addition: I did the test with data mentioned above but with owncloud server version 8.0

[danimo]

Did you test with ownCloud 8?

[mkymikky]

Yes, I did, corrected it above.

[axllent]

For what it is worth, I have been having the same issue described here - very slow uploads, especially noticeable with many small files, and based on the thread I have been blaming the clients. Whilst I don't run heavy hardware on the server and only have a handful of clients, the mysql database resides on an regular ext4 SATA 7200RPM HD (running Debian). Using iostat -x 1 I was able to clearly identify a bottleneck as being the harddrive on the server which the mysql database is stored on with 95-98% I/O use.

After trying disabling the default barrier mount option (mount -o remount,nobarrier / - or whichever partition your mysql database resides on) the I/O dropped to < 10% and the clients now fly in comparison. This seems to indicate that the MySQL updates/writes (when files are added / deleted) are probably the underlying issue here, briding the approximate client sync times down from around 1-2 hours down to < 10 minutes.

Now I'm not sure exactly what the consequences of the nobarrier mount option may be (still to investigate), however it makes a significant differenace to the server-side performance, which significantly impacts the client's performance.

It's just worth noting that (in my case) the client is not to blame, but rather the server.

[mkymikky]

I will try that soon.

If the impact is big enough to mention, I think the use of nobarrier is no problem.
Even though 'barrier' reduces data inconsistency on powerloss this will be either compensated by enterprise-level power buffering or should not be too risky for personal users. In either case, making backups is reducing the additional risk to an acceptable level in my eyes.

As written on
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/writebarrieronoff.html

Redhat states:

The use of nobarrier is no longer recommended in Red Hat Enterprise Linux 6 as the negative performance impact of write barriers is negligible (approximately 3%). The benefits of write barriers typically outweigh the performance benefits of disabling them. Additionally, the nobarrier option should never be used on storage configured on virtual machines.

But they also say:

For devices with non-volatile, battery-backed write caches and those with write-caching disabled, you can safely disable write barriers at mount time using the -o nobarrier option for mount.

[dragotin]

There is work in progress on the server to speed up this use case. I am closing this hence.