Synchronizing small files = very slow

[ghost]

Expected behaviour

Files sync at about the speed of the broadband connection, independent of size unless limited by CPU or IO on client or server.

Actual behaviour

Synchronizing a folder with many small files is terribly slow. For instance a test folder with 10k 3-byte-files synchronizes with a speed of about 1 file per second over a broadband connection. The load on both client and server while doing this is negligible. Tested with server running Ubuntu 12.04 with current owncloud 6.0.1-1, client current Linux or Macos clients. The transfer rates on the broadband connection are very low as well, so this is not the bottleneck.

Steps to reproduce

1. Download http://xn--grgen-jua.name/owncloudStressTest.tar.bz2
2. unpack it into an owncloud folder that syncs with a remote owncloud machine.
3. Observe upload speeds of one to a few 3-byte-files per second.

Server configuration

Operating system: Ubuntu 12.04

Web server: lighttpd

Database: mysql

PHP version: PHP 5.3.10-1ubuntu3.9

ownCloud version: 6.0.1-1

Updated from an older ownCloud or fresh install: Updated from ownCloud 5

The content of config/config.php:
$CONFIG = array ( 'instanceid' => 'removed', 'passwordsalt' => 'removed', 'datadirectory' => '/var/www/owncloud/data', 'dbtype' => 'mysql', 'version' => '6.0.0.16', 'dbname' => 'dropbox', 'dbhost' => 'localhost', 'dbtableprefix' => 'removed', 'dbuser' => 'removed', 'dbpassword' => 'removed', 'installed' => true, 'theme' => '', 'maintenance' => false, );

no external storage used.

Log files do not show error messages or warnings that seem related.

Client configuration

Both MacosX and Ubuntu 13.10 running current owncloud versions are affected. Browser was not used.

Why I think this is an issue

I regularly sync folders with several thousands of files. I think I am not the only one.

What I think is the reason

I am guessing that owncloud does at least one if not several http requests per file sent. It works sequentially, so the next file can only be sent when the last file is acknowledged. Therefore while preparing the request takes no time and effort, and storing the file does not either, the network delay comes into play: For each file to be sent you have to wait at least one if not several round-trip times.

Another indication is that syncing the folder on the local network or between two VMs on the same machine is much faster. This emphasizes the network delay being of importance in this scenario.

How I think it could be circumvented

Sync not file-by-file but instead several files per request, at least if the files are small.

[IngLP]

Or, sync with multiple threads and split the work. Much more easy.
Only one thread search for the files to sync, splits the list of files to be uploaded, and starts the threads.

[PVince81]

@dragotin did you guys have plans looking into uploading multiple files through the same connection ?

[PVince81]

Just had a try locally, using "localhost" and the sync client is able to upload many files within a second:

I can imagine that each connection might take more time over the internet, also if it includes name resolution.

Have you tried directly with the IP address to rule out the name resolution case ?

[ghost]

I used the IP directly (or rather a name, but one which is taken from /etc/hosts, not looked up via DNS). I can reproduce your results when uploading locally. Unfortunately fast local performance is academic as the usual case will be a server somewhere on the internet.

[PVince81]

It still feels a bit slow, this is about 41 M which I'd expect to be uploaded much faster.
The problem also is that it might wait for the server to finish creating/saving/indexing the file before continuing, as the server won't give an answer until that happened.

[PVince81]

I tried with the tar file without compression (bunzip2) which is 11Mb big, it took less than a second to upload. I'd almost think that before uploading many files those could be tar'ed together, uploaded then untarred on the server...

[ghost]

Yes I was thinking about something like that when I said "multiple files per request". I think the issue is really delay somehwere on the way. However It seems to be not only the round-trip time. A ping to my server says roundtrip is only 6 miliseconds, so I am wondering where the rest of the time is spent. My accesslog looks like:

<my ip> frickel - [04/Feb/2014:19:20:38 +0100] "PUT /owncloud/remote.php/webdav/owncloudtest/36/42.txt HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh) csyncoC/0.91.4 neon/0.30.0"
<my ip> frickel - [04/Feb/2014:19:20:40 +0100] "PUT /owncloud/remote.php/webdav/owncloudtest/36/43.txt HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh) csyncoC/0.91.4 neon/0.30.0"
<my ip> frickel - [04/Feb/2014:19:20:42 +0100] "PUT /owncloud/remote.php/webdav/owncloudtest/36/44.txt HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh) csyncoC/0.91.4 neon/0.30.0"
<my ip> frickel - [04/Feb/2014:19:20:44 +0100] "PUT /owncloud/remote.php/webdav/owncloudtest/36/45.txt HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh) csyncoC/0.91.4 neon/0.30.0"
<my ip> frickel - [04/Feb/2014:19:20:46 +0100] "PUT /owncloud/remote.php/webdav/owncloudtest/36/46.txt HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Macintosh) csyncoC/0.91.4 neon/0.30.0"

Load on both client and server are negligible. Still only 1 file in two seconds.

The results are reproducable on a second server (also ubuntu 12.04, but apache instead of lighttpd). Round trip (ping) for the second server is 14ms.

Uploading several files at once might also allow to add several files to the database at once, in case that is the bottleneck.

[ghost]

@PVince81 Are you sure it is several files a second locally? The timestamp in the image you posted are in hours:minutes.

[PVince81]

@gluap hmm right. I was actually looking at the progress bar and saw the file name changing 3-4 times within a second and thought the log would reflect that.

[ghost]

@gulap

You might also want to check one of this reported issues: owncloud/client#331, #5084, #3118, owncloud/client#209 and many more of the same/similar reports.

[ghost]

@RealRancor: thanks for the hint. Apparently my search before creating the issue was not thorough enough. My apologies.
I think I am seeing the same problems as #3118 and owncloud/client#209 and possibly owncloud/client#331. I hope my method of reproducing the issue helps a little. For me this issue is a real dealbreaker.

[enoch85]

Dont know if it helps you guys, but theese are my results.

100/10 Mbit/s Server
100/10 Mbit Client.

[jcfischer]

We are in the same boat (and have filed #6937 ). I have traced the application with New Relic and I see hundreds of SELECTs for every file uploaded - that slows the processing down considerably.

[ghost]

@jcfischer Would it be correct to summarize the current results of #6937 as "The main delay of transferring a small file comes from the fact that
a) every file transfer generates a ton of SQL requests and
b) files are transferred serially"?

[jcfischer]

I can't say conclusively yet. Yes, there are a ton of SQL requests which cost time.

Yes, every HTTP request incurs startup cost (I measure 60ms as the baseline for ownCloud - this is the time it takes status.php to load and I assume that this is loading and initializing the complete ownCloud stack.

I don't know if there are more time sinks involved.

Sent from my iPhone

On 05.02.2014, at 10:42, gluap notifications@github.com wrote:

@jcfischer Would it be correct to summarize the current results of #6937 as "The main delay of transferring a small file comes from the fact that
a) every file transfer generates a ton of SQL requests and
b) files are transferred serially"?

—
Reply to this email directly or view it on GitHub.

[sunny-user]

I´m using the OwnCloud Sync Tool for windows to sync about 100.000 files.
In my activity log I can see, that it takes more and more time for uploading a file as the number of uploaded files increase. The first files took about 2 seconds, now I uploaded 9000 files to 1und1.de and it takes about 17 seconds. The more files there are uploaded, the more time needs a file to upload. Because of small files the internet transfer speed does not play any role.
I would say, Owncloud has a very bad database / transaction design. It needs a much better structure / programming for much better performance.

[PVince81]

@bantu @DeepDiver1975 @karlitschek @icewind1991 any idea ?

[nickbunyan]

Having worked with a variety of backup transfer products professionally for some years I am afraid to say there appears to be a fundamental problem with the methodology being used for Client syncing.

Without going into the specifics of how OC is working it clearly should not rely on sequential transfer of individual files when 'adding' a new folder and yet it appears to be issuing multiple queries to the server before even considering uploading each file.

Clearly this type of activity should be handled in a batch fashion - perhaps by allowing the client to toggle a setting - or simply warning that the number of 'new' files means it will work in batch mode only until the new folder sync is completed.

As an example, Batch Mode could mean that all files from the 'New Folder' are uploaded to a temporary location on the server, the server resources are then used to 'Scan' the folder and create records for each of the new files, setting timestamps etc and migrating the files to their proper place. At the end of the process Batch Mode is turned off and a consistency check is performed.

After tweaking MySQL settings and cache and so forth I have got to a situation where OC running on Ubuntu 12.04, with 1.5.1 on Windows 7 works well enough to be usable for general day to day updates but if you need to include another folder; do it like this:- Use Filezilla or similar to copy the original files to the correct location on the Server. Change the permissions on the uploaded files to match the rest of the OC files. Add the folder to Owncloud at the Client side. Wait while the Client syncs all the 'New Files' down from the server. [It still takes a while, but hours rather than weeks].

This project has fantastic potential, but the fundamental architecture just doesn't appear to scale, sadly.

[Supermathie]

@sunny-user I experienced this as well and tracked it down to the oc_jobs table - as the number of jobs increases the cost of adding a new one increases as well, leading to higher web transactional latency and cpu% on the DBMS getting higher and higher.

Turns out the cron job which processes scheduled jobs wasn't running. This made a huge difference right away.

But... I have other problems. Specifically, discovering that sync performance scales linearly with # of directories.

[ralphholzmann]

Migrating to ownCloud from Dropbox, and currently feeling the pain of this issue as well :(

[ghost]

@ralphholzmann

Have you seen:

http://mailman.owncloud.org/pipermail/devel/2014-April/000110.html

[ralphholzmann]

@RealRancor Yep. Still no luck. Still extremely slow. Here's a video to illustrate the pain with small files (less than 100k).

https://dl.dropboxusercontent.com/u/47777/slow_own_cloud.mov

[ghost]

Hi,

@ralphholzmann

Posting a video is probably not very useful in this case as it don't say anything about your environment. If you're running SQLite you might want to try to change your environment to a "real" Database like MySQL or PostgreSQL.

[ralphholzmann]

@RealRancor the reason I posted the video is because there was confusion about timestamps in the activity pane earlier in the thread and the video provided a much better visual of the problem -- but to answer your question, I'm running ownCloud 6.0.2 on my Netgear ReadyNAS connected to my local network, and it's configured to use MySQL.

[tetsuo76]

I am facing the same problem while trying to sync a large netbeans project which includes a .git repository. It takes ownCloud around 2-3 seconds for each file and most of them are less than 1kb. Sync'ing 20,000 files (~200mb in total) takes me almost a day..

[nerzhul]

I have the same problem, users with many files are waiting many time. I tested on MySQL 5.5 and PostgreSQL 9.2, same problem (it's database clusters, optimized for performance)

[andydvsn]

I'm experiencing exactly the same issue; I keep a backup copy of my Aperture library in Dropbox and have been attempting to sync it to my server with ownCloud. Many small files in there.

Using the current 1.6.0rc2 client on OS X, connecting to ownCloud on Ubuntu 14.04 (Apache 2.4, MySQL 5.5) and the small files are taking forever - very similar performance as shown by the @ralphholzmann video (approximately 1 file per second, usually around 2-3KB each). Destination server is a dedicated 2GHz Intel Atom box in a data centre with solid 100Mbps connection, client machine is on JANET, so bandwidth is no issue. Individual large files transfer extremely quickly.

[enoch85]

@Tralapo @batamire I just tested to sync the Stress Test package with 10 300 files of 3 b each, and it took me 8 minutes with 2.0.2 testpilot version: https://download.owncloud.com/desktop/daily/testpilotcloud-2.0.2.5525-nightly20151008-setup.exe

Package: https://www.en0ch.se/stress-test/

Look in previous comments for server setup. Now using server 8.1.3 and PHP 5.6

[Tralapo]

@enoch85 Thanks for the tip. I installed the 2.0.2 Testpilot and it indeed seems to perform a lot better then 2.0.1. I don't get the stress test done in 8 minutes, it expects around 8-10 hours. But that's because of the hardware (RaspberryPi) I would say. The 2.0.1. gives me a expectation of several days(!), so an improvement for sure.

Stable:

Beta:

[enoch85]

For future testing, here are the latest daily builds: https://download.owncloud.com/desktop/daily/

[ApolloLV]

Using Desktop App 2.1.0 and Owncloud 8.2.2 on a Ubuntu 14.04 LTS Server running on my Cubietruck, backend is MYSQL 5.5.47. I am using APCu and static assets with Apache 2.4.7 and PHP 5.5.9. I now get 2 days!!! for 92 MBs in 4370 files. The Sync activity does not even bottleneck anything. The Network connection is wifi and the machines are in the same network. The HDD is directly accessed via SATA. The only possible bottleneck is the CPU, it experiences spikes for some seconds and then goes back to idling for several seconds.
What could be wrong?

[Traiden]

I increased a lot my speed with this methode, but I don't understand really. Somebody can explain to me please?

http://forum.owncloud.org/viewtopic.php?f=3&t=6621#p14737

Thank you!

[mgscreativa]

Hi! I have installed php 7.0.4 on my home server because all the praises about being fast. I installed it on my Ubuntu Server 14.04.4 LTS with ispConfig. The installation went just fine, and I also managed to enable PHP-FPM 7.0.4 on an apache virtual host subfolder only (yes, I have php 5.5.9 on virtualhost and in /cloud subfolder PHP 7.0.4 FPM) . Despite of this, my OC 2.1.1 sync client tells me that the sync of the stress test files will take above 3 hours!

Copying the stress test files through CIFS to my server takes about 3 to 5 minutes, maybe less.

This is not the first time I try to tackle this issue so I have prepared a very detailed bug report for @enoch85, @ApolloLV, @gluap, @PVince81 and @guruz so they can take a look at it. For you guys, basically I was able to setup PHP 7.0.4 with PHP-FPM on apache 2.4 virtualhost subfolder and seems to be woking right, I was also able to setup APCu 5.1.3 and seems to be caching ok (see details of apcu stats). Maybe you can take a look at my PHP-FPM pool config to optimize it a bit.

I have to mention that my sync client seems to sync 3 files then do a little sleep and then another 3 files, and so on...¿¿??

That being said, lets start to report this bug (or enhancement, as you like)...

Steps to reproduce

1. On Ubuntu Server 14.04 Install PHP 7.0.4 following this guide https://www.howtoforge.com/tutorial/how-to-install-php-7-on-debian/#comments
2. Enable PHP-FPM through ispConfig CP to enable it on web server
3. Install APCu 5.1.3 with this command /opt/php-7.0.4/bin./pecl install channel://pecl.php.net/apcu-5.1.3
4. Enable memcache in OwnCloud server config file
5. Enable APCu extension=apcu.so in main php.ini file /opt/php-7.0.4/lib/php.ini
6. Follow apache optimization guide here https://doc.owncloud.org/server/9.0/admin_manual/configuration_server/performance_tuning/webserver_tips.html#apache-tuning
7. On the client side install latest OwnCloud Client (2.1.1)
8. Put to sync the uncompressed stress test package

Expected behaviour

I expected that the sync will last less time!

Actual behaviour

The sync process is very slow!

Server configuration

Operating system: Ubuntu Server 14.04.4 LTS (Linux lucas 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux)

Web server: Server version: Apache/2.4.7 (Ubuntu) - Server built: Jan 14 2016 17:45:23

Database: MySQL 5.5.47-0ubuntu0.14.04.1

PHP version: 7.0.4 (Server API: FPM/FastCGI)

ownCloud version: 9.0

Updated from an older ownCloud or fresh install: Updated from an older ownCloud

Where did you install ownCloud from: Manual Install

Apache virtual host PHP-FPM config:

#PHP-FPM
<IfModule mod_fastcgi.c>
    KeepAlive On
    KeepAliveTimeout 100
    MaxKeepAliveRequests 200

    <IfModule mod_headers.c>
      Header always set Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    </IfModule>

    <Directory /var/www/it.mga.com/web/cloud>
        HostnameLookups Off   
        <FilesMatch "\.php[345]?$">
            SetHandler php7-fcgi
        </FilesMatch>
    </Directory>

    <Directory /var/XXX/clients/XXX/XXX/web/cloud/>
        HostnameLookups Off   
        <FilesMatch "\.php[345]?$">
            SetHandler php7-fcgi
        </FilesMatch>
    </Directory>

    <LocationMatch "/cloud/(fpm-ping|fpm-status)">
        HostnameLookups Off    
        SetHandler php7-fcgi-virt
        Action php7-fcgi-virt /php7-fcgi virtual
    </LocationMatch>

    Action php7-fcgi /php7-fcgi
    Alias /php7-fcgi /var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com
    FastCgiExternalServer /var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com -idle-timeout 300 -socket /var/lib/php5-fpm/XXX-cloud.sock -pass-header Authorization
</IfModule>

<IfModule mod_proxy_fcgi.c>
    ProxyPassMatch ^/(.*\.php[345]?(/.*)?)$ fcgi://127.0.0.1:9011/var/XXX/clients/XXX/XXX/web/cloud/$1
</IfModule>

PHP-FPM pool config:

[XXX-cloud]

listen = /var/lib/php5-fpm/XXX-cloud.sock
listen.allowed_clients = 127.0.0.1
listen.owner = XXX
listen.group = XXX
listen.mode = 0660
listen.backlog = 4096

user = XXX
group = XXX

;request_slowlog_timeout = 5s
;slowlog = '/var/XXX/clients/XXX/XXX/tmp/phplogs/php_cloud_fpm_slow.log'

request_terminate_timeout = 120s
rlimit_files = 131072
rlimit_core = unlimited
catch_workers_output = yes

pm = dynamic
pm.max_children = 10
pm.start_servers = 3
pm.min_spare_servers = 2
pm.max_spare_servers = 4
pm.max_requests = 400
pm.status_path = /cloud/fpm-status
ping.path = /cloud/fpm-ping
ping.response = 'Pong: *** XXX-cloud ***'

;chroot = '/var/XXX/clients/XXX/XXX/web/cloud/'
;chdir = /

; PHP Environment Variables Overrides
env[TMP] = '/var/XXX/clients/XXX/XXX/tmp'
env[TMPDIR] = '/var/XXX/clients/XXX/XXX/tmp'
env[TEMP] = '/var/XXX/clients/XXX/XXX/tmp'
env[PHP_DOCUMENT_ROOT] = '/var/XXX/clients/XXX/XXX/web/cloud'

; php.ini Overrides
php_admin_value[sendmail_path] = '/usr/sbin/sendmail -t -i -fwebmaster@it.mga.com'
php_admin_value[open_basedir] = '/var/XXX/clients/XXX/XXX/web/cloud:/var/XXX/clients/XXX/XXX/private:/var/XXX/clients/XXX/XXX/tmp:/var/www/it.mga.com/web/cloud:/srv/www/it.mga.com/web/cloud:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/urandom'
php_admin_value[session.save_path] = '/var/XXX/clients/XXX/XXX/tmp'
php_admin_value[upload_tmp_dir] = '/var/XXX/clients/XXX/XXX/tmp'
php_admin_value[disable_functions] = 'XXX,XXX,XXX'
php_admin_value[upload_max_filesize] = 1024M
php_admin_value[post_max_size] = 1024M
php_admin_value[memory_limit] = 1024M
php_admin_flag[mbstring.func_overload] = Off
php_admin_value[default_charset] = 'UTF-8'
php_admin_flag[output_buffering] = Off
php_admin_value[max_execution_time] = 6000
php_admin_flag[short_open_tag] = Off
php_admin_value[date.timezone] = 'America/XXX/XXX'
php_admin_flag[display_errors] = Off
php_admin_flag[html_errors] = Off
php_admin_value[error_reporting] = E_ALL
php_admin_flag[log_errors] = On
php_admin_value[error_log] = '/var/XXX/clients/XXX/XXX/tmp/phplogs/php_cloud.log'
php_admin_flag[expose_php] = Off

APCu 5.1.3 stats:

General Cache Information
APCu Version    5.1.3
PHP Version 7.0.4
APCu Host   it.mga.com (lucas) (192.168.1.77)
Server Software Apache/2.4.7 (Ubuntu)
Shared Memory   1 Segment(s) with 32.0 MBytes (mmap memory)
Start Time  2016/03/24 17:05:51
Uptime  1 hour and 13 minutes

Cache Information
Cached Variables    372 (250.0 KBytes)
Hits    1033293
Misses  17093
Request Rate (hits, misses) 239.54 cache requests/second
Hit Rate    235.64 cache requests/second
Miss Rate   3.90 cache requests/second
Insert Rate 3.90 cache requests/second
Cache full count    0

Runtime Settings
apc.coredump_unmap  0
apc.enable_cli  
apc.enabled 1
apc.entries_hint    4096
apc.gc_ttl  3600
apc.mmap_file_mask  
apc.preload_path    
apc.serializer  php
apc.shm_segments    1
apc.shm_size    32M
apc.slam_defense    1
apc.smart   0
apc.ttl 0
apc.use_request_time    1
apc.writable    /tmp

Server hardware

CPU: Intel(R) Core(TM) i7 860 @ 2.80GHz

Motherboard: Gigabyte P55M-UD2

RAM: 12GB 1333 MHz

Server System HDDs: SSD Samsung 840 120Gb

Server Storage HDDs: 2x WD Green WD20EARX 2TB (software RAID1 with mdadm)

Network: (LAN) 1000 Mbit/s down / 1000 Mbit/s up

Stress Test Info

Stress Test Files: https://www.en0ch.se/stress-test/

Client: 2.1.1

Top:

top - 17:25:49 up  3:26,  2 users,  load average: 2,68, 2,47, 2,36
Tareas: 225 total,   3 ejecutar,  222 hibernar,    0 detener,    0 zombie
%Cpu(s):  5,8 usuario,  3,3 sist,  0,0 adecuado, 80,6 inact,  9,9 en espera,  0,
KiB Mem:  12296816 total, 12149720 used,   147096 free,   193332 buffers
KiB Swap:        0 total,        0 used,        0 free. 10387584 cached Mem

  PID USUARIO   PR  NI    VIRT    RES    SHR S  %CPU %MEM     HORA+ ORDEN       
13827 web2      20   0  197296  21948   5596 S  17,9  0,2   0:59.61 php-fpm     
13739 web2      20   0  197292  21908   5556 R  11,6  0,2   1:03.46 php-fpm     
13857 web2      20   0  274888  25812   7424 S  10,6  0,2   0:35.13 php-fpm     
13826 web2      20   0  197296  21908   5556 S  10,0  0,2   0:49.06 php-fpm     
13740 web2      20   0  197292  21936   5636 S   9,3  0,2   1:12.93 php-fpm 

Signing status (ownCloud 9.0 and above):

No errors have been found.

List of activated apps:

Enabled:
  - activity: 2.2.1
  - comments: 0.2
  - dav: 0.1.5
  - federatedfilesharing: 0.1.0
  - federation: 0.0.4
  - files: 1.4.4
  - files_pdfviewer: 0.8
  - files_sharing: 0.9.1
  - files_texteditor: 2.1
  - files_trashbin: 0.8.0
  - files_versions: 1.2.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 14.5.0
  - notifications: 0.2.3
  - provisioning_api: 0.4.1
  - systemtags: 0.2
  - templateeditor: 0.1
  - updatenotification: 0.1.0
Disabled:
  - encryption
  - external
  - files_external
  - user_external
  - user_ldap

The content of config/config.php:

list system
{
    "system": {
        "instanceid": "ocl00gp133cg",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "it.mga.com"
        ],
        "datadirectory": "\/var\/XXX\/clients\/XXX\/XXX\/web\/cloud\/data",
        "overwrite.cli.url": "https:\/\/it.mga.com\/cloud",
        "dbtype": "mysql",
        "version": "9.0.0.19",
        "dbname": "c1_cloud",
        "dbhost": "127.0.0.1",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "America\/Argentina\/Buenos_Aires",
        "installed": true,
        "forcessl": true,
        "theme": "",
        "maintenance": false,
        "mail_from_address": "martinb",
        "mail_smtpmode": "php",
        "mail_domain": "mga.com",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "loglevel": 1,
        "trashbin_retention_obligation": "auto"
    }
}

Are you using external storage, if yes which one: Nope

Are you using encryption: no

Are you using an external user-backend, if yes which one: Nope

Client configuration

Browser: Firefox 45.0

Operating system: Ubuntu 16.10

Logs

Web server error log

[Thu Mar 24 16:05:11.397590 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:05:13.857488 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:05:13.897382 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:05:23.729197 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:05:28.773171 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:05:33.817182 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:05:33.838210 2016] [:error] [pid 8598] [client 192.168.1.2:39140] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:05:38.861196 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:05:40.817157 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:06:12.785681 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:06:12.789222 2016] [:error] [pid 8713] [client 192.168.1.2:39136] FastCGI: server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" stderr: Primary script unknown
[Thu Mar 24 16:31:08.858862 2016] [fastcgi:error] [pid 10480] (104)Connection reset by peer: [client 192.168.1.2:39306] FastCGI: comm with server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com" aborted: read failed
[Thu Mar 24 16:31:08.858949 2016] [fastcgi:error] [pid 10480] [client 192.168.1.2:39306] FastCGI: incomplete headers (0 bytes) received from server "/var/XXX/clients/XXX/XXX/cgi-bin/php7-fcgi-*-443-it.mga.com"

ownCloud log (data/owncloud.log)

eqId":"yKsUAo6Olzfy0HP+dx3Z","remoteAddr":"192.168.1.2","app":"core","message":"Login failed: 'apc' (Remote IP: '192.168.1.2')","level":2,"time":"2016-03-24T19:18:41+00:00"}
{"reqId":"BFdB08p0dkI7GAQc2Z7Z","remoteAddr":"192.168.1.2","app":"no app in context","message":"Exception: {\"Exception\":\"OCP\\\\Files\\\\NotFoundException\",\"Message\":\"\\\/soft\\\/Programas\\\/iso2usb\\\/tuxboot-0.8.3.exe not found while trying to get owner\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/filesystem.php(913): OC\\\\Files\\\\View->getOwner('\\\/soft\\\/Programas...')\\n#1 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(191): OC\\\\Files\\\\Filesystem::getOwner('\\\/soft\\\/Programas...')\\n#2 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(145): OCA\\\\Activity\\\\FilesHooks->getSourcePathAndOwner('\\\/soft\\\/Programas...')\\n#3 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(101): OCA\\\\Activity\\\\FilesHooks->addNotificationsForFileAction('\\\/soft\\\/Programas...', 'file_created', 'created_self', 'created_by')\\n#4 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooksstatic.php(60): OCA\\\\Activity\\\\FilesHooks->fileCreate('\\\/soft\\\/Programas...')\\n#5 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/hook.php(105): OCA\\\\Activity\\\\FilesHooksStatic::fileCreate(Array)\\n#6 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/filechunking.php(223): OC_Hook::emit('OC_Filesystem', 'post_create', Array)\\n#7 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(436): OC_FileChunking->file_assemble(Object(OCA\\\\Files_Trashbin\\\\Storage), 'files\\\/soft\\\/Prog...', '\\\/martinb\\\/files\\\/...')\\n#8 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(103): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->createFileChunked(Resource id #328)\\n#9 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/directory.php(134): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #328)\\n#10 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1036): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('tuxboot-0.8.3.e...', Resource id #328)\\n#11 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('soft\\\/Programas\\\/...', Resource id #328, NULL)\\n#12 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#13 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#14 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(459): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#15 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(248): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#16 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(55): Sabre\\\\DAV\\\\Server->exec()\\n#17 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/remote.php(138): require_once('\\\/var\\\/www\\\/client...')\\n#18 {main}\",\"File\":\"\\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/view.php\",\"Line\":1583}","level":3,"time":"2016-03-24T19:19:27+00:00"}
{"reqId":"IHfPmO1eM6+12tCntmmF","remoteAddr":"192.168.1.2","app":"no app in context","message":"Exception: {\"Exception\":\"OCP\\\\Files\\\\NotFoundException\",\"Message\":\"\\\/soft\\\/Programas\\\/joomla\\\/core\\\/3.4\\\/3.4.3\\\/Joomla_3.4.3-Stable-Full_Package.zip not found while trying to get owner\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/filesystem.php(913): OC\\\\Files\\\\View->getOwner('\\\/soft\\\/Programas...')\\n#1 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(191): OC\\\\Files\\\\Filesystem::getOwner('\\\/soft\\\/Programas...')\\n#2 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(145): OCA\\\\Activity\\\\FilesHooks->getSourcePathAndOwner('\\\/soft\\\/Programas...')\\n#3 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(101): OCA\\\\Activity\\\\FilesHooks->addNotificationsForFileAction('\\\/soft\\\/Programas...', 'file_created', 'created_self', 'created_by')\\n#4 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooksstatic.php(60): OCA\\\\Activity\\\\FilesHooks->fileCreate('\\\/soft\\\/Programas...')\\n#5 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/hook.php(105): OCA\\\\Activity\\\\FilesHooksStatic::fileCreate(Array)\\n#6 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/filechunking.php(223): OC_Hook::emit('OC_Filesystem', 'post_create', Array)\\n#7 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(436): OC_FileChunking->file_assemble(Object(OCA\\\\Files_Trashbin\\\\Storage), 'files\\\/soft\\\/Prog...', '\\\/martinb\\\/files\\\/...')\\n#8 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(103): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->createFileChunked(Resource id #328)\\n#9 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/directory.php(134): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #328)\\n#10 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1036): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('Joomla_3.4.3-St...', Resource id #328)\\n#11 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('soft\\\/Programas\\\/...', Resource id #328, NULL)\\n#12 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#13 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#14 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(459): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#15 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(248): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#16 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(55): Sabre\\\\DAV\\\\Server->exec()\\n#17 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/remote.php(138): require_once('\\\/var\\\/www\\\/client...')\\n#18 {main}\",\"File\":\"\\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/view.php\",\"Line\":1583}","level":3,"time":"2016-03-24T19:21:01+00:00"}
{"reqId":"b5tL16ANZWItah\/uW6rM","remoteAddr":"192.168.1.2","app":"no app in context","message":"Exception: {\"Exception\":\"OCP\\\\Files\\\\NotFoundException\",\"Message\":\"\\\/soft\\\/Programas\\\/joomla\\\/core\\\/3.4\\\/old\\\/3.3.6\\\/Joomla_3.3.6-Stable-Full_Package.zip not found while trying to get owner\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/filesystem.php(913): OC\\\\Files\\\\View->getOwner('\\\/soft\\\/Programas...')\\n#1 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(191): OC\\\\Files\\\\Filesystem::getOwner('\\\/soft\\\/Programas...')\\n#2 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(145): OCA\\\\Activity\\\\FilesHooks->getSourcePathAndOwner('\\\/soft\\\/Programas...')\\n#3 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(101): OCA\\\\Activity\\\\FilesHooks->addNotificationsForFileAction('\\\/soft\\\/Programas...', 'file_created', 'created_self', 'created_by')\\n#4 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooksstatic.php(60): OCA\\\\Activity\\\\FilesHooks->fileCreate('\\\/soft\\\/Programas...')\\n#5 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/hook.php(105): OCA\\\\Activity\\\\FilesHooksStatic::fileCreate(Array)\\n#6 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/filechunking.php(223): OC_Hook::emit('OC_Filesystem', 'post_create', Array)\\n#7 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(436): OC_FileChunking->file_assemble(Object(OCA\\\\Files_Trashbin\\\\Storage), 'files\\\/soft\\\/Prog...', '\\\/martinb\\\/files\\\/...')\\n#8 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(103): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->createFileChunked(Resource id #328)\\n#9 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/directory.php(134): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #328)\\n#10 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1036): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('Joomla_3.3.6-St...', Resource id #328)\\n#11 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('soft\\\/Programas\\\/...', Resource id #328, NULL)\\n#12 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#13 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#14 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(459): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#15 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(248): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#16 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(55): Sabre\\\\DAV\\\\Server->exec()\\n#17 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/remote.php(138): require_once('\\\/var\\\/www\\\/client...')\\n#18 {main}\",\"File\":\"\\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/view.php\",\"Line\":1583}","level":3,"time":"2016-03-24T19:21:12+00:00"}
{"reqId":"BpgfGBqn0xrqvtVmjyp7","remoteAddr":"192.168.1.2","app":"no app in context","message":"Exception: {\"Exception\":\"OCP\\\\Files\\\\NotFoundException\",\"Message\":\"\\\/soft\\\/Programas\\\/joomla\\\/core\\\/3.5\\\/3.5.0\\\/Joomla_3.5.0-Stable-Full_Package.zip not found while trying to get owner\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/filesystem.php(913): OC\\\\Files\\\\View->getOwner('\\\/soft\\\/Programas...')\\n#1 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(191): OC\\\\Files\\\\Filesystem::getOwner('\\\/soft\\\/Programas...')\\n#2 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(145): OCA\\\\Activity\\\\FilesHooks->getSourcePathAndOwner('\\\/soft\\\/Programas...')\\n#3 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(101): OCA\\\\Activity\\\\FilesHooks->addNotificationsForFileAction('\\\/soft\\\/Programas...', 'file_created', 'created_self', 'created_by')\\n#4 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooksstatic.php(60): OCA\\\\Activity\\\\FilesHooks->fileCreate('\\\/soft\\\/Programas...')\\n#5 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/hook.php(105): OCA\\\\Activity\\\\FilesHooksStatic::fileCreate(Array)\\n#6 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/filechunking.php(223): OC_Hook::emit('OC_Filesystem', 'post_create', Array)\\n#7 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(436): OC_FileChunking->file_assemble(Object(OCA\\\\Files_Trashbin\\\\Storage), 'files\\\/soft\\\/Prog...', '\\\/martinb\\\/files\\\/...')\\n#8 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(103): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->createFileChunked(Resource id #328)\\n#9 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/directory.php(134): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #328)\\n#10 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1036): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('Joomla_3.5.0-St...', Resource id #328)\\n#11 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('soft\\\/Programas\\\/...', Resource id #328, NULL)\\n#12 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#13 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#14 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(459): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#15 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(248): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#16 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(55): Sabre\\\\DAV\\\\Server->exec()\\n#17 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/remote.php(138): require_once('\\\/var\\\/www\\\/client...')\\n#18 {main}\",\"File\":\"\\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/view.php\",\"Line\":1583}","level":3,"time":"2016-03-24T19:21:25+00:00"}
{"reqId":"QPxQx5jaQ0\/If3sn+CiI","remoteAddr":"192.168.1.2","app":"core","message":"Login failed: 'apc' (Remote IP: '192.168.1.2')","level":2,"time":"2016-03-24T19:30:40+00:00"}
{"reqId":"6ZuftMwfs73rfrom29OC","remoteAddr":"192.168.1.2","app":"no app in context","message":"Exception: {\"Exception\":\"OCP\\\\Files\\\\NotFoundException\",\"Message\":\"\\\/soft\\\/Programas\\\/joomla\\\/extensiones\\\/Componentes\\\/chat\\\/jbolo 2.7\\\/jbolo_2.7.zip not found while trying to get owner\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/filesystem.php(913): OC\\\\Files\\\\View->getOwner('\\\/soft\\\/Programas...')\\n#1 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(191): OC\\\\Files\\\\Filesystem::getOwner('\\\/soft\\\/Programas...')\\n#2 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(145): OCA\\\\Activity\\\\FilesHooks->getSourcePathAndOwner('\\\/soft\\\/Programas...')\\n#3 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(101): OCA\\\\Activity\\\\FilesHooks->addNotificationsForFileAction('\\\/soft\\\/Programas...', 'file_created', 'created_self', 'created_by')\\n#4 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooksstatic.php(60): OCA\\\\Activity\\\\FilesHooks->fileCreate('\\\/soft\\\/Programas...')\\n#5 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/hook.php(105): OCA\\\\Activity\\\\FilesHooksStatic::fileCreate(Array)\\n#6 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/filechunking.php(223): OC_Hook::emit('OC_Filesystem', 'post_create', Array)\\n#7 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(436): OC_FileChunking->file_assemble(Object(OCA\\\\Files_Trashbin\\\\Storage), 'files\\\/soft\\\/Prog...', '\\\/martinb\\\/files\\\/...')\\n#8 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(103): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->createFileChunked(Resource id #328)\\n#9 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/directory.php(134): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #328)\\n#10 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1036): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('jbolo_2.7.zip-c...', Resource id #328)\\n#11 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('soft\\\/Programas\\\/...', Resource id #328, NULL)\\n#12 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#13 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#14 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(459): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#15 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(248): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#16 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(55): Sabre\\\\DAV\\\\Server->exec()\\n#17 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/remote.php(138): require_once('\\\/var\\\/www\\\/client...')\\n#18 {main}\",\"File\":\"\\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/view.php\",\"Line\":1583}","level":3,"time":"2016-03-24T19:34:13+00:00"}
{"reqId":"5zsHMJBXqiGdGUZqXCkC","remoteAddr":"192.168.1.2","app":"no app in context","message":"Exception: {\"Exception\":\"OCP\\\\Files\\\\NotFoundException\",\"Message\":\"\\\/soft\\\/Programas\\\/joomla\\\/extensiones\\\/Componentes\\\/clasificados\\\/joomlistings\\\/jl217.zip not found while trying to get owner\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/filesystem.php(913): OC\\\\Files\\\\View->getOwner('\\\/soft\\\/Programas...')\\n#1 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(191): OC\\\\Files\\\\Filesystem::getOwner('\\\/soft\\\/Programas...')\\n#2 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(145): OCA\\\\Activity\\\\FilesHooks->getSourcePathAndOwner('\\\/soft\\\/Programas...')\\n#3 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooks.php(101): OCA\\\\Activity\\\\FilesHooks->addNotificationsForFileAction('\\\/soft\\\/Programas...', 'file_created', 'created_self', 'created_by')\\n#4 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/activity\\\/lib\\\/fileshooksstatic.php(60): OCA\\\\Activity\\\\FilesHooks->fileCreate('\\\/soft\\\/Programas...')\\n#5 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/hook.php(105): OCA\\\\Activity\\\\FilesHooksStatic::fileCreate(Array)\\n#6 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/filechunking.php(223): OC_Hook::emit('OC_Filesystem', 'post_create', Array)\\n#7 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(436): OC_FileChunking->file_assemble(Object(OCA\\\\Files_Trashbin\\\\Storage), 'files\\\/soft\\\/Prog...', '\\\/martinb\\\/files\\\/...')\\n#8 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/file.php(103): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->createFileChunked(Resource id #328)\\n#9 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/lib\\\/connector\\\/sabre\\\/directory.php(134): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #328)\\n#10 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1036): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('jl217.zip-chunk...', Resource id #328)\\n#11 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('soft\\\/Programas\\\/...', Resource id #328, NULL)\\n#12 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#13 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#14 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(459): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#15 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(248): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#16 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(55): Sabre\\\\DAV\\\\Server->exec()\\n#17 \\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/remote.php(138): require_once('\\\/var\\\/www\\\/client...')\\n#18 {main}\",\"File\":\"\\\/var\\\/www\\\/clients\\\/client1\\\/web2\\\/web\\\/cloud\\\/lib\\\/private\\\/files\\\/view.php\",\"Line\":1583}","level":3,"time":"2016-03-24T19:39:44+00:00"}
{"reqId":"t3nRvb3sClxUaoyHYadL","remoteAddr":"192.168.1.2","app":"core","message":"Login failed: 'apc' (Remote IP: '192.168.1.2')","level":2,"time":"2016-03-24T19:42:40+00:00"}
{"reqId":"hf5t3CPzKWZhtambZVx2","remoteAddr":"192.168.1.2","app":"core","message":"Login failed: 'apc' (Remote IP: '192.168.1.2')","level":2,"time":"2016-03-24T19:46:39+00:00"}
{"reqId":"a9wc0cf2zFscw+3fkZ\/6","remoteAddr":"192.168.1.2","app":"core","message":"Login failed: 'apc' (Remote IP: '192.168.1.2')","level":2,"time":"2016-03-24T19:46:40+00:00"}
{"reqId":"jgyAzSCjjaQPFDMXqKn1","remoteAddr":"192.168.1.2","app":"core","message":"Login failed: 'apc' (Remote IP: '192.168.1.2')","level":2,"time":"2016-03-24T19:46:40+00:00"}
{"reqId":"7NBJLMA7BClg73yp6sJs","remoteAddr":"192.168.1.2","app":"core","message":"Login failed: 'apc' (Remote IP: '192.168.1.2')","level":2,"time":"2016-03-24T19:46:46+00:00"}

[enoch85]

@ghiamar PHP-FPM is not as fast as CLI for some reason. For me with almost the same setup (CLI) it takes 4-5 minutes.

[mgscreativa]

Hi @enoch85! Would you please disclose your detailed apache/php/apcu config so I can try that?

Thanks!

[enoch85]

My setup involves a NGINX reverse proxy, Apache 2.4, MySQL 5.5, Redis cache, and a regular core i5 processor (3570K) clocked at 4.2 Ghz. My network is 100/100 Mb Swedish fiber and performs very good. #20963 (comment)

It would be eaiser for you of you want to test "my system" to download this VM: https://www.techandme.se/pre-configured-owncloud-installaton/

You could also connect to my demoserver: https://owncloudphp7.techandme.se
USER: testuser
PASS: testingphp7

[mgscreativa]

Ok @enoch85, so if I got this right, I shoud try your vm and check the config there to try to figure it out best config for me, that's right?

Thanks!

[enoch85]

@ghiamar I would leave PHP-FPM if possible. There are no special config tweaks or so, just a basic install really, but with CLI and not FPM. Then it could be so many different factors to your slowness than just your conifg, network, server, and so on...

Here are the script(s) I use for installing: https://github.com/enoch85/ownCloud-VM/blob/master/production/owncloud_install_production.sh

[mgscreativa]

Thanks @enoch85, will check that script and let you know

[mgscreativa]

Hi @enoch85! I want to report that following your VM scripts installation I finally got the speed desired for the stress test (around 5 minutes).

I did clean install Ubuntu Server 14.04.4, and manually followed your scripts commands and got it just fine. Now I will try to replicate that on my production server to see if I can get that speed. Maybe the REDIS cache will do the trick, will check and let you know.

[mgscreativa]

Well, after the whole day testing, I may say that maybe is something wrong with my server config/setup, because trying the VM methox from here https://github.com/enoch85/ownCloud-VM/blob/master/production/owncloud_install_production.sh it works just fine and synces all small files in about 5 minutes and on my real server it takes about 1 hour!, at least I was able to reduce it from 3 hours!

Thanks a lot for all the support!

[divinity76]

the problem's still present.
it looks like a Nagle Algorithm problem, you'll have the same effect with SSH if you disable TCP_NODELAY in your ssh client (not all ssh clients support disabling it, but PuTTY does. all sane SSH clients enable it by default, though);
add TCP_NODELAY to the OwnCloud client socket, does that make small files faster?

[mgscreativa]

hi @divinity76!, Where should I add that exactly?

By the way, I have tested in three scenarios, one VM installed with https://github.com/enoch85/ownCloud-VM/blob/master/production/owncloud_install_production.sh, a second VM using a fresh Ubuntu Server 14.04.4 installing ISPConfig and a third scenario using a owncloud docker image, in all of them, the file syncing goes just fine, the only slow thing is my real LAN server.

I really don't know what can be causing the slowness, but I really don't care now, because I know that is my server, will waith till Ubuntu Server 16.04 releases so I can test in that new version...

[divinity76]

do i look like an ownCloud Client dev? :p or to say it another way, i don't know, but my best guess is
client/src/libsync/abstractnetworkjob.cpp in void AbstractNetworkJob::setupConnections

[guruz]

If you don't have issues with server performance/load I suggest to try a nightly build of oC client 2.2.x
It automatically syncs more files at once if they are small files
http://download.owncloud.com/desktop/daily/
https://software.opensuse.org/download.html?project=isv%3AownCloud%3Acommunity%3Anightly&package=owncloud-client

[mgscreativa]

Nope...installed Revisión Git 173056 en Mar 30 2016 but the slowness persists. I really don't care now, because I was able to check that is an issue on my 2 years old server installation, maybe in that time, I setup something that causes the pauses for security. Anyway, will wait to Ubuntu Server 16.04 release an try again. Will let you know then.

[divinity76]

@ghiama why do you think this is an OS issue?

[mgscreativa]

Hi @divinity76.

I think of it because the test I've made, in three different VM scenarios, using my Laptop connected to my WiFi LAN the stress test files sync is really quick, but testing to my real LAN server, the stress test files sync take a very long time to sync compared to the other scenarios.

As you can see on my previous posts here, you can check my server specs and realize that is solid rock, but still OC is slow, so, in my opinion, maybe there's something I installed on my server on this two years run that maybe, got this slowness with OC, maybe it's a security measure, because the files sync very fast, but by batches of three.

Thanks in you interest.

[adrianvg]

@guruz;
Are there any nightly builds for CentOS 7 available as well?
I took the (maybe premature) step to move my local homefolder to Owncloud to get a sort of poor-mans-backup solution, and am now seeing very slow update checks.

[dragotin]

Sure, here is the centos 7 nightly: http://download.opensuse.org/repositories/isv:/ownCloud:/community:/nightly/CentOS_7/

[adrianvg]

I must be going blind...
Thanks @dragotin !

[mSys-mislav]

Why is this closed if it's not fixed? There is still issue with small file uploading - even on 9.0.2. - as mentioned above, bigger file upload is not the problem, but small file upload.

I've tried to upload through my linux network drive, mapping the same in windows machine, WinSCP - it's slow, very slow. There are no errors in logs, just slow speed - in bytes!!!

I've created few hundred of files of size in range of 10-1000kB and always the same problem. I've also tried setting as suggested "innodb_flush_log_at_trx_commit = 2" but that didn't help. We also changed disks, also no luck.

Any idea how to proceed? It's making a huge problem for us/clients.

[PVince81]

@mislav-eu it's likely closed because the slowness is on a case by case basis. For one person it can be environment issue A. For another it's environment issue B. For some people and environments it's not a problem at all.

In your case need to find out what environment you are using so best would be to make a new report using the issue template: https://raw.githubusercontent.com/owncloud/core/master/issue_template.md

I assume that you already checked https://owncloud.org/support/ or the forums for known environment issues that would cause upload slowness ?

[mSys-mislav]

I will open then new report. But yes, mostly people have the same problem:
https://forum.owncloud.org/viewtopic.php?t=17477
https://forum.owncloud.org/viewtopic.php?t=30083

I've tried mentioned things, but no result.

[PVince81]

During the upload of small files there is no chunking involved, so no chunk assembly.

This means that in most cases the file is directly streaming by Apache to PHP and PHP passes it to the ownCloud code which stream-copies it into a part file. Then the part file gets renamed to the final file. So the part where the bytes are copied should be optimal.
Now if there's encryption in the middle it can be a little slower because the stream copy also encrypts every block.

So the slowness could be due to the stream copying from the web server to PHP being slow. Some web servers even create a temporary file before streaming the file to ownCloud (seen it happening with some nginx settings).

Then the other part which could be slow is setting up the network connection and later on PHP setting up its internal filesystem, but this happens before the stream copy. So the question is whether the transfer speed values are based on the stream copy part or whether they look lower (averaged?) because the start of the transfer was slower.

Also to check if whether the target folder is a received shared folder, and whether that one was reshared. If the target folder is on external storage, it will likely be much slower due to the extra upload to the external storage.

[mSys-mislav]

Dear @PVince81 - I will create separated report for this. #25115

[PVince81]

@mislav-eu thanks!