owncloud fails to synchronize directories with large amount of files

[cyrilleberger]

Expected behaviour

owncloud synchronize successfully large amount of files.

Actual behaviour

owncloud fails to synchronize.

Steps to reproduce

1. Have a directory with many directories countaining many files (like a music collection)
2. Try to synchronize it on a computer

Server configuration

Operating system: Debian 6.0

Web server: Apache (problem occurs with https and http)

Database: MySql

PHP version: 5.3.3-7+squeeze14

ownCloud version: 4.5.5

Client configuration

Client version: 1.2.0, from the debian repository on the build service (it also happened with older version)

Operating system: Linux, Debian

OS language: english

Installation path of client: /usr/bin/owncloud

Logs

output of owncloud --logwindow or owncloud --logfile log.txt

01-29 10:20:45:792 oc_module: opendir method called on ownclouds://owncloud.XXXXX.net/remote.php/webdav/Music/XXXXX
01-29 10:20:45:867 oc_module: Simple propfind result code 200.
01-29 10:20:45:867 oc_module: ERROR: Content type of propfind request not XML: text/html.
01-29 10:20:45:867 oc_module: WRN: propfind named failed with 5, request error: 200 OK
01-29 10:20:45:867 oc_module: Errno set to 10011
01-29 10:20:45:867 csync_ftw: opendir failed for ownclouds://owncloud.XXXXX.net/remote.php/webdav/Music/XXXXX - Unknown error 10011 (errno 10011)
01-29 10:20:45:867 oc_module: closedir method called 0x7f63b428df40!
01-29 10:20:45:867 oc_module: closedir method called 0x7f63b4842040!
01-29 10:20:45:867 csync_update: Update detection for remote replica took 7.93 seconds walking 1337 files.
01-29 10:20:45:867 csync_memstat_check: Memory: 633316K total size, 100360K resident, 26836K shared
01-29 10:20:45:867 #### ERROR csync_update: "A HTTP transmission error happened.
Backend Message: Server error: PROPFIND reply is not XML formatted!"

[cyrilleberger]

It is worth to note that the synchronization fails on different file each time. Which led me to write the following patch to the csync's owncloud module, using the "dav" branch (sorry, no idea how to attach files in github):

diff --git a/modules/csync_owncloud.c b/modules/csync_owncloud.c
index ff30115..899f2f0 100644
--- a/modules/csync_owncloud.c
+++ b/modules/csync_owncloud.c
@@ -953,8 +953,6 @@ static void results(void *userdata,
     /* DEBUG_WEBDAV( "results for URI %s: %d %d", newres->name, (int)newres->size, (int)newres->type ); */
 }

-
-
 /*
  * fetches a resource list from the WebDAV server. This is equivalent to list dir.
  */
@@ -1111,6 +1109,20 @@ static struct listdir_context *fetch_resource_list(const char *uri, int depth)
     return fetchCtx;
 }

+static struct listdir_context *fetch_resource_list_attempts(const char *uri, int depth)
+{
+  struct listdir_context *fetchCtx = NULL;
+  for(int i = 0; i < 10; ++i)
+  {
+    fetchCtx = fetch_resource_list(uri, depth);
+    if(fetchCtx) break;
+    DEBUG_WEBDAV("=> Errno after fetch resource list for %s: %d", uri, errno);
+    DEBUG_WEBDAV("   New attempt %i", i);
+
+  }
+  return fetchCtx;
+}
+
 /*
  * helper: convert a resource struct to file_stat struct.
  */
@@ -1238,7 +1250,7 @@ static int owncloud_stat(const char *uri, csync_vio_file_stat_t *buf) {
     }

     /* fetch data via a propfind call. */
-    fetchCtx = fetch_resource_list( uri, NE_DEPTH_ONE);
+    fetchCtx = fetch_resource_list_attempts( uri, NE_DEPTH_ONE);
     DEBUG_WEBDAV("=> Errno after fetch resource list for %s: %d", uri, errno);
     if (!fetchCtx) {
         return -1;
@@ -1762,7 +1774,7 @@ static csync_vio_method_handle_t *owncloud_opendir(const char *uri) {

     dav_connect( uri );

-    fetchCtx = fetch_resource_list( uri, NE_DEPTH_ONE );
+    fetchCtx = fetch_resource_list_attempts( uri, NE_DEPTH_ONE );
     if( !fetchCtx ) {
         /* errno is set properly in fetch_resource_list */
         DEBUG_WEBDAV("Errno set to %d", errno);

the patch basically retries 10 times after a failure in "fetch_resource_list", it is very ugly, but:

1. it solves the issue, owncloud is able to synchronize my music collection, while without the patch after an hour, no file have been downloaded
2. when I look in the log, the "PROPFIND" error still happen (obviously) but it is always successfull after a second try

[cyrilleberger]

I found other people with a similar problem on the forum http://forum.owncloud.org/viewtopic.php?f=14&t=6551 (last message), http://forum.owncloud.org/viewtopic.php?f=14&t=5109 (no log error, but difficulties to synchronize 12000 files).

[dragotin]

@cyrilleberger thanks a lot for the patch! Seems like a good idea. However, I will try to involve server devs to investigate why the PROPFIND fails.

[dragotin]

Hmm, I tried with a torture test with 51495 in many directories. The problem did not appear. How can we reproduce this problem? What is special with your setups?

[dragotin]

I added commit 6d7040467bf4f7e745780240e0e63af6b8254068 to csync to at least temporarily fix this as suggested. I also added some code that logs the answer coming from the server in the error case. This should show us what the problem actually is.

[czerwinski1977]

I have the same problem in my installation. Unforunately I failed to retrieve the commit mentioned in the previous post (http 404). Is there maybe a nightly build for the client to download and test?

[danimo]

We have nightly builds for windows and mac here: http://download.owncloud.com/download/nightly/

[czerwinski1977]

Daniel, thanks for the quick reply! I work on Ubuntu Linux, so I am afraid these builds don't suit my system... No Linux builds available? Thanks anyway!

[danimo]

No, building for all the linux distros is a bit more effort, we are still working on getting them done.

[czerwinski1977]

Ok, I made it to build the client (1.2.1pre) from latest git repo. However, the problem persisted. I dug into the sources and found:
(1) The error regarding PROPFIND not being XML formatted is caused by PHP notices -- found out by parsing network packages via wireshark. (Actually, this can also be found in a PHP log file on the server.)
(2) The first PHP notice is "Undefined index: id in /mnt/HD_a2/www/pages/owncloud/lib/filecache.php on line 381" -- previous statement in the source (on the server, of course) was "$parent = OC_FileCache_Cached::get($path);"
(3) the OC_FileCache_Cached::get($path) function from the file owncloud/lib/filecache/cached.php returns an empty array (line 43), which causes the caller to fail.

I don't know how the empty array is supposed to work anyway, but I became aware that the problem has got to do with caching. I decided to rescan the filesystem using the "Rescan Filesystem Icon" app on the web interface (as admin). Synchronization started and came up with several conflicts (which I am unable to interpret right now), and eventually succeeded.

I hope this contributes in clearing this up :)

Cheers, Reinhard.

[gabooh]

Having this problem too I guess. mirall 1.2.0 from ubuntu repository. Owncloud 4.5.7. Will try as soon as a binary is available :)

Edit : not better with mirall-1.2.1nightly5.tar.bz2 from ubuntu devel:daily repository.

I have been getting this error for a very long time now. What could I do to help ?

[gabooh]

I get theses errors on each try:

[ ... ]
02-21 12:15:46:597 oc_module: opendir method called on ownclouds://.org/remote.php/webdav/Photos/**
02-21 12:15:46:704 oc_module: Simple propfind result code 200.
02-21 12:15:46:704 oc_module: ERROR: Content type of propfind request not XML: text/html.
02-21 12:15:46:704 oc_module: WRN: propfind named failed with 5, request error: 200 OK
02-21 12:15:46:704 oc_module: Errno set to 10011
02-21 12:15:46:704 csync_ftw: opendir failed for ownclouds://_.org/remote.php/webdav/Photos/_**** - Erreur inconnue 10011 (errno 10011)
02-21 12:15:46:704 oc_module: closedir method called 0x7f025a758590!
02-21 12:15:46:704 oc_module: closedir method called 0x7f0254f8ed60!
02-21 12:15:46:704 csync_update: Update detection for remote replica took 11,11 seconds walking 1372 files.
02-21 12:15:46:704 csync_memstat_check: Memory: 1145312K total size, 174236K resident, 28216K shared
02-21 12:15:46:704 #### ERROR during csync_update : "Une erreur de transmission HTTP s'est produite.
Message sous-jacent :Server error: PROPFIND reply is not XML formatted!"
02-21 12:15:46:707 csync_lock_remove: Removing lock file: /home/gaboo/.local/share/data//ownCloud//lock
02-21 12:15:46:717 CSync run took 14104 Milliseconds
02-21 12:15:51:049 Check status.php from statusdialog.
02-21 12:15:51:049 Get Request to "status.php"

The file on which it stops is also a different one each time. The sync used to sometimes work if I let it try for a few hours, but in fact in now never works.

The folder is a big one I guess : 63.8gb, ~25000 files, 700+ subfolders.

[dragotin]

@gabooh have you also updated the ocsync package to the nightly version (currently nightly25)? That contains the workaround suggested above.

[gabooh]

@dragotin probably not, trying now. thanks :)

[gabooh]

An apt-get update / upgrade does not automatically update all packages, not sure why. Removing and installing owncloud-client and ocsync* again fetched all latest ones from devel/daily repository.

The workaround seams working 💃

I still get the error, but the retry seams enough to make it work :

02-21 13:09:34:060 oc_module: opendir method called on ownclouds://_/remote.php/webdav/Photos/****
02-21 13:09:34:235 oc_module: Simple propfind result code 200.
02-21 13:09:34:235 oc_module: ERROR: Content type of propfind request not XML: text/html.
02-21 13:09:49:224 oc_module: ERROR: Content was of size -1:
02-21 13:09:49:224 oc_module: WRN: propfind named failed with 5, request error: Ne peut lire la taille du chunk: connection was closed by server
02-21 13:09:49:224 oc_module: => Errno after fetch resource list for ownclouds://*******/remote.php/webdav/Photos/_*: 10011
02-21 13:09:49:224 oc_module: New attempt 0
02-21 13:09:49:556 oc_module: Simple propfind result code 207.

Thank you @cyrilleberger and @dragotin !

If you need more testing, let me know.

[tomse]

I suspect I have the same issue, symptoms are the same.
Client 1.2.1 says in info: Undefined Error State
Server is 4.5.7 on FreeBSD.

(Just wanted to add this to the log as I spent countless hours trying to find out why it wouldn't sync)

Looking forward to see a new release, great work otherwise.

[dragotin]

@tomse better open a new issue if you want and add log files.

[wiz561]

Make sure your directory and file permissions are set properly. I went to..

/owncloud/remote.php/webdav/clientsync

in my web browser and it told me to fix my permissions. Once I did that, everything started working fine again.

[brainstorm]

@wiz561, which permissions and on which directories did you set them?

[brainstorm]

Could it be an issue about how the server reports the content type(s) back to the client?:

Please, have a look at this comment on issue #749 (comment)