Cyclic sync problems still in 4.5.6 with umlauts

[childnode]

While having sync problems on 4.5.5 I thought 4.5.6 will fix that but it won't:
^^ refer to those bugs: https://github.com/owncloud/core/issues?labels=Bug%2Cv4.5.5&page=2&state=closed

like #1058

the environment
The filename / path is as following:
<OWNCLOUD-STORAGE>/02_Rechnungen & Auftragsbestätigungen/*

Clients tested: Windows 1.2.0 and Mac 1.2.0

Owncloud Server updated to 4.5.6 yesterday.

the problem

• I freshly created a full-sync of the owncloud storage (using / not /clientsync)

• sync started, went fine until the path with umlauts got reached

• Now I'll find one file on my client:
  ~/ownCloud/02_Rechnungen & Auftragsbestätigungen/o2/2012-11-01-1234567890_01-RG.PDF

• on serverside, a new / second directory has been created which is named exactly the same!

• both URLs on owncloud webclient ends up with
  &dir=//02_Rechnungen %26 Auftragsbestätigungen

• on server side, both directories seems to be named exactly the same same too (very strange)

  drwxr-xr-x 19 www-data www-data 4.0K Oct 7 15:14 02_Rechnungen & Auftragsbestätigungen
  drwxr-xr-x 3 www-data www-data 4.0K Feb 18 23:04 02_Rechnungen & Auftragsbestätigungen

• LANG=en_US.UTF-8 console encoding UTF-8

• tab-completion with UTF-8 ä will enhance to the NEW directory.

• Directories displayed in ISO:

  drwxr-xr-x 19 www-data www-data 4.0K Oct 7 15:14 02_Rechnungen & Auftragsbestätigungen
  drwxr-xr-x 3 www-data www-data 4.0K Feb 18 23:04 02_Rechnungen & Auftragsbestätigungen

• on client side the sync tool will cyclic fall into "CSYNC Error, then syncs automatically again, will notify that one file in the directory 02_Rechnungen & Auftragsbestätigungen/o2 has been synced (and yes, it is here) but then falls back to the same error and loop again

so:

1. how to solve this locally. What happens if I rename one of the "doubled" directories.
   1.1. will owncloud delete files because of
2. should I Rsync / Download all Files from Server, startup with a clean environment to then re-add all files via client?
3. the Umlaut bugs are NOT fixed...please test this with server side non-UTF8 files too.

[BernhardPosselt]

cc @dragotin @danimo

[dragotin]

This seems to be a server setup problem. We tried to reproduce this with Mac (client side) and it worked fine with the mentioned folder.

@childnode what do you mean with 3. ..."server side non-UTF8 files"?

[childnode]

@dragotin it seem to, that this issue came up in of a combination of

• using owncloud and its client since beta
• in any constellation before, the server files got saved in wrong format (non-UTF8) but it looked like correct UTF-8
  While "Auftragsbestätigungen" is the latin1 encoded UTF-8 String "Auftragsbestätigungen" (Umlaut: Hex C3 A4 == U+00E4 == ä == UTF-8 LATIN SMALL LETTER A WITH DIAERESIS) the old directory displayed like "Auftragsbestätigungen" in latin1 (Umlaut: Hex 61 cc 88 == a (61) PLUS U+0308 == COMBINING DIAERESIS) console has some strange umlauts in neither correct UTF8 nor latin encoded. Perhaps it was saved once in a false encoding on sync with an old client version and somewhere rewritten. possibly this happened several times in distortion order so this umlaut got broken.

unfortunatly I'm also not able to reproduce this atm ;(

server didn't changed since one year, so the setup wasn't the problem. It's still the same and correct (for me).

so back to your question: can you please test what happens if you'll "inject" some wrong encoded files in the /files folder on your serverside and resync? It seems to error handling if there are two files calling the same in UTF-8 but written either in latin1 and any other non-utf8 encoding.
Anyway: you might see in the description above, there seems to be a change in writing umlauts on sync. It was both "correct" UTF-8 but the older one was created by using combinating characters.

so I hope the problem is a little bit clearer now and I didn't write too confusing ;)

[danimo]

@childnode That explains it then. Mac has a different Unicode normalization (http://en.wikipedia.org/wiki/Unicode_equivalence#Combining_and_precomposed_characters). We fixed the mac client to do a NFD->NFC conversion for the mac when uploading, but unfortunately. We cannot convert the files on the server retrospectively.

[childnode]

so then:

• there should be an advisory for those who (used) that mac version with NFD.
• perhaps you might provide a "upgrade script" to fix it accordingly or list all filenames that include cc 80 - cd af combining characters?! Perhaps http://docs.oracle.com/javase/tutorial/i18n/text/normalizerapi.html or http://durak.org/sean/pubs/software/php-5.4.6/class.normalizer.html might help to do this?!

[danimo]

@childnode Is the server also running on Mac OS?

[childnode]

@danimo no, the server is a classic Debian server on ext3 with apache

[danimo]

@childnode: Sounds to me like the directory was uploaded with a mac client prior to 1.1.4. That would explain the problems you are seeing.

[childnode]

Just to have said, i've fixed it manually for my instance (I think) by renaming all in Server, then resync. Seems to i catched up all files

[danimo]

@childnode Ok, so those are legacy problems then, or can you reproduce them with newly synced files/directories?

[childnode]

No, currently all working fine except my other issue regarding sync state client problems :-)
But I can't believe to be the only one having this problem? Perhaps all others were frustrated and killed owncloud then went back to dropbox :-D

[jancborchardt]

I’m closing this issue because it has been inactive for a few months. This probably means that the issue is not reproducible or it has been fixed in a newer version.

Please reopen if you still encounter this issue with the latest stable version (currently ownCloud 5.0.9) and then please use the issue template. You an also contribute directly by providing a patch – see the developer manual. :)

Thank you!