Error when uploading changes on files with umlauts

[ir0nhide]

Expected behaviour

All changed files will be uploaded to the server.

Actual behaviour

Changes to larger files (e.g. 20 MB) are not uploaded to the server. The process terminates with the following error: Server is unable to maintain the header compression context for the connection.
This error only occurs if either the filenames or their paths contain umlauts.
With nextcloud client version 2.3.3 everything is working as expected.

Steps to reproduce

1. Create a file which contains umlauts in its filename and has a size of at least 20MB in your nextcloud folder on your desktop
2. Wait until nextcloud client uploads the file to the server
3. Edit the file and save the changes
4. Sync changes with nextcloud client

Client configuration

Client version: 2.5.2git (build 20190319)

Operating system: Windows 10 - 1803

OS language: German

Qt version used by client package (Linux only, see also Settings dialog):

Client package (From Nextcloud or distro) (Linux only):

Installation path of client: C:\Program Files (x86)\Nextcloud

[danielrheinbay]

Also affects Linux desktop client:

Client version: 2.5.2-20190319.015224
Operating system: Ubuntu 18.04
OS language: German
Qt version used by client package: Qt 5.9.5
Client package: from official Nextcloud PPA

[valentingc]

I can confirm this issue aswell - having it since 2.5.X
It works fine with 2.3.3.

OS Language: German.
Client version: v2.5.2
OS: Windows 10 Pro 1809

[PhBobo]

I have the same issue in French on Ubuntu 19.04 with file where name has accented characters, punctuation and blank space.

Client : 2.5.1git

[marco44]

Same issue with 2.5.2 on archlinux.
Syncing works with owncloud-client too, by the way.

[PhBobo]

I have the same issue in French on Ubuntu 19.04 with file where name has accented characters, punctuation and blank space.

Client : 2.5.1git

It does not seems to be a problem with accented characters, as I have a lot of other files in this case and it works fine. The problem arise with big files, in my case mp4 files.

[Serverfox]

Same issue when edditing docx file at > 11,9MB incl German Umlaut in file name like "Mündliche Prüfung Testkandidat.docx" with MS-Word 2019 under Windows Pro 1809.

No issues encountered using the NC Web app and Document Server by Onlyoffice.
Nextcloud Client: git-Revision 56c905 auf Mar 19 2019, 13:34:10 verwendet Qt 5.11.1, OpenSSL 1.0.1h 5 Jun 2014

Ubuntu Server 18.04
locale: LANG=de_DE.UTF-8 ....
nginx 1.15.7
MariaDB 10.3.11 (utf-8, big int)
PHP 7.2.10 (utf-8, 512 MB)
Nextcloud 16.0.1

[papjul]

Same issue. I'm out of sync since a few weeks now.
Could it have been caused by upgrade to Nextcloud 16? I also remember by seeing @Serverfox comment that I did the bigint conversion at approximately the time the issue began appearing.

I believe we must many others affected by this bug, can this be prioritized?

Server:
nginx 1.16.0
LANG=fr_FR.UTF-8
MariaDB utf8mb4 10.2.x
PHP 7.2.x
Nextcloud 16.0.1

Client:
Version 2.5.2
Linux (Gentoo)

[0x47]

I have the same issue even without umlauts. Manually removing the local (client) sync DB most of the time fixes it for a while. The file is hidden in the top-most Nextcloud directory (e.g. C:/Users/MyUser/Nextcloud/xyz.db). Deleting it forces a re-scan of the Nextcloud-directory and usually a bunch of files are updated that have not been synced...

[papjul]

I have the same issue even without umlauts.

I'm French, issues seems to be with special characters in general.

Manually removing the local (client) sync DB most of the time fixes it for a while. The file is hidden in the top-most Nextcloud directory (e.g. C:/Users/MyUser/Nextcloud/xyz.db). Deleting it forces a re-scan of the Nextcloud-directory and usually a bunch of files are updated that have not been synced...

Don't do this. This will not actually sync, this will just duplicate all files that were updated since the issue began, in conflict. Both conflicted versions stay on your local PC and don't actually sync. Nextcloud Desktop will say sync is up-to-date but should show you a "there are conflicted files" banner. Also, all files you deleted will reappear in your folders, so it's really not a good idea.

[0x47]

Thanks, that's good to know. I thought it was equivalent to setting up a new account and choosing the option "keep local files" in the desktop client, letting Nextcloud compare the offline and online folder structure.

So, how can this be achieved then without deleting the Files in .../AppData/Local/Nextcloud etc.? This requires to completely setup the account again, including app password. After all, there is no "repair sync database" option in the client.

[Serverfox]

This issue is 100% reproducible, 100% out of NC specification and I want this to be taken more seriously!!!

This bug prevents me to install NC in a productive environment. I've spend time and money to set up NC properly and I really do not want to be forced to get back Google Drive, Dropbox etc. Neither of these are suffering from such an issue.

[papjul]

I know :( This is a critical regression bug and we did not receive a single reply from Nextcloud developers, I don't know how we can raise awareness among the hundreds of issues here…

@marco44 Can you confirm you safely replace Nextcloud client with ownCloud desktop client with success? How did you proceed? Thanks!

@0x47 don't lose time setting up your Nextcloud desktop app for the "first time", the issue is not in the database. It will become out-of-sync again as soon as you put newer versions of files containing a special character in its name. In case of emergency, you can upload newer versions of files from the web app, but that's not convenient at all especially if you have numerous and big files. You also need to track the changes manually.

[marco44]

@papjul : It's been working flawlessly for 3 weeks. I simply uninstalled nextcloud-client, installed owncloud-client, and re-setup all the shares from scratch... As only a few files were out of sync, there weren't so many conflicts. I even have the feeling the owncloud client is faster, but of course it may only be an impression. The only downside is that I have for now is a warning each time the client starts, telling me there may be incompatibilities.

On a side note, just a reminder, this is free/open source software. If you want a prompt response, you'd get it from a support contract from the nextcloud company. Not cheap, but we're not their target. We're getting this for free. I'm sad we don't get an answer either, but nobody's owing us anything. We could diagnose the problem ourselves, we have the code. Which I may do when I have spare time, which is not now :(

[0x47]

I installed the ownCloud Client on one of my friends PC and it works flawlessly so far. Thanks for the tip! It was already kind of embarrassing having this issue after I convinced friends and family to switch over to Nextcloud from proprietary software...

[Serverfox]

@Ox47 I feel the same. NC on Ubuntu is a real alternative to Dropbox, Google Drive and the like.

Shame on the Windows client that prevents the useage in family, friends.

I was alsoup to install NC as a save platform to exchange data with my students in school. Without a stable Windows client I can't proceed.

Would you be so kind to keep us up to date on your experience with the Owncloud client?

Thank you

[valentingc]

Why don't you guys use the Nextcloud Client v2.3.3?
This one does not suffer from this issue, at least in my case.

[Serverfox]

@valentingc Found v2.3.3.1 under https://download.nextcloud.com/desktop/releases/Windows/
and it works fine with or without Umlaut. Thank you very much for your help.

Are there known limitations in this version?

[valentingc]

@Serverfox Not as far as I know. I've been using it daily basically since it was released (tried the new versions out in between but made a downgrade because of this issue). Works fine for me with the latest stable Nextcloud version (16.0.1). However, this is of course only a temporary fix in the long run.

[Serverfox]

@valentingc Let's cross fingers.

[0x47]

@Serverfox Since switching to the ownCloud client I have had no issues on Windows as well.

Because of this issue, I tried the ownCloud client on Linux as well to tackle #1205. Since then, I don't have any crashes on Linux any more.

It seems that maybe choosing Nextcloud over ownCloud was a premature decision and I need to re-evaluate in the long-term.

[Serverfox]

Seems as these issues are related to the client v2.5.x.

BTW the NC server is running without fault since 1st installation. That's the reason I'm disapointed by the client, no by NC as such.

[Serverfox]

Why don't you guys use the Nextcloud Client v2.3.3?
This one does not suffer from this issue, at least in my case.

Works perfectly since a couple of weeks, Thanks much!!

[papjul]

I don't recommend using an old version of NextCloud Client released almost 2 years ago, unless you are compiling it from sources. NextCloud Client relies on libraries such as OpenSSL 1.1.x, QtKeychain, Qt 5.x.x and zlib, which may contain severe vulnerabilities discovered since 2017 (or even in NextCloud Client itself). Your best option is to use latest Owncloud Client if you care about security.

[Serverfox]

@papjul Thanks for your advice. I swapped over to the Owncloud client, too. Works fine so far. Hoping the incompatibility warning concerning the NC Server won't do any harm on the long term.

[FreeMinded]

I'm also hit by this problem. What I found is, that moving the file causing the error out of the folder (on to the desktop i.e.) waiting for the sync to go through and then putting the same file back, makes the sync work again. Until it's modified again and the problem starts over...

In case it helps I uploaded a Sync Client Log showing the issue
nextcloud-log-obfuscated-1907022.log

[FreeMinded]

@karlitschek can you have someone look at this? This is a serious problem. Thanks!

[camilasan]

Hi @FreeMinded I was made aware of your request. I am trying to reproduce it first. Will let everyone know if I find the problem.

[FreeMinded]

Hi @camilasan thanks a lot! Let me know if I can help you with something, testing, providing logs, samples, test account...

[camilasan]

Hi @FreeMinded, I couldn't reproduce the problem. I tried against Nextcloud 16 and 17. Could you provide a testing account and one of the big files that are not getting synced (if that is possible at all)? Thanks.

[Cybso]

We're observe the same problem on our OSX clients using the web interface. I suspect that this might be related to the way Apple filesystems encode Umlauts. Apple is using NFD (Normalization Form Canonical Decomposition) while everybody else is using NFC (Normalization Form Canonical Composition). This is a problem I've also observed with SFTP transfers between OSX and Unix clients.

[dodancs]

Ubuntu 19, NextCloud client 2.6.2-20191224.111326~eoan1.

The problem is still here, I cannot sync my data successfully.

Any updates on the possible fix?

It doesn't seem to be server-side since Windows client works just fine. Even with enormous files (1GB+).

[ArchiPOS]

Having similar problems with "ö" or "ä" or "ü" when updating files via sync client.

macOS 10.14.6 using NextCloud Client 2.6.2 and 2.7 Beta
Server version 17 (update to 18 did not fix it)
Server seems not affected. Webupload also unaffected. So just a client problem.
Server log shows nothing.

Client responds with "Error transferring https://DOMAIN/remote.php/dav/uploads/USER/4191192125/.file - server replied: (File with name //Auftrge could not be" and below the Folder path "Aufträge/XY - FOLDER NAME/2.1 Entwürfe/"

All the special characters like "-" or "." and space work fine.
It is just the Umlauts "äöü" causing problems.

found a temporal fix: downgrade client to 2.5.3
This ist the last unaffected version.

[DominiqueFuchs]

Hey everyone.
I gave this another shot and have been able to safely reproduce the issue where changing a file with umlauts results in an GUI error, not syncing the file and spawning log messages regarding the final MOVE instruction of the change synchronization. This was a bug with the header definition/encoding when pushing out the corresponding DAV request and is fixed in PR #1768

As it is well possible that this issue contains more than one bug or circumstances where umlauts can lead to failures, testing is necessary and would be very welcome. However, this specific bug was squashed.

Edit: Forgot to mention that this PR is now in review - if everything is fine, I guess we're backporting this to stable very soon.

[papjul]

Good catch!
It is already part of Owncloud: owncloud/client@87a6e03#diff-f9f22ece9a01d2e1cd06e3d8a088209b
But it was used to fix a bug when uploading large files with a "%" in its name: owncloud/client#7176
Not sure it covers all cases like you said, but we will see ;)

[Milokita]

owncloud reported bug in Qt again causing "Server stopped accepting new streams before this stream was established" & disabled HTTP2.0
owncloud/client#7610

[DominiqueFuchs]

@Milokita Please lets not make this a everything-HTTP2 thread, as I can‘t see any relationship between the owncloud issue you linked and the umlaut sync issue this one cares about.

Do you experience the issue regarding network stream blocking mentioned in the OC link? If yes, please use a separate issue for this. Thanks!

[papjul]

The link between the two issues is that the first fix was to move to Qt v5.12.4 and integrate the 1st PR from @Milokita. It was supposed to fix the issue but a regression seems to have been introduced in Qt 5.12.5, through what appears to be a bugfix, making the fix useless (the Qt bug was mentioned earlier in this thread).
Everything is connected together and I came to the same conclusion as made in Owncloud bug referenced: disabling HTTP2 makes everything work flawlessly.

So yes, we can make a separate bug more generic on all HTTP2 issues on which this bug can depend since it's not reproduceable with HTTP 1.

Everything is tied together around HTTP 2 and Qt. Quoting from the owncloud bug:

the QT implemrntation of H2 has been broken since forever. There's always: "ah, there's this bug, which will be fixed in QT x.x.x", then it takes forever to switch the client to use that QT version and then it starts all over.

Which sums up approximately what we've been doing the past year on our bug (which is more like a "sync issues when HTTP2 is enabled" bug).

@Milokita Can you open a new bug with your findings? I disabled HTTP2 long time ago, it will not be very helpful if I open it myself

[DominiqueFuchs]

I understand the issue about HTTP/2, what I'm saying is that this thread (titled 'Error when uploading changes on files with umlauts') hinted on missing code fixes (#1768) that are directly tied to changes on existing files no matter the size and HTTP/2. This is what I'd like to review here as soon as the PR is merged and backported. Thanks for creating a new issue regarding the latter topic!

[realies]

Having the "Could either be a network problem on the sending side or a problem writing to the storage on the server side." issue on a HTTP/1.1 server. Server 17.0.3, client 2.6.2 on macOS. The client thinks the server is HTTP/2 when I look at the SSL padlock info. Curl verifies that the server listens on HTTP/1.1.

[Milokita]

As the http2 will be disabled by default in next release (manual override is possible) I think this problem would be fixed by now. #1825

[feutl]

As the http2 will be disabled by default in next release (manual override is possible) I think this problem would be fixed by now. #1825

When is the new release available for download, I struggling syncing most of my pictures because of syncing EXIF data by Digikam. This triggered 30.000 pictures do be uploaded, some folders with umlaute and some panorama files bigger than 20MB.

Thanks

[Milokita]

As the http2 will be disabled by default in next release (manual override is possible) I think this problem would be fixed by now. #1825

When is the new release available for download, I struggling syncing most of my pictures because of syncing EXIF data by Digikam. This triggered 30.000 pictures do be uploaded, some folders with umlaute and some panorama files bigger than 20MB.

Thanks

As the fix have been merged into master branch, you can try the latest daily build https://download.nextcloud.com/desktop/daily/

[feutl]

@Milokita
Thanks, I looked there yesterday already, but because they are all 2.7 and the Milestone is 2.6.4 - I was not sure if this would work.
Thanks I am going to test this right now :)

[feutl]

I tested versions from today and yesterday on Ubuntu via AppImage, they do start but I am unable to get to the GUI to see any outcome or results.

[misch7]

Hey @feutl,

could you try a special 2.6 build I've just created? (Current state of the stable-2.6 branch, including the HTTP/2 fix.):

https://download.nextcloud.com/desktop/daily/Linux/Nextcloud-stable-2.6.20200303-test-1-x86_64.AppImage

[feutl]

Hey @feutl,

could you try a special 2.6 build I've just created? (Current state of the stable-2.6 branch, including the HTTP/2 fix.):

https://download.nextcloud.com/desktop/daily/Linux/Nextcloud-stable-2.6.20200303-test-1-x86_64.AppImage

Testing right now, the AppImage starts with GUI so I am able to start the sync. I report back after a while.

One thing is quite curious, the AppImage asked for big folders if they should be synced, even they were being set to be synced with the previous official 2.6 build. lets see how this works out.

[feutl]

It looks quite promising already, around 300GB (10 hours) to go and 95000 files.

Lots of files raise the "file has changed since discovery" notification in the activity log, I need to wait until all of this gets resolved, but the build is already more stable than the official 2.6 build.

[misch7]

Thanks for testing @feutl 👍

I'll build the 2.6.4 release then finally :)

[feutl]

The sync worked well the last hours but for some reason it created thousands of . (hidden) files like .IMG_7974-export.jpg.~6279f317 and filled up my disk. I have no clue why, could that be related to the client or just bad luck?
But those files are only stored locally, nothing on Nextcloud itself, also no files got deleted so far on the NC Server, if I can rely on the NC "trash" :)

[feutl]

Still have some connection closed errors, but much much much more stable.
What I synced last few hours was 100times more than with the official 2.6 in a few days.

[Milokita]

The sync worked well the last hours but for some reason it created thousands of . (hidden) files like .IMG_7974-export.jpg.~6279f317 and filled up my disk. I have no clue why, could that be related to the client or just bad luck?
But those files are only stored locally, nothing on Nextcloud itself, also no files got deleted so far on the NC Server, if I can rely on the NC "trash" :)

These files are the temp files created when syncing, it should be gone after the file is synced with server

[misch7]

The 2.6.4 release is out:
https://github.com/nextcloud/desktop/releases/tag/v2.6.4

@feutl Strange issue with the temp files; did you try to reboot the system just to be sure?

[feutl]

@Milokita
Could be possible they got created by the old Nextcloud Desktop Client which struggled downloading the files and lost the connection much too often.

@misch7
I am still seeing Connection closed (skipped due to earlier error) errors in my setup, but it also says retrying in about 50 minutes. I have the log output window open, anything specific I should look at to resolve the last pieces?

update: added some debug information
searched for one file which raises the above error and this is what I found

[_csync_detect_update 	file: path2017-08-21T16_05_32.jpg, instruction: INSTRUCTION_NEW <<=
[_csync_merge_algorithm_visitor 	INSTRUCTION_CONFLICT           server file: path/2017-08-21T16_05_32.jpg
[OCC::SyncEngine::checkErrorBlacklisting 	blacklist entry for  "path/2017-08-21T16_05_32.jpg"  has expired!
[OCC::SyncEngine::deleteStaleDownloadInfos 	Deleting stale temporary file:  "local path/.2017-08-21T16_05_32.jpg.~6fcdc4a5"
[OCC::PropagateItemJob::scheduleSelfOrChild 	Starting INSTRUCTION_CONFLICT propagation of "path/2017-08-21T16_05_32.jpg" by OCC::PropagateDownloadFile(0x557aecd33b30)
[OCC::AccessManager::createRequest 	2 "" "https://cloud.feutl.com/remote.php/dav/files/user/path/2017-08-21T16_05_32.jpg" has X-Request-ID "a240f0fa-cd60-4d60-a988-fee0622d0903"
[OCC::AbstractNetworkJob::start 	OCC::GETFileJob created for "https://cloud.feutl.com" + "path/2017-08-21T16_05_32.jpg" "OCC::PropagateDownloadFile"
[OCC::SyncJournalDb::setErrorBlacklistEntry 	Setting blacklist entry for  "path/2017-08-21T16_05_32.jpg" 3 "Connection closed" 1583308327 625 1538211017 "4492465e42c59f923ad970c8d7f430d7" "" 0
[OCC::blacklistUpdate 	blacklisting  "path/2017-08-21T16_05_32.jpg"  for  625 , retry count  3
[OCC::PropagateItemJob::done 	Could not complete propagation of "path/2017-08-21T16_05_32.jpg" by OCC::PropagateDownloadFile(0x557aecd33b30) with status 10 and error: "Connection closed"
[OCC::ActivityWidget::slotItemCompleted 	Item  "path/2017-08-21T16_05_32.jpg"  retrieved resulted in  "Connection closed"
[OCC::ActivityWidget::slotItemCompleted 	Item  "path/2017-08-21T16_05_32.jpg"  retrieved resulted in error  "Connection closed"

[feutl]

I fixed most of the above errors manually by replacing the files reported with a version from my backup. still quite confusing why this happened, some files seemed broken, could have been generated by the errors with the old 2.6 release and the connection losses I had.

All seems good now on the Linux Client.
Windows Client updated as well, sync works great, some Connection closed errors which I try to fix after the rest was synced successfully.

[feutl]

Windows Client also synced 100GB without having problems. Thanks for finding the issue and providing a fix 👍