Bulk upload issues

[joshtrichards]

(WIP: will be updated as needed)

Summary: Indications are that there are one or more scenarios that trigger problems with bulk uploads, creating serious performance problems (e.g. 100% CPU usage, non-responsive). Many find success through workarounds like disabling bulk uploads or adjusting upload limits. Some reporters have noted that #5680 resolved their situation, while others it has not. There may either be additional triggers or simply multiple unrelated root causes.

Note: Many of these are likely duplicates (which is part of what needs to get sorted out)

Issues:

• [Bug]: after upgrade to >=3.8.2 the client consumes 100% CPU and shows "synchronising" when opened from the tray #5726
  • OP reports fixed (by Fix unrecoverable freezing when PutMultiFileJob is used with upload rate limits enabled #5680): [Bug]: after upgrade to >=3.8.2 the client consumes 100% CPU and shows "synchronising" when opened from the tray #5726 (comment))
    • Should likely be closed as a result; new reporters clearly have different circumstances so should create new Issue(s) as needed
• [Bug]: Desktop Client freezes in a 100% CPU load loop after server upgrade to 25 #5094
  • OP reports fixed (by Fix unrecoverable freezing when PutMultiFileJob is used with upload rate limits enabled #5680): [Bug]: Desktop Client freezes in a 100% CPU load loop after server upgrade to 25 #5094 (comment)
    • OP reports still fixed: [Bug]: Desktop Client freezes in a 100% CPU load loop after server upgrade to 25 #5094 (comment)
    • Should stay closed as a result; new reporters clearly have different circumstances so should create new Issue(s) as needed
• [Bug]: Windows Desktop Client UI completely unresponsive during preparation for sync #5031
• [Bug]: Unknown error and network error 99 when syncing .vscode extensions #5349
• Windows, Linux, and macOS clients freezing when bulk uploading #4106
  • OP reports fixed (by Fix unrecoverable freezing when PutMultiFileJob is used with upload rate limits enabled #5680): Windows, Linux, and macOS clients freezing when bulk uploading #4106 (comment)
• [Bug]: Windows Client Error on Every File Creation #5888
• [Bug]: Computed md5 hash is incorrect #6511
• Bulk Upload Issues server#29702
• [Bug]: InvalidArgumentException: X-OC-MTime header must be a valid positive integer (regression?) & The X-File-MD5 header must not be null.  #6889
• [Bug]: Please re-open issue 5094, bulk upload still triggers 100%CPU client usage and hangs  #6941

PRs:

• Fix unrecoverable freezing when PutMultiFileJob is used with upload rate limits enabled #5680
• https://github.com/nextcloud/desktop/pulls?q=bulk+is%3Apr+is%3Aclosed+

[sapphirepro]

Yeah, when uploading bulk of files, it wastes cpu and after a tiny upload goes rescan etc in a loop. With fiber network uploading like 1 GB of photos can take even an hour to complete. So solution still needed

[mgallien]

thanks for the help raising the importance of this
is there known reproducible scenario that are still failing with recent server and client releases ?
no big hopes of a quick fix if I first need to investigate and spend hours trying to analyze possible ways to trigger the bugs

[sapphirepro]

thanks for the help raising the importance of this is there known reproducible scenario that are still failing with recent server and client releases ? no big hopes of a quick fix if I first need to investigate and spend hours trying to analyze possible ways to trigger the bugs

Ok. Giving information. Latest NextCloud stable release. Latest Linux desktop client. Now situation is that to client side dropped 700 mb of files (approx 1-1.5 Mb per file, photos). So key scenario to test reasoanble size like 700 MB or above, while they should be small and a lot of them, like 100 files or above. Sync will be like dropping fast maybe 30-40 mb, then huuuge wait like few minutes, then again short time upload of 30-40 mb and so on in the loop till all uploaded.

Expected behavious is if dropped 700mb of many files to upload, then upload them all non stop and only then resync. What I suspect as a reason of this performance bug, is that it starts upload, uploads tiny files like 40 mb in total and then server reports changes (those same files), upload gets held, resync and then proceed. It should throw all files in a bunch and only then resync something. Also a tray icon and clicking on it while such sync doesn't often show floating window of nextcloud desktop client, by which I assume that there is process violation as well, that sync or something being done in main process and blocking UI, while it should be a separate thread.

I tried the best I could to explain. If any more info needed, do tell, I will try to assist.

[Bockeman]

My Windows10 Nextcloud client is 3.13.2 (Stable).
I ugraded my (Fedora 40)server to Nextcloud Hub 8 (30.0.0 RC1) (Beta and soon to be RC2).
Consequently, I got loads of errors and warnings which I have largely eliminated apart from "Error no app in context Computed md5 hash is incorrect" which appears to be related to this topic.

I have "'bulkupload.enabled' => false," in my server nextcloud.cfg (though I am not convinced this is effective).

My Windows10 Nextcloud client settings shows 20TB in use, and this includes Sync Connections for AppData/Roaming. The important point being that files are changing constantly at the whim of Windows.

Bulk transfer seems a good option IMHO, but there is a high likelyhood of file changes before the transfer is complete. Hence I am expecting all files uploaded to the server to retain the file modified date, as captured in the bulk, and a subsequent transfer to handle files changed since the previous bulk capture.

I am getting "Error no app in context Computed md5 hash is incorrect" every 10 minutes or so, therefore I think I have a suitable environment for testing possible fixes/patches.

Please advise if there is any patch you wish me to apply to verify possible solutions.

[keithkhl]

another frustrated user here.
thought it is due to misconfiguration and wasted days, if not weeks.

One more catch for bulk uploading files.
When files are inside the nested folders, it does not copy any files. It only copies folder structure.

it is not even browser or OS issue.
For the last a month or so, I've tried bare metal installation to AIO, and even on windows.

OneDrive used to be notoriously slow, but it got way faster now. Yet, many people still stick to Dropbox's paid option because of the performance.
Gotta tell you that what I see so far for the last a month is beyond expectation.
It's not like my server is not equipped. You don't usually assign a 64-core + 256GB RAM w/ 10TB NVMe and 10Gbps just for an ERP type service.
Give us at least a reasonable option to expand bandwidth and max use of server resources.

[sol8712]

#7615 Another one to add to the random bulk uploads issues.

[timinator2020]

Having the same issue - added about 3000 files, but all totaling only 32MB. It uploads 100 files, then pauses for 30-60 seconds (no network activity). After 30-60 seconds, it instantly copies another 100 files and pauses again.

I have tried changing the minChunkSize/maxChunkSize values in the desktop config.ini (even went so far as to set the minChunkSize value to 1). I disabled the redis container to rule that out (this is mostly just for me, so file locking isn't a big concern). I added some timeout settings to the fastcgi nginx proxy. I also tried a few other things, but I can't remember what they were at the moment. They didn't help either.

The only thing that seems to help is adding 'bulkupload.enabled' => false to config.php (which completely disables the bulk-upload functionality). Transfer rates are slower, but at least they don't freeze (so the total transfer time is less). I'd really like to see the issue fixed properly, since it would definitely help with use cases like this (thousands of tiny files).