Asynchronous operation

[borouhin]

Is your feature request related to a problem? Please describe.
The program is sometimes extremely slow due to synchronous processing of all tasks. For example:

• I add a new subdirectory with ~300.000 files, 200 Gb to local OneDrive directory;
• onedrive process hangs for ~2 hours (!) eating 100% of one CPU core, no log entries, no network activity (presumably, adding DB entries, calculating CRCs etc.)
• After that onedrive starts uploading files one by one, this process is much slower with a lot of small files than with a few large ones, network bandwidth allows much faster upload.

Describe the solution you'd like
The program should run CPU-consuming and network-dependent tasks in parallel whenever possible. For example, CRCs may be calculated by several threads/processes utilizing all CPU cores, whereas network upload may start as soon as at least some items are ready. Network operations themselves should be performed in several parallel threads/processes.
This should accelerate lengthy operations considerably.

Describe alternatives you've considered
Writing my own client from scratch :)

Additional context
I realize this may require an almost full rewrite, so not expecting immediate realization, just sharing my considerations.

[borouhin]

As an example of inefficient bandwidth usage with multiple small files - here is a graph of traffic from my server. At first, several large files (1-15 Gb each) were uploaded, and almost all bandwidth was utilized. Then small files began uploading and onedrive used only a small portion of available bandwidth:

(no other program used network significantly during the time of this graph)

If several network operations were performed in parallel, upload would complete much earlier.

[abraunegg]

As per #158 - much of this branch of the client is all about fixing the defects of the 'skilion' codebase. Yes this is a performance defect, but I have a feeling that re-writing the application to be more multi-threaded for many processes & operations is what is required here.

[modelmat]

The below is from the logs file, it is taking about this long to sync them. Most of them are 1KB.

2018-Sep-16 09:16:59.8788677 Downloading file Documents/Home/Programming/Python/Discord Bots/Farming/venv/tcl/tcl8.6/tzdata/Asia/Damascus ... done.
2018-Sep-16 09:17:00.3930902 Downloading file Documents/Home/Programming/Python/Discord Bots/Farming/venv/tcl/tcl8.6/tzdata/Asia/Chongqing ... done.
2018-Sep-16 09:17:01.032453 Downloading file Documents/Home/Programming/Python/Discord Bots/Farming/venv/tcl/tcl8.6/tzdata/Asia/Chita ... done.
2018-Sep-16 09:17:01.687136 Downloading file Documents/Home/Programming/Python/Discord Bots/Farming/venv/tcl/tcl8.6/tzdata/Asia/Choibalsan ... done.

[abraunegg]

onedrive process hangs for ~2 hours (!) eating 100% of one CPU core, no log entries, no network activity (presumably, adding DB entries, calculating CRCs etc.)

Most likely this is solved via #419

Other aspects are still in plan to resolve

[borouhin]

Most likely this is solved via #419

This is an old comment indeed, many things have improved since then. CPU usage in particular is much less now and no such problems as before.

[davidbarratt]

@borouhin could you try out https://github.com/onedrivejs/onedrive ?

[borouhin]

@borouhin could you try out https://github.com/onedrivejs/onedrive ?

@davidbarratt, thanks for the link, looks promising! I gave up attempts to achieve stable 2-way sync and switched to using rclone for 1-way sync only. Had to reorganize my files, but now it works like a charm without any manual interference. I'll give JS client a try when I have time for some experiments.

[abraunegg]

@borouhin
Stable 2 way sync is certainly possible with this client - lots of improvements have been made.

Also - I am closing this issue, as it is better referenced by #232

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.