Replies: 19 comments 58 replies
-
|
I tried to do a full resync of my live files (150GB,200k files, 30k dirs) in monitor mode, but it keeps crashing after about 20 minutes. I did not loose internet connectivity. It seems to only happen when I use Bellow is |
Beta Was this translation helpful? Give feedback.
-
|
Interesting ... Could you roll back to 'alpha-4' and see if you get the problem? |
Beta Was this translation helpful? Give feedback.
-
|
Here are two more crashes. First after 17 minutes in monitor mode, second after 10minutes. Note that I never lost internet connectivity. Here is example output from Perhaps some kind of a rate limit at OneDrive servers? If so, the client will probably need improved error handling. It would also be great if the debug output showed which thread generated it. Crash 2:Crash 3: (without --resync)
|
Beta Was this translation helpful? Give feedback.
-
Yes, I actually encountered the issue twice with alpha-4 in the past, but I assumed I lost connectivity at the time. It worked fine when I restarted the client, and I was getting the final I have now re-tested with Perhaps I triggered some kind of a rate limit at OneDrive servers after I started a full resync today? I will re-try tomorrow and let you know if it cleared. |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena I am no suggesting there your crashes are a result of this oversight, but if you could rebuild to |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena Unfortunately I didnt have debugging fully enabled ... re-testing now with this enabled. |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena This is very IPv6 specific .... and, v2.5.x will use IPv6 and IPv4 for resolution, unless told otherwise. I am going to 'lock' my IP protocol version to IPv4 via 'ip_protocol_version' - https://github.com/abraunegg/onedrive/blob/onedrive-v2.5.0-alpha-5/docs/application-config-options.md#ip_protocol_version |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena This has been running for >45 mins no crash or issue, using an automated script to generate data / delete data on repeat. Will let this process run overnight and see what happens. If no crash overnight, will remove the 'ip_protocol_version' and re-test. |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure I followed what you said, but I can't get it to reuse the connection. I keep seeing many new TCP requests being generated. I'm also not seeing the line I tested on to before compiling it again, but curl still keeps closing the connections after every request. The curl debug also shows that HTTP1.1 is being used instead of HTTP2. Not sure if that's intended either. Not reusing the connection seems like extreme waste of time and bandwidth as there needs to be new TCP and TLS handshake for every single API request. In my case it also makes a new DNS request before every API request because I have DNS caching disabled. Based on my packet capture about 7KB of data (in 13 packets) and 150ms of time is "wasted" with every request, which ads up when done thousands of times per day. |
Beta Was this translation helpful? Give feedback.
-
Correct - In v2.4.x the client was explicitly reusing TCP connections. In v2.5.x this reuse is explicitly disabled.
The way that multi-threading has been implemented, a new instance of libcurl is being used for each thread, and then the instance is being destroyed post use - however this can lead to open files and sockets hanging around which never get closed (thus free) - I saw this in the development of multi-threading quite heavily which impacted even aspects such as reading the local CA Certificate (which was removed in 'alpha-4' - see: aa2ecdc) One of the aspects that I have on the whiteboard however is to change how the 16 thread operate. The design on the whitebaord is so that rather 16 threads being created / destroyed all the time - they get created then poll & pop the next item off the array and process it. The idea here is that the thread stays alive whilst there are still upload|download actions to take. This way, once the thread is initialised, TCP connections can be reused in a more efficient manner and potentially this libcurl option can be switched to '0' so that everything is correctly reused as required. However, there are other instances of libcurl that are started as well - and any open sockets that are then left open once the instance is closed - this may cause socket exhaustion on long running activities - and sharing a 'common' libcurl' instance for common tasks causes other issues which would negate this problem.
HTTP2 will be used when the connection allows it. For some connections HTTP1.1 will be used - it is entirely dependent on the request and the server as to if HTTP1.1 or HTTP2 is being used and the server will dictate what to use.
Correct, however there is no easy way to get around this given that each thread is a separate libcurl instance, and is destroyed post activity. DNS Caching in the OS is generally what occurs here - that you have this explicitly disabled is beyond my control. |
Beta Was this translation helpful? Give feedback.
-
Please recheck #2526 to confirm that this flag is actually not intended to address socket exhaustion issues but rather may lead to exhaustion. |
Beta Was this translation helpful? Give feedback.
-
|
In my opinion the only correct solution is to keep When these two approaches are combined, you should not see more than 16 active TCP sessions with 16 curl threads. I'm not sure what other issues you faced previously that caused you to run out of sockets, but perhaps we can find a better solution to it. |
Beta Was this translation helpful? Give feedback.
-
|
On a slightly separate note, I think it could be helpful if users could configure the maximum number curl threads. Especially users who have multiple onedrive instances running with different accounts. |
Beta Was this translation helpful? Give feedback.
-
|
I was curious why the sync was taking so long and then I noticed that the client is syncing even all my skip_dirs, oops! I than figured that the client started sync with all default values due to a deprecated key This config loading bug happens only when If SCENARIO 1: Config does not load when a deprecated option is at the beginning of the config file:Scenario 2: Config loads OK when a deprecated option is at the end of the config file: |
Beta Was this translation helpful? Give feedback.
-
|
Interesting .. will have to look into that |
Beta Was this translation helpful? Give feedback.
-
This should now be fixed in config file
Display Config |
Beta Was this translation helpful? Give feedback.
-
|
Fix confirmed in |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena |
Beta Was this translation helpful? Give feedback.
-
Please look at This is now a configurable option: https://github.com/abraunegg/onedrive/blob/onedrive-v2.5.0-alpha-5/docs/application-config-options.md#threads Default is '8' , max is '16' I have also made some libcurl adjustments, please can you test this against your much larger data set please and advise. |
Beta Was this translation helpful? Give feedback.
-
|
Great, thanks. I was able to complete my sync now, when I limited the client to just one thread. But it still crashed with the default of 8 threads as I hit the TCP session rate-limit at Microsoft servers. |
Beta Was this translation helpful? Give feedback.
-
|
@abraunegg Build version Opened fd Logs |
Beta Was this translation helpful? Give feedback.
-
|
After some investigation, I believe, in my case, it is likely the cause that the OneDriveApi instance is not being properly shutdown and recycled if the path already exists online. I'll suggest checking if all instances in other functions are being shutdown and recycled for all scenarios. Lines 4312 to 4322 in e92947e |
Beta Was this translation helpful? Give feedback.
-
|
@JC-comp, I'm glad you can also reproduce this issue. Could you please confirm if you are seeing the same symptoms as I do? I'm trying to find out if the crash happens after connections are rate-limited upstream (modem/isp/Microsoft, etc.).
Here is a simple script that I kept re-running every second or so just before and after the crash. It just lists all onedrive sessions and socket file descriptors. Their number should match the number of onedrive threads, which it does in my case. Just before the crash, when there are no issues while onedrive client is configured to use 16 threads: Just before the crash, when TCP connections start timing out: When onedrive crashes, all sessions and socket FDs are removed: After crash, when the process does not exist anymore |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena When uploading constant random data: I cannot reproduce any crash or anything like you are experiencing. |
Beta Was this translation helpful? Give feedback.
-
|
Additionally, the largest values I ever get to are (when uploading random data): |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena |
Beta Was this translation helpful? Give feedback.
-
|
Likewise when running this on my development system - no freeze or issue, or even rate limit drop on network .. |
Beta Was this translation helpful? Give feedback.
-
I strongly suggest that you fix your environment first and foremost. That you are increasing all your network connectivity by disabling DNS caching is really a bad item to do, but also if you are using your ISP's provided modem and it fails after 2000 TCP connections - that is something that you really should be fixing. Running your own DNS Server at home is a really easy item to do (gives you better response times than even most ISP or home routers) and caches your responses. Your ISP modem however - you need to get this updated and/or replaced. |
Beta Was this translation helpful? Give feedback.
-
DNS caching is disabled on my laptop because I have caching enabled on my upstream firewall for multiple zones. DNS is not an issue. The point I was trying to make is that curl needs to try to resolve the hostname tens of thousands of times per day because the thread is destroyed after every API request. The server certificate also needs to be downloaded tens of thousands of times per day for the same reason. There is not much users can do when their home ISP, employer, school, hotel,etc. has DoS protection or any kind of rate-limiting enabled on network. With the current implementation, onedrive client generates more IP sessions than any torrent client with default settings. It's just unnecessary and slow. My suggestion was to keep the onedrive/curl threads active and reuse them. This way, there would be just 16 TCP sessions (with 16 configured threads) for the entire onedrive re-sync run. But if you don't want to implement it, it's fine. Also, the onedrive client should not just crash when a curl connections times out for any reason. For instance, when the sync is running on a laptop and it disconnects from WiFi for a few seconds, the client will freeze and then crash as well. |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena Example, with the updated watcher script posted above: The highest ever number of open sockets to Microsoft OneDrive is 16 ... which matches the number of threads I am running. You are getting something different ... why .. I do not know your environment as to what you are using or what limitations you have either self imposed and/or upstream imposed. I know in my setup:
|
Beta Was this translation helpful? Give feedback.
-
In most functions the API is called and shutdown - and in that particular function it is (at the end). The instance 'createDirectoryOnlineOneDriveApiInstance' is reused throughout the entire function. I have made a 'tweak' to this function only - please can you re-run any testing you did when you came to the conclusion that this function was causing issues. If you rebuild your 'alpha-5' client version, the fix is in If this fix solves the issue you were seeing, then, all functions where an API is being created will need a similar fix. I think however a wider solution is looking how API instances are created & reused is what is needed - and combined with the refactor API piece as well. |
Beta Was this translation helpful? Give feedback.
-
|
No errors were found in this fixed version. Agreed that the next refactoring should carefully consider the lifetime, error handling and logging of the API instance altogether. |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena Additionally, using the modified 'watcher' script, these were the high watermarks. I also fixed the script as it was counting dud entries for sockets and file descriptors as well: I do not think that there is a TCP socket issue with the application - I think your issue is 100% environmental to you. Given the reproducibility of your issue, can you please start detailing more of your environment so that I can built potentially something that is reflective and then try and duplicate the very odd situation you are seeing. You may be encountering a router bug, a firewall bug, a Linux bug .... but essentially you potentially have a network bug in your setup somewhere is what I suspect.
This is not the case as is shown above - only 16 TCP connections at maximum is used
This is what is happening however .. combined with your request to make the threads user configurable.
When timeouts are reached, the client will 'retry' and should not crash. The timeout logic attempts to retry also with flagging curl to use a 'fresh connection'. When these 'curl' timeouts have been seen, 100% this has been when 'libcurl' is deciding for whatever reason to use IPv6 resolution rather than IPv4 responses. This is why I have also been advising you to use Most importantly - I cant fix something if I cant reproduce it to understand what is going on. Please can you:
|
Beta Was this translation helpful? Give feedback.
-
Regarding the crashes reported above, I think the likely cause is missing checks to verify if the JSON instance returned by OneDrive is a valid JSON object. Perhaps in those cases, OneDrive returns an empty response or, somehow, the Curl library or the program drops them, resulting in a null JSON object. If this is the case, I believe there is a need to reevaluate when the object is considered credible and how to better handle those invalid objects, as I found many places where the object is not validated before accessing them, similar to #2601. Below is one of the examples where the returned object is not validated: Lines 4833 to 4841 in e2f1b68 Lines 969 to 971 in e2f1b68 |
Beta Was this translation helpful? Give feedback.
-
There are no issues with my system. I use the same laptop for data processing in one of my projects where i regularly write millions of measurements into a time series database over HTTP. I do have an issue that my home ISP as it has a rate limit to block malicious users/apps, which is nothing I can do about. No other app I've ever used has ever triggered this ISP rate limit.
Please note that there is a huge difference between the number of concurrent sessions and session rate. Every time you refresh the I did a test on different network with You can use wireshark or tcpdump to see all of the actual traffic, but you can also use the command Here is an example of onedrive alpha-5 with 16 threads creating/closing 60 TCP sessions per second (each with unique source port number). $ unbuffer ss -t -E -n -a | while read line; do echo "[$(date)] $line";done
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:51814 20.190.190.102:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37664 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60434 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37668 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60448 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:52712 20.190.190.96:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:48268 20.190.190.101:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37656 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60482 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60460 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:48244 20.190.190.101:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60468 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:51830 20.190.190.102:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:48284 20.190.190.101:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37684 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:51834 20.190.190.102:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:56590 20.190.190.99:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:48290 20.190.190.101:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:56574 20.190.190.99:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:48310 20.190.190.101:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:52724 20.190.190.96:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:56596 20.190.190.99:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60500 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60488 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37698 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:56592 20.190.190.99:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60498 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:48296 20.190.190.101:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60504 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37714 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37722 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60526 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37706 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60516 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37750 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:56606 20.190.190.99:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37728 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37724 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:52756 20.190.190.96:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:48324 20.190.190.101:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37744 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:52744 20.190.190.96:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37730 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:52738 20.190.190.96:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60534 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:52786 20.190.190.96:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:52792 20.190.190.96:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37766 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:52770 20.190.190.96:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37774 20.190.190.100:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60556 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:56618 20.190.190.99:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:56634 20.190.190.99:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:51848 20.190.190.102:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:56648 20.190.190.99:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60542 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:60562 20.190.190.103:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:56660 20.190.190.99:443
[Wed Jan 24 02:26:02 PM CET 2024] UNCONN 0 0 172.26.52.86:37802 20.190.190.100:443With With Example: |
Beta Was this translation helpful? Give feedback.
-
Please can you re-test your observations regarding curl session reuse. When I enable HTTPS debugging - the connections are being reused: |
Beta Was this translation helpful? Give feedback.
-
With 16 threads, full sync, nothing transferred because everything is already in sync (as per your test): This is a data set with ~50K files and folders ....... Only 9 TCP connections for the entire sync process ... so this is again something that cannot be replicated using my environment using enterprise grade equipment. I would start looking at your environment and setup as something is not right. |
Beta Was this translation helpful? Give feedback.
-
|
@abraunegg
|
Beta Was this translation helpful? Give feedback.
-
One is Onedrive personal account and another is Microsoft 365 business account |
Beta Was this translation helpful? Give feedback.
-
|
@JC-comp |
Beta Was this translation helpful? Give feedback.
-
One is |
Beta Was this translation helpful? Give feedback.
-
|
Very odd as I cannot reproduce at all ...... |
Beta Was this translation helpful? Give feedback.
-
|
Please raise this as an API bug here: https://github.com/OneDrive/onedrive-api-docs Add all evidence including |
Beta Was this translation helpful? Give feedback.
-
No .. HTML characters should not be in filenames or folder names anywhere - to ensure that the names used are as 'safe' as possible, This is why this warning exists. |
Beta Was this translation helpful? Give feedback.
-
|
@abraunegg Lines 1967 to 1977 in 5a32a32 Lines 920 to 926 in 5a32a32 |
Beta Was this translation helpful? Give feedback.
-
|
@JC-comp When attempting to use, for instance, something like this: I just was not able to able to get this working in the early prototypes ... either using free standing functions with appropriate call backs or anything else ... If you want to re-work / attempt to re-work this - no issue there ... but when using the 'batch' method, to split the work into groups that equal the thread count .. this worked correctly to create 16 download|upload actions running concurrently. |
Beta Was this translation helpful? Give feedback.
-
|
I think taskPool will handle all these stuff (batching, fetching jobs, ) for us, something like this, and it will adhere to what we have set for defaultPoolThreads. So actually we don't need to manually batch them before sending them to |
Beta Was this translation helpful? Give feedback.
-
|
Certainly worth trying .... |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena , @JC-comp Does this PR put together by @JC-comp make any difference in your environment ? |
Beta Was this translation helpful? Give feedback.
-
|
@abraunegg @JC-comp Issue 1 - client crashesThe client is still prone to crashing when it's recovering from connectivity issue (after curlException). It happened once when I had relatively low WiFi signal. I then managed to reproduce it twice by manually disconnecting and connecting to Wi-Fi. Though it seems to be somewhat random. Sometimes I just disconnect twice and it crashes, sometimes I have to reconnect 30x to cause the crash, sometimes I keep re-connecting like a monkey and it resumes just fine... Not sure if it could be related to multiple threads trying to read/modify the same item in the taskPool at the same time. Issue 2 - unnecessary API requestsRegarding the 429 response codes, the client currently asks for quota information with every legitimate request. So, with 16 threads enabled, I can see 16 GET requests for files followed by 16 GET requests for quotas. I'd imagine there is no need to ask for quota this frequently and removing these requests would technically double the speed of the resync process as there would be less requests processed by the onedrive servers. Log example: $ unbuffer ./onedrive --confdir='/home/bob/.config/onedrive/accounts/bpozdena@company.com' --monitor -vvv --resync --resync-auth | while read line; do echo "[$(date)] $line";done 2>&1 | tee reuse9.log
227393: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
227535: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230120_152602.png: HTTP/1.1
227576: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230308_143250.png: HTTP/1.1
227582: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230106_102822.png: HTTP/1.1
227593: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230124_165209.png: HTTP/1.1
227599: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshots%2Fksnip_20220330-152748.png: HTTP/1.1
227607: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230206_162209.png: HTTP/1.1
227614: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230206_162553.png: HTTP/1.1
227620: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230301_145325.png: HTTP/1.1
227626: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230308_083438.png: HTTP/1.1
227632: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230212_210024.png: HTTP/1.1
227640: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2Ficons8-settings-50.png: HTTP/1.1
227646: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FSniffers.zip: HTTP/1.1
227656: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230212_210022.png: HTTP/1.1
227662: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230210_123426-1.png: HTTP/1.1
227668: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230120_140609.png: HTTP/1.1
227689: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230118_164110.png: HTTP/1.1
228320: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228358: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/root:/.%2Fjunk%2FScreenshot_20230118_164110.png: HTTP/1.1
228395: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228404: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228413: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228425: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228431: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228443: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228449: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228458: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228476: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228482: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228488: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228494: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228503: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228515: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228521: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1
228530: [Thu Jan 25 01:13:31 PM CET 2024] > GET /v1.0/drives/b!XXXXXX/?select=quota HTTP/1.1It looks like sync.d
----------
// Upload changed local files to OneDrive in parallel
void uploadChangedLocalFileToOneDrive(string[3][] array) {
foreach (i, localItemDetails; taskPool.parallel(array)) {
[...]
// Query the available space online
// This will update appConfig.quotaAvailable & appConfig.quotaRestricted values
remainingFreeSpace = getRemainingFreeSpace(dbItem.driveId);
[...]
// Upload a new file to OneDrive
void uploadNewFile(string fileToUpload) {
[...]
if (thisFileSize <= maxUploadFileSize) {
// Is there enough free space on OneDrive when we started this thread, to upload the file to OneDrive?
remainingFreeSpaceOnline = getRemainingFreeSpace(parentItem.driveId); |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena , @JC-comp |
Beta Was this translation helpful? Give feedback.
-
|
@bpozdena , @JC-comp Please validate |
Beta Was this translation helpful? Give feedback.
-
|
@abraunegg |
Beta Was this translation helpful? Give feedback.
-
|
@JC-comp Please raise a new Issue with correct verbose & debug log evidence please. |
Beta Was this translation helpful? Give feedback.
-
|
Currently, the exit handler in the design only supports monitor mode to clean up certain resources. Here are some related design issues
Regarding signal safety, performing heavy operations, including logging, in a signal handler is ill-advised. Considering points 1 and 2, and #2610, do you believe a rewrite that only sets the flag in the sync engine's signal handler, allowing the sync engine to gracefully shut down and return control to the main function to trigger the cleanup process, is feasible? |
Beta Was this translation helpful? Give feedback.
-
As part of the ongoing development of v2.5.0, client version v2.5.0-alpha-5 is now available for testing.
This version is going to be used to again validate 90+ Use Cases for Personal, Business and SharePoint Libraries to validate functionality as per intention|expectation.
Summary of Changes for 'alpha-5'
Feature Requests
Bug Fixes
Other Changes | Fixes
Documentation
Whats still missing?
Can I use this version with my normal account?
You can use your normal account if you wish (this is your choice), however this is 100% alpha quality software at the moment. If you experience data loss or something else with your real data - whilst not good, please take every effort to not use this code with your real data.
There will be functionality differences, there will be bugs or things that are not working right - the Use Case Testing done with all 'alpha' versions should have shaken most of this out, but you never know.
My ask is that you create a test report - what works, what does not work, what could be potentially changed | improved - basically I am soliciting some feedback on where this total re-write is heading as it matures to a v2.5.0 release.
I envisage that 'alpha-5', once it has gone through all my Use Cases for each account, will be pushed as a 'beta-1' pre-release
How can you get v2.5.0-alpha-5 ?
You need to build this client from source. Follow the high level directions here to get the development requirements installed first: https://github.com/abraunegg/onedrive/blob/master/docs/INSTALL.md#building-from-source---high-level-requirements
Next clone and checkout the PR:
To run the PR, you need to run the client from the PR build directory:
When running the PR, your version should be as per below or greater:
The important point is
v2.5.0-alpha-5. Thev2.4.25-XX-abcdefgwill change as new commits are put into the PR branch, whilst things are added/fixed/updated.Documentation for 'alpha-5'
The existing GitHub master documentation does not match what v2.5.0 does or supports at present. Please refer to the updated docs here: https://github.com/abraunegg/onedrive/tree/onedrive-v2.5.0-alpha-5/docs
I have no test files or test data - what can I use?
In developing this application I use a script as per below to generate random data:
Update
DESTDIRwith the right directory and adjustFILESandFOLDERSas desired. The above settings will create 10 folders, with each folder having 10 files. with each file size between 6KB and 125KB.To increase (or decrease) the random file size, change
BLOCKSIZE, generally use ^2 values: 1,2,4,8,16,32,64,128,256 and so on.At 128 you will get files from 100KB -> ~3.8MB, at 256 most files will be between 500KB --> 7.5MB
Beta Was this translation helpful? Give feedback.
All reactions