Crash on Status 429 (Too Many Requests)

[tdaff]

After syncing about 20 GB in many files over the course of an hour, I started getting limited. Apparently the status comes with a Retry-After header, but maybe it is worth slowing down requests for a while if you get a 429 in the middle of a complete re-sync?

Unfortunately I don't think the limits are made publicly available.

sync.SyncException@src/sync.d(140): HTTP request returned status code 429 (Too Many Requests)
----------------
??:? void sync.SyncEngine.applyDifferences() [0x4e6e2b]
??:? void main.performSync(sync.SyncEngine) [0x4d6ac3]
??:? _Dmain [0x4d62ee]
??:? _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv [0x83a59c57]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x83a59b83]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll() [0x83a59bfc]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x83a59b83]
??:? _d_run_main [0x83a59ae7]
??:? main [0x4d9375]
??:? __libc_start_main [0x829091df]
onedrive.OneDriveException@src/onedrive.d(387): HTTP request returned status code 429 (Too Many Requests)
----------------
??:? void onedrive.OneDriveApi.download(const(char)[], immutable(char)[]) [0x4de357]
??:? void onedrive.OneDriveApi.downloadById(const(char)[], immutable(char)[]) [0x4dc12f]
??:? void sync.SyncEngine.applyNewItem(itemdb.Item, immutable(char)[]) [0x4e822e]
??:? void sync.SyncEngine.applyDifference(std.json.JSONValue) [0x4e7c4a]
??:? void sync.SyncEngine.applyDifferences() [0x4e6930]
??:? void main.performSync(sync.SyncEngine) [0x4d6ac3]
??:? _Dmain [0x4d62ee]
??:? _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv [0x83a59c57]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x83a59b83]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll() [0x83a59bfc]
??:? void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x83a59b83]
??:? _d_run_main [0x83a59ae7]
??:? main [0x4d9375]
??:? __libc_start_main [0x829091df]

[ggdx]

Is there a way of self-throttling?

[nikolaslada]

I got same results. After syncing ~20GB the Skilion/OneDrive client interrupted with status code 429. One hour later I ran it again. OneDrive let me sync ~1GB and then the client interrupted again. OneDrive seems limited. So the client could react to this status code. Fortunately a cron can be solution.

[kolomparrudi]

Hi,

I have the same issue. I made a workaround by adding retry function:

diff --git a/src/onedrive.d b/src/onedrive.d
index 032aa03..9a99304 100644
--- a/src/onedrive.d
+++ b/src/onedrive.d
@@ -1,6 +1,6 @@
import std.net.curl: CurlException, HTTP;
import std.datetime, std.exception, std.file, std.json, std.path;
-import std.stdio, std.string, std.uni, std.uri;
+import std.stdio, std.string, std.uni, std.uri, core.time, core.thread;
import config;
static import log;

@@ -132,7 +132,15 @@ final class OneDriveApi
mkdirRecurse(dirName(saveToPath));
}
const(char)[] url = itemByIdUrl ~ id ~ "/content?AVOverride=1";

•           download(url, saveToPath);
  

•           int i = 0;
  

•           do {
  

•               download(url, saveToPath);
  

•               if (http.statusLine.code / 100 != 2) {
  

•                   log.log("http error retrying\n");
  

•                   Thread.sleep(dur!"seconds"(i));
  

•                   i++;
  

•               }
  

•           } while (http.statusLine.code / 100 != 2);
    }
  
    // https://dev.onedrive.com/items/upload_put.htm
  

@@ -300,7 +308,7 @@ final class OneDriveApi
return data.length;
};
http.perform();

•           checkHttpCode();
  

•           //checkHttpCode();
    }
  
    private auto patch(T)(const(char)[] url, const(T)[] patchData)
  

[thomasfedb]

@tdaff can you replicate this with the latest release?

[tdaff]

@thomasfedb sorry, I'm not able to test it anymore, gone back to Windows! Maybe someone else can test it, but it's not an issue for me anymore.

[thomasfedb]

@tdaff might be best to close this issue in that case. Cheers.

[abraunegg]

@tdaff

Can you close this case if no longer applicable ?

[abraunegg]

@tdaff
Can you please provide an update to this issue. If no longer valid, please can you close this case.

[abraunegg]

Considering this issue closed due to no response from original submitter.

If you have further issues please or come back to this, open a new case here: https://github.com/abraunegg/onedrive

[jucardi]

I'm getting this issue myself. I believe it is related to the amount of HTTP requests sent to the server rather than the about GBs synced.

It looks like the client app is sending a request per file, so if the are multiple small files, the server bounces with a Too Many Requests response. Looks like the amount of requests should be throttled, although I wonder if there is a way to bulk get files to avoid sending soo many requests to the server.

[grfx1985]

I am getting the same issue, half way through retrieving all of my data from OD, 56 gb to go but it seems to constantly spit this now, even after waiting for 12 hours it only just downloaded 3 gb more before spitting it again. After that when resuming it manages to get some files 50 maybe sometimes less, sometimes not even 2 :), and raises again. At this point I kind of ran out of ideas. Tried to log out and log in again but seems not able to get passed this error. Will try on a different pc as soon i sync some other data and will be able to change the conf matching the failing setup on initial machine.

[wn2000]

Just started using this and I'm getting this issue as well.
Synced maybe 200GB of data and started to get this. A restart synced another 3GB data and failed with same error again.

[abraunegg]

@wn2000
The 'skilion' client does not receive any support (reference) and contains issues / defects that put you at risk of data loss. Despite a new version release in Oct 2020, these defects remain. If you wish to use a client which is fully supported please refer to: https://github.com/abraunegg/onedrive

Regarding your specific issue, 429 responses are correctly handled in the above supported codebase.

Major differences / Enhancements:

• Existing bugs with this client fixed
• File upload / download validation to ensure data integrity
• Support for SharePoint / Office 365 Shared Libraries
• Support for OneDrive Business Shared Folders
• Support for Docker
• Desktop notifications via libnotify
• Dry-run capability to test configuration changes
• Prevent major OneDrive accidental data deletion after configuration change
• Support National Azure Cloud instances
• Supports single & multi-tenanted applications
• Supports rate limiting of traffic

[wn2000]

@abraunegg Thank you! Will give it a try.