Update backoff strategy #2026
Update backoff strategy #2026
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2026 +/- ##
==========================================
+ Coverage 82.40% 82.41% +0.01%
==========================================
Files 66 66
Lines 8222 8222
==========================================
+ Hits 6775 6776 +1
+ Misses 1447 1446 -1 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
not sure about retrying on 500 (cc @XciD @coyotte508 WDYT?) but the other ones look good to me
|
We are already retrying on 500 when uploading to S3 as we already noticed transient HTTP 500 there. |
|
yes I meant |
|
Wauplin
force-pushed
the
http-backoff-retry-on-any-5xx
branch
from
5749ce1 to
283d9b6
Compare
Wauplin
changed the title
|
I've updated the PR (only changing upload retry). |
Wauplin
changed the title
Instead of only 504 by default. Those might be transient errors happening when downloading or uploading files. Let's retry when that happens.Related to huggingface/datasets#6577 (comment) (cc @sanchit-gandhi) which most likely happened due to the outage on Saturday (so a retry wouldn't have helped) but I still reviewed the process as we were currently having different retry mechanism spread in the codebase. (+it's not the first time I've seen this kind of feedback so let's "fix it" once and for all).EDIT: Retry on any 5xx on upload. Retry on any 5xx on download except 500. (see below)