S3 store fails without retrying

[criccomini]

Describe the bug
When doing a high amount of requests on Tigris (S3 clone), I see this sometimes:

called `Result::unwrap()` on an `Err` value: ObjectStoreError(Generic { store: "S3", source: RetryError(RetryErrorImpl { method: GET, uri: Some(https://t3.storage.dev/***/slatedb-bencher_20_128/compacted/01JZ9FMS2RJ801EXBKYCX43J95.sst), retries: 0, max_retries: 10, elapsed: 18.220024ms, retry_timeout: 180s, inner: Status { status: 429, body: Some("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>ErrTooManyRequest</Code><Message>Too many simultaneous request count</Message><Resource>/***/slatedb-bencher_20_128/compacted/01JZ9FMS2RJ801EXBKYCX43J95.sst</Resource><RequestId>1751591317331388824</RequestId><Key>slatedb-bencher_20_128/compacted/01JZ9FMS2RJ801EXBKYCX43J95.sst</Key><BucketName>***</BucketName></Error>") } }) })

@Barre reported the same issue. The confusing thing is it says retries: 0 but max_retries: 10. Shouldn't it keep retrying until it hits the max?

To Reproduce

Get rate limited on an object store.

Expected behavior

I expect the S3ObjectSore to back off and retry.

Additional context

I tried both 0.12.0 and 0.12.2. Both exhibit the same behavior.

[tustvold]

Duplicate of #309