feat: add a separate retry loop on object contents download

[westonpace]

object_store has automatic retries. However, these retries only apply on the initial HTTP request (the sending and receiving of the headers. The download of the response body is not retried in the same way.

This download was previously covered by our outer retry loop but this had two problems:

• It was not configurable
• It was not obvious what the error was (the error would say "error decoding response body".

The download retry count is now configurable and the error message has been enhanced.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 69.84127% with 19 lines in your changes missing coverage. Please review.

Project coverage is 77.86%. Comparing base (ce1e61c) to head (38d61b8).

Files with missing lines	Patch %	Lines
rust/lance-io/src/object_store.rs	59.25%	9 Missing and 2 partials ⚠️
rust/lance-io/src/object_reader.rs	63.63%	8 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2874   +/-   ##
=======================================
  Coverage   77.85%   77.86%           
=======================================
  Files         231      231           
  Lines       70382    70442   +60     
  Branches    70382    70442   +60     
=======================================
+ Hits        54799    54848   +49     
- Misses      12458    12462    +4     
- Partials     3125     3132    +7     

Flag	Coverage Δ
unittests	77.86% <69.84%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wjones127]

Getting deja vu like I did this earlier, but maybe I am thinking about uploads.

[westonpace]

Getting deja vu like I did this earlier, but maybe I am thinking about uploads.

😆 yes, we have MAX_CONN_RESET_RETRIES in the object writer for something very similar.