[Remote Store] [Repository Download Enhancement] Implement the enhanced download mechanism

[kotwanikunal]

Is your feature request related to a problem? Please describe.

• Details on [Repository] Support multi-stream downloads within Repository #8596

Describe the solution you'd like

1. The existing Repository API (https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/common/blobstore/BlobContainer.java#L80-L92) of readBlob(blobName, position, length) will be utilized to create multiple streams for a single blob file (segment/metadata)

2. The download path will be updated with an Observer/ActionListener pattern to enable async fetches for different parts of a blob in a non-blocking fashion

   1. At the file level, when all the streams have completed, the corresponding listener will be notified. (parallel streams for a file)
   2. At the segment restore level, when all of the files have notified download complete, the restore process will be continued forward. (parallel files)

3. A metadata fetch will occur to calculate/load the hash as well as the length of the file stored within the repository before the streams are opened for a file.

   1. Parallel streams for a file will read data into a buffer, save it as a part of a local file in a temporary location.
   2. Once all the streams are marked as completed, the file will be merged and stored in the segment directory

4. Vendor plugins will take the decision of determining individual part size for a given file size. An abstraction can take an input of part size determined by plugin to be most suitable for a given file size and provide a list of respective number of suppliers similar to the uploads implementation.

Additional context

• [POC] Support multi-stream downloads within Repository #8729