Add async blob read and download support using multiple streams #9592

kotwanikunal · 2023-08-28T22:09:51Z

Description

Adds async blob read and download support using multiple streams using new APIs
The entire code is laid out as a part of Add interface changes for async repository downloads #9182

Implementation details

The request flow can be visualized using the following diagram:

Related Issues

Partially resolves #9031

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-28T22:15:57Z

Compatibility status:

Checks if related components are compatible with change 6a7e26c

Incompatible components

Skipped components

Compatible components

github-actions · 2023-08-28T22:42:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23661/
CommitID: 6a7e26c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

.../src/main/java/org/opensearch/common/blobstore/stream/read/listener/ReadContextListener.java

...main/java/org/opensearch/common/blobstore/stream/read/listener/StreamCompletionListener.java

...c/main/java/org/opensearch/common/blobstore/stream/read/listener/FileCompletionListener.java

...t/java/org/opensearch/common/blobstore/stream/read/listener/FileCompletionListenerTests.java

github-actions · 2023-08-28T23:31:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23667/
CommitID: 1250157
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

kotwanikunal · 2023-08-30T17:01:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/23842/

CommitID: f77ed82
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Force pushed in quick succession.

github-actions · 2023-08-30T17:21:27Z

Compatibility status:

Checks if related components are compatible with change f77ed82

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-30T17:30:49Z

Compatibility status:

Checks if related components are compatible with change f85700b

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-30T18:07:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"false"}}
      1 org.opensearch.search.SearchTimeoutIT.testSimpleTimeout {p0={"search.concurrent_segment_search.enabled":"true"}}

URL: https://build.ci.opensearch.org/job/gradle-check/23843/
CommitID: f85700b
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

.../src/test/java/org/opensearch/common/blobstore/stream/read/listener/FilePartWriterTests.java

...er/src/test/java/org/opensearch/common/blobstore/stream/read/listener/ListenerTestUtils.java

github-actions · 2023-08-30T23:22:11Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23888/
CommitID: 3ebea21
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

kotwanikunal · 2023-08-30T23:23:48Z

https://build.ci.opensearch.org/job/gradle-check/23888/

The connection to jenkins went down.

github-actions · 2023-08-30T23:40:30Z

Compatibility status:

Checks if related components are compatible with change 3ebea21

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-31T00:08:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod
      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testReplicaPromotion
      1 org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryRefreshCommit

URL: https://build.ci.opensearch.org/job/gradle-check/23890/
CommitID: 3ebea21
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

github-actions · 2023-08-31T23:53:03Z

Compatibility status:

Checks if related components are compatible with change 39c3920

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-01T00:02:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/24025/
CommitID: 39c3920

Bukhtawar · 2023-09-01T09:05:37Z

server/src/main/java/org/opensearch/common/blobstore/stream/read/listener/FilePartWriter.java

+ int bytesRead;
+ while ((bytesRead = inputStream.read(buffer)) != -1) {
+ Channels.writeToChannel(buffer, 0, bytesRead, outputFileChannel, streamOffset);
+ streamOffset += bytesRead;
+ }


Assuming this isn't doing DIRECT_IO, it might thrash page cache for encrypted data transfers? Can we instead use FileChannel#transferFrom

transferFrom does not support repositioning the channel beyond the file length. With multi threaded operation and a new file, we do not have the complete file at hand. We might have to resort using individual files and stitching logic in that case.

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 6765b16) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

… (#9696) (cherry picked from commit 6765b16) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Bukhtawar · 2023-09-05T06:20:04Z

server/src/main/java/org/opensearch/common/blobstore/stream/read/listener/FilePartWriter.java

+ }
+ }
+
+ void processFailure(Exception e) {


What happens if the JVM crashes before the clean up gets triggered? Can you please write an IT to ensure that this case doesn't lead to corruption?

cc: @kotwanikunal @andrross @gbbafna

Good call out. Should we write to a file with a different name (i.e. add something like a ".multipart" suffix) and rename to the actual filename upon completion?

Added an issue here: #9784

Thanks @Bukhtawar and @andrross!

…search-project#9592) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…search-project#9592) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…search-project#9592) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

kotwanikunal force-pushed the multipart-downloads branch 2 times, most recently from 1250157 to 62fbe0d Compare August 28, 2023 23:00

andrross reviewed Aug 28, 2023

View reviewed changes

...t/java/org/opensearch/common/blobstore/stream/read/listener/FileCompletionListenerTests.java Outdated Show resolved Hide resolved

andrross reviewed Aug 30, 2023

View reviewed changes

kotwanikunal force-pushed the multipart-downloads branch from f85700b to 3ebea21 Compare August 30, 2023 23:09

andrross approved these changes Aug 31, 2023

View reviewed changes

Add async blob read and download support using multiple streams

39c3920

Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>

kotwanikunal force-pushed the multipart-downloads branch from 3ebea21 to 39c3920 Compare August 31, 2023 23:22

kotwanikunal merged commit 6765b16 into opensearch-project:main Sep 1, 2023
12 checks passed

Bukhtawar reviewed Sep 1, 2023

View reviewed changes

kotwanikunal mentioned this pull request Sep 1, 2023

Add async read support for S3 plugin #9694

Merged

6 tasks

kotwanikunal added the backport 2.x Backport to 2.x branch label Sep 1, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 1, 2023

[Backport 2.x] Add async blob read and download support using multiple streams #9696

Merged

kotwanikunal mentioned this pull request Sep 1, 2023

Add async segment file download support from remote store within OpenSearch core #9710

Merged

6 tasks

Bukhtawar reviewed Sep 5, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add async blob read and download support using multiple streams #9592

Add async blob read and download support using multiple streams #9592

kotwanikunal commented Aug 28, 2023

github-actions bot commented Aug 28, 2023

github-actions bot commented Aug 28, 2023

github-actions bot commented Aug 28, 2023

kotwanikunal commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

kotwanikunal commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Aug 31, 2023

github-actions bot commented Sep 1, 2023

Bukhtawar Sep 1, 2023

kotwanikunal Sep 1, 2023

Bukhtawar Sep 5, 2023

Bukhtawar Sep 5, 2023

andrross Sep 5, 2023

andrross Sep 5, 2023

kotwanikunal Sep 6, 2023

Add async blob read and download support using multiple streams #9592

Add async blob read and download support using multiple streams #9592

Conversation

kotwanikunal commented Aug 28, 2023

Description

Implementation details

Related Issues

Check List

github-actions bot commented Aug 28, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 28, 2023

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 30, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 31, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 31, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 1, 2023

Gradle Check (Jenkins) Run Completed with:

Bukhtawar Sep 1, 2023

Choose a reason for hiding this comment

kotwanikunal Sep 1, 2023

Choose a reason for hiding this comment

Bukhtawar Sep 5, 2023

Choose a reason for hiding this comment

Bukhtawar Sep 5, 2023

Choose a reason for hiding this comment

andrross Sep 5, 2023

Choose a reason for hiding this comment

andrross Sep 5, 2023

Choose a reason for hiding this comment

kotwanikunal Sep 6, 2023

Choose a reason for hiding this comment