-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add async segment file download support from remote store within OpenSearch core #9710
Add async segment file download support from remote store within OpenSearch core #9710
Conversation
server/src/main/java/org/opensearch/index/shard/IndexShard.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/indices/replication/RemoteStoreReplicationSource.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/shard/IndexShard.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/shard/IndexShard.java
Outdated
Show resolved
Hide resolved
Compatibility status:Checks if related components are compatible with change d31c829 Incompatible componentsIncompatible components: [https://github.com/opensearch-project/k-nn.git] Skipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git] |
Gradle Check (Jenkins) Run Completed with:
|
Compatibility status:Checks if related components are compatible with change 31ea3ec Incompatible componentsIncompatible components: [https://github.com/opensearch-project/security.git] Skipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git] |
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
server/src/main/java/org/opensearch/index/shard/IndexShard.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/indices/replication/RemoteStoreReplicationSource.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/indices/replication/RemoteStoreReplicationSource.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/store/RemoteSegmentStoreDirectory.java
Outdated
Show resolved
Hide resolved
31ea3ec
to
ccb5168
Compare
Gradle Check (Jenkins) Run Completed with:
|
@Bukhtawar, here is my understanding of how the threading works.
I'll try to describe this in prose as well: Starting from IndexShard.copySegmentFiles(), for each file the calling thread does an initial blocking call to get the object metadata, but then returns immediately while registering listeners to do the rest of the work. IndexShard.copySegmentFiles() blocks because it is synchronous and expects the files to be downloaded when it returns. As for the multi-threading work, the initial call to get the S3 InputStreams is done by the I think the initial call must block unless we do a more major refactoring to make the code flows that call it asynchronous. Let me know if you have any concerns here. I have a couple comments/concerns here:
|
We should create another threadpool with much bigger size purely to do I/O . I think we are already tracking it in #10106 . For |
server/src/main/java/org/opensearch/indices/replication/RemoteStoreReplicationSource.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/indices/replication/RemoteStoreReplicationSource.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/shard/IndexShard.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/shard/IndexShard.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
ccb5168
to
d31c829
Compare
Updated the S3 reference here: #10192 |
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> (cherry picked from commit 9e90671) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@andrross Handoff to generic pool is definitely not right and we can also make metadata call non-blocking by chaining it with subsequent downloads. There is a way by which downloads can be made completely async and we wouldn't need any threadpool for multiple parallel downloads but that requires significant changes in stream processing layer (decryption/data integrity). To add more on this, there are two types of IO happening in downloads - fetch from S3 and disk writes. Fetch from S3 is async even in the current state but we are not able to truly benefit from it because each part download happens within a separate thread. And this is done to parallelize part downloads and because we need some pre-processing work like decryption to be done before committing a buffer to disk. We can make both disk writes and fetch from S3 async and still be able to do pre-processing work like decryption by implementing We can still take up this as a follow up task thoughtand as @gbbafna mentioned, since we will only be doing IO in this new pool, for now we can assign it a large size. This would mean that we would still continue to bear context switch cost and minimal CPU provided stream processing work is further bounded in a small pool. I would still very much like to see downloads happening in async given we are making certain compromises. |
…-project#9710) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
…-project#9710) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
…-project#9710) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com>
…-project#9710) Signed-off-by: Kunal Kotwani <kkotwani@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Description
RemoteSegmentStoreDirectory
path to copy over segment files from the remote store.Related Issues
Resolves #8596
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.