HotToWarmTieringService changes and changes in shard balancer to tier shards #144

neetikasinghal · 2024-07-16T22:41:57Z

Description

Adds in the service side of the changes needed for hot to warm tiering.
Also, adds in the logic to run shard allocation for the shards accepted for tiering - it does not handle tiering of replicas yet (Allocator changes to support tiering with replicas opensearch-project/OpenSearch#14850)
Also tested the tiering changes end-to-end by pulling in recovery changes: Writable warm relocation recovery opensearch-project/OpenSearch#14670

Related Issues

opensearch-project#14545
opensearch-project#13980

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

server/src/main/java/org/opensearch/common/settings/IndexScopedSettings.java

...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java

kotwanikunal · 2024-07-17T17:01:05Z

...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java

+     * pool.
+     */
+    public void tierShards() {
+        List<ShardRouting> shardsPendingTiering = new ArrayList<>();


It might be better to break this down into replicas and primary.
We can run the operation on replicas first, and the migration for primary would handle the scenario of flipping replica to primary.
(There would be a case where you have no replica - which is fine. But when you do, it is safer)

...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java

server/src/main/java/org/opensearch/cluster/routing/allocation/allocator/ShardsBalancer.java

server/src/main/java/org/opensearch/node/Node.java

server/src/main/java/org/opensearch/tiering/HotToWarmTieringService.java

...c/internalClusterTest/java/org/opensearch/remotestore/tiering/HotToWarmTieringServiceIT.java

...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java

server/src/test/java/org/opensearch/indices/tiering/HotToWarmTieringServiceTests.java

server/src/main/java/org/opensearch/indices/tiering/HotToWarmTieringService.java

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java

server/src/main/java/org/opensearch/node/Node.java

…arch-project#14851) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com>

…roject#14812) * Use default value when index.number_of_replicas is null Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add integration test Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * Add changelog Signed-off-by: Liyun Xiu <xiliyun@amazon.com> --------- Signed-off-by: Liyun Xiu <xiliyun@amazon.com>

…ile. (opensearch-project#14684) * Implement write and read flow to upload/download shard diff file. Signed-off-by: Shailendra Singh <singhlhs@amazon.com>

…oject#14426) * Optimize Cluster Stats Indices to precomute node level stats Signed-off-by: Pranshu Shukla <pranshushukla06@gmail.com>

…ary shards per index (opensearch-project#14908) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

…-project#14761) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com>

…ch-project#14918) The test would previously fail if the randomness led to only a single indexing request being included in the bulk payload. This change guarantees multiple indexing requests in order to ensure the batch logic kicks in. Also replace some unneeded mocks with real classes. Signed-off-by: Andrew Ross <andrross@amazon.com>

…shAndMergeHeaders (opensearch-project#15051) Signed-off-by: Craig Perkins <cwperx@amazon.com>

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

…ch distribution (opensearch-project#15082) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Resolves: opensearch-project#14706 Signed-off-by: Peter Nied <peternied@hotmail.com>

…ensearch-project#14711) --------- Signed-off-by: kkewwei <kkewwei@163.com>

…ensearch-project#15072) Signed-off-by: Jay Deng <jayd0104@gmail.com>

…ject#15037) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

…ta stream but is unused (opensearch-project#15080) * Fix delete not-using index template failed when the index pattern matches a data stream Signed-off-by: Gao Binlong <gbinlong@amazon.com> * modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Fix version check Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Signed-off-by: Craig Perkins <craig5008@gmail.com>

…e failed (opensearch-project#15101) Signed-off-by: Gao Binlong <gbinlong@amazon.com>

…pensearch-project#15110) * Bump org.tukaani:xz from 1.9 to 1.10 in /plugins/ingest-attachment Bumps [org.tukaani:xz](https://github.com/tukaani-project/xz-java) from 1.9 to 1.10. - [Release notes](https://github.com/tukaani-project/xz-java/releases) - [Changelog](https://github.com/tukaani-project/xz-java/blob/master/NEWS.md) - [Commits](tukaani-project/xz-java@v1.9...v1.10) --- updated-dependencies: - dependency-name: org.tukaani:xz dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…NDEX_SETTING feature (opensearch-project#15076) Signed-off-by: Neetika Singhal <neetiks@amazon.com>

* Bump actions/setup-java from 1 to 4 Bumps [actions/setup-java](https://github.com/actions/setup-java) from 1 to 4. - [Release notes](https://github.com/actions/setup-java/releases) - [Commits](actions/setup-java@v1...v4) --- updated-dependencies: - dependency-name: actions/setup-java dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…#15086) GitHub doesn't suport dynamic days since created/modified, I've created a simple redirect on my website that will support this use case. See https://peternied.github.io/redirect/issue_search.html for full context on what is avaliable. Source is available on https://github.com/peternied/peternied.github.io Signed-off-by: Peter Nied <peternied@hotmail.com>

…y-hdfs (opensearch-project#15119) * Bump org.apache.avro:avro from 1.11.3 to 1.12.0 in /plugins/repository-hdfs Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com>

…astle:bc-fips from 1.0.2.5 to 2.0.0 (opensearch-project#15122) * Bump org.bouncycastle:bcpg-fips from 1.0.7.1 to 2.0.8 and org.bouncycastle:bc-fips from 1.0.2.5 to 2.0.0 in /distribution/tools/plugin-cli Signed-off-by: Craig Perkins <cwperx@amazon.com> * Add to CHANGELOG Signed-off-by: Craig Perkins <cwperx@amazon.com> --------- Signed-off-by: Craig Perkins <cwperx@amazon.com>

Signed-off-by: David Zane <davizane@amazon.com>

…onstant_keyword field type (opensearch-project#15127) Signed-off-by: Gao Binlong <gbinlong@amazon.com>

…y-azure (opensearch-project#15111) * Bump com.azure:azure-core in /plugins/repository-azure Bumps [com.azure:azure-core](https://github.com/Azure/azure-sdk-for-java) from 1.49.1 to 1.51.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core_1.49.1...azure-core_1.51.0) --- updated-dependencies: - dependency-name: com.azure:azure-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Updating SHAs Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…ct#15134) Signed-off-by: Rishabh Singh <sngri@amazon.com>

Signed-off-by: Liyun Xiu <xiliyun@amazon.com>

Signed-off-by: Jay Deng <jayd0104@gmail.com>

--------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com>

…-project#13897) * initial code for the sandbox resource tracking and cancellation framework Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Fix Failing Tests Signed-off-by: Kiran Prakash <awskiran@amazon.com> * spotless Apply Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update SandboxService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update SandboxService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update SandboxTask.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Add java docs Signed-off-by: Kiran Prakash <awskiran@amazon.com> * spotless Signed-off-by: Kiran Prakash <awskiran@amazon.com> * javadocs Signed-off-by: Kiran Prakash <awskiran@amazon.com> * javadocs Signed-off-by: Kiran Prakash <awskiran@amazon.com> * java docs Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update AbstractTaskCancellation.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update SandboxModule.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Some tests and stubs Signed-off-by: Kiran Prakash <awskiran@amazon.com> * spotless Signed-off-by: Kiran Prakash <awskiran@amazon.com> * :server:testingConventions Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update AbstractTaskCancellation.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * more tests Signed-off-by: Kiran Prakash <awskiran@amazon.com> * addressing comments Signed-off-by: Kiran Prakash <awskiran@amazon.com> * revert some accidentally pushed files Signed-off-by: Kiran Prakash <awskiran@amazon.com> * resolve flakiness Signed-off-by: Kiran Prakash <awskiran@amazon.com> * renaming sandbox to querygroup and adjusting code based on merged PRs Signed-off-by: Kiran Prakash <awskiran@amazon.com> * jvm to memory Signed-off-by: Kiran Prakash <awskiran@amazon.com> * missing java docs Signed-off-by: Kiran Prakash <awskiran@amazon.com> * spotless Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <awskiran@amazon.com> * pluck cancellation changes out of this PR Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove unused Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove cancellation related code and add more tests coverage Signed-off-by: Kiran Prakash <awskiran@amazon.com> * us only memory and not jvm Signed-off-by: Kiran Prakash <awskiran@amazon.com> * test conventions Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Bring back enum Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update SearchBackpressureService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * revert changes Signed-off-by: Kiran Prakash <awskiran@amazon.com> * revert changes Signed-off-by: Kiran Prakash <awskiran@amazon.com> * all required changes Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <awskiran@amazon.com> * cleanups Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Delete QueryGroupService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * cleanups Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update QueryGroupLevelResourceUsageViewTests.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update QueryGroupLevelResourceUsageViewTests.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <awskiran@amazon.com> * rebasing with latest main Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove experimental Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove queryGroupId Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * change code comments Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remmove QueryGroupUsageTracker Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update QueryGroupResourceUsageTrackerService.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove QueryGroupTestHelpers Signed-off-by: Kiran Prakash <awskiran@amazon.com> * cleanups Signed-off-by: Kiran Prakash <awskiran@amazon.com> * remove queryGroupHelper Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update ResourceTypeTests.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * extend OpenSearchTestCase Signed-off-by: Kiran Prakash <awskiran@amazon.com> * pr comments Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update CHANGELOG.md Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update QueryGroupResourceUsageTrackerServiceTests.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update ResourceTypeTests.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update ResourceTypeTests.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update ResourceType.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> * Update ResourceType.java Signed-off-by: Kiran Prakash <awskiran@amazon.com> --------- Signed-off-by: Kiran Prakash <awskiran@amazon.com>

Signed-off-by: Jay Deng <jayd0104@gmail.com>

…search-project#14234) Signed-off-by: Andrew Ross <andrross@amazon.com>

Signed-off-by: Neetika Singhal <neetiks@amazon.com>

neetikasinghal requested a review from sohami as a code owner July 16, 2024 22:41

neetikasinghal commented Jul 16, 2024

View reviewed changes

server/src/main/java/org/opensearch/common/settings/IndexScopedSettings.java Outdated Show resolved Hide resolved

kotwanikunal reviewed Jul 17, 2024

View reviewed changes

jed326 reviewed Jul 17, 2024

View reviewed changes

neetikasinghal mentioned this pull request Jul 17, 2024

Add rest, transport layer changes for Tiering opensearch-project/OpenSearch#13980

Merged

9 tasks

harishbhakuni reviewed Jul 17, 2024

View reviewed changes

neetikasinghal force-pushed the tiering branch from 5569b43 to da7c5c2 Compare July 18, 2024 07:35

neetikasinghal force-pushed the tiering-service branch from 044ea5b to 0a64950 Compare July 19, 2024 00:22

harishbhakuni reviewed Jul 19, 2024

View reviewed changes

.../src/main/java/org/opensearch/cluster/routing/allocation/allocator/RemoteShardsBalancer.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/node/Node.java Show resolved Hide resolved

neetikasinghal force-pushed the tiering-service branch from 0a64950 to dbb74c8 Compare July 19, 2024 19:12

neetikasinghal force-pushed the tiering branch 3 times, most recently from 9f55e78 to b1107ec Compare July 22, 2024 17:29

neetikasinghal force-pushed the tiering-service branch from dbb74c8 to fde4f91 Compare July 22, 2024 18:33

neetikasinghal force-pushed the tiering branch from b1107ec to 31256f3 Compare July 22, 2024 22:38

neetikasinghal changed the base branch from tiering to main July 23, 2024 05:08

neetikasinghal changed the base branch from main to tiering July 23, 2024 05:09

neetikasinghal marked this pull request as draft July 23, 2024 05:10

neetikasinghal force-pushed the tiering-service branch from fde4f91 to 2e9f80d Compare July 23, 2024 05:12

neetikasinghal force-pushed the tiering branch from 31256f3 to 8ff3bcc Compare July 23, 2024 05:16

neetikasinghal mentioned this pull request Jul 23, 2024

HotToWarmTieringService changes to tier shards opensearch-project/OpenSearch#14891

Open

3 tasks

Caching avg total bytes and avg free bytes inside ClusterInfo (opense…

1305002

…arch-project#14851) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com>

neetikasinghal force-pushed the tiering-service branch from 2e9f80d to 052d551 Compare July 23, 2024 05:38

chishui and others added 6 commits July 23, 2024 08:14

[Remote Routing Table] Implement write and read flow for shard diff f…

f85a58f

…ile. (opensearch-project#14684) * Implement write and read flow to upload/download shard diff file. Signed-off-by: Shailendra Singh <singhlhs@amazon.com>

Optimized ClusterStatsIndices to precomute shard stats (opensearch-pr…

5026af6

…oject#14426) * Optimize Cluster Stats Indices to precomute node level stats Signed-off-by: Pranshu Shukla <pranshushukla06@gmail.com>

Fix constraint bug which allows more primary shards than average prim…

3497081

…ary shards per index (opensearch-project#14908) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Optmising AwarenessAllocationDecider for hashmap.get call (opensearch…

e46d1d8

…-project#14761) Signed-off-by: RS146BIJAY <rishavsagar4b1@gmail.com>

cwperks and others added 23 commits July 31, 2024 20:23

Add javadoc about ThreadContextPermission for stashWithOrigin and sta…

67a2e4c

…shAndMergeHeaders (opensearch-project#15051) Signed-off-by: Craig Perkins <cwperx@amazon.com>

Add 2.17.0 in main branch (opensearch-project#15053)

d4e7766

Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com>

Add MacOS aarch64 to precommit since we rolled out the support for su…

7c471a0

…ch distribution (opensearch-project#15082) Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Bump Netty to 4.1.112.Final (opensearch-project#15081)

48634bd

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Decommission the Core Triage meeting (opensearch-project#15085)

f829a9f

Resolves: opensearch-project#14706 Signed-off-by: Peter Nied <peternied@hotmail.com>

support rangeQuery and regexpQuery in constant_keyword field type (op…

bd226c2

…ensearch-project#14711) --------- Signed-off-by: kkewwei <kkewwei@163.com>

Support scripting for composite aggs in concurrent segment search (op…

a785073

…ensearch-project#15072) Signed-off-by: Jay Deng <jayd0104@gmail.com>

Fix RemoteCloneIndex flaky test by using sync FS repo (opensearch-pro…

47171f8

…ject#15037) Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

Update README to 2.17.0 (opensearch-project#15099)

e8c6f0f

Signed-off-by: Craig Perkins <craig5008@gmail.com>

Fix version check in yml test for the bug fix of delete index templat…

b911b6f

…e failed (opensearch-project#15101) Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Fix NODE_SEARCH_CACHE_SIZE_SETTING initialization for TIERED_REMOTE_I…

f0ef14d

…NDEX_SETTING feature (opensearch-project#15076) Signed-off-by: Neetika Singhal <neetiks@amazon.com>

Add took time to request nodes stats (opensearch-project#15054)

76b9931

Signed-off-by: David Zane <davizane@amazon.com>

Fix version check for adding rangeQuery and regexpQuery support for c…

c725431

…onstant_keyword field type (opensearch-project#15127) Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Add baseline-cluster-config key to benchmark config (opensearch-proje…

f980924

…ct#15134) Signed-off-by: Rishabh Singh <sngri@amazon.com>

Fix bulk ingest NPE with empty pipeline (opensearch-project#15033)

b47b401

Signed-off-by: Liyun Xiu <xiliyun@amazon.com>

CODEOWNERS personalizations for jed326 (opensearch-project#15137)

212597e

Signed-off-by: Jay Deng <jayd0104@gmail.com>

neetikasinghal force-pushed the tiering-service branch from 4774394 to 7da1aa6 Compare August 7, 2024 00:41

bharath-techie and others added 5 commits August 7, 2024 12:27

Add changes to build star tree in off heap (opensearch-project#14817)

a918530

--------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com>

Fix CHANGELOG for opensearch-project#15054 (opensearch-project#15150)

348c04e

Signed-off-by: Jay Deng <jayd0104@gmail.com>

Add 'ShardManagement:*' labels to Cluster Manager triage search (open…

7f72a6e

…search-project#14234) Signed-off-by: Andrew Ross <andrross@amazon.com>

HotToWarmTieringService changes to tier shards

d99f55f

Signed-off-by: Neetika Singhal <neetiks@amazon.com>

neetikasinghal force-pushed the tiering-service branch from 7da1aa6 to d99f55f Compare August 7, 2024 23:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HotToWarmTieringService changes and changes in shard balancer to tier shards #144

HotToWarmTieringService changes and changes in shard balancer to tier shards #144

neetikasinghal commented Jul 16, 2024 •

edited

Loading

kotwanikunal Jul 17, 2024

HotToWarmTieringService changes and changes in shard balancer to tier shards #144

Are you sure you want to change the base?

HotToWarmTieringService changes and changes in shard balancer to tier shards #144

Conversation

neetikasinghal commented Jul 16, 2024 • edited Loading

Description

Related Issues

Check List

kotwanikunal Jul 17, 2024

Choose a reason for hiding this comment

neetikasinghal commented Jul 16, 2024 •

edited

Loading