Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add allocation decider for mixed cluster #12505

Merged

Conversation

ltaragi
Copy link
Contributor

@ltaragi ltaragi commented Mar 1, 2024

Description

A new allocation decider for migration of document replication clusters to remote store backed clusters

  • For STRICT compatibility mode, the decision is always YES
  • For remote_store_enabled indices, relocation or allocation/relocation can only be towards a remote node
  • For REMOTE_STORE migration direction:
    • New primary shards can only be allocated to a remote node
    • New replica shards can be allocated to a remote node iff the primary has been migrated/allocated to a remote node

Related Issues

As part of #12518

Check List

  • New functionality includes testing.
  • All tests pass
  • New functionality has been documented.
  • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ltaragi ltaragi changed the title Remote store shard allocation Add allocation decider for mixed cluster Mar 1, 2024
Copy link
Contributor

github-actions bot commented Mar 1, 2024

❌ Gradle check result for bfa399c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

github-actions bot commented Mar 1, 2024

Compatibility status:

Checks if related components are compatible with change 20f5207

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git]

Copy link
Contributor

github-actions bot commented Mar 4, 2024

❌ Gradle check result for 9fc1b0f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@ltaragi ltaragi force-pushed the remote-store-shard-allocation branch from 9fc1b0f to b68ec88 Compare March 4, 2024 12:46
Copy link
Contributor

github-actions bot commented Mar 4, 2024

❌ Gradle check result for b68ec88: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@ltaragi ltaragi force-pushed the remote-store-shard-allocation branch from b68ec88 to 87f2ce7 Compare March 4, 2024 13:38
Copy link
Contributor

github-actions bot commented Mar 4, 2024

❌ Gradle check result for 87f2ce7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@ltaragi ltaragi force-pushed the remote-store-shard-allocation branch from 87f2ce7 to 3de0a79 Compare March 4, 2024 16:15
Copy link
Contributor

github-actions bot commented Mar 4, 2024

❌ Gradle check result for 3de0a79: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Collaborator

@gbbafna gbbafna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gave 1 pass . Will review IT and UT later .

Copy link
Contributor

github-actions bot commented Mar 6, 2024

❌ Gradle check result for b98b0bd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@ltaragi ltaragi force-pushed the remote-store-shard-allocation branch from b98b0bd to 689f2f6 Compare March 14, 2024 11:18
Copy link
Contributor

❌ Gradle check result for 689f2f6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 194aba3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ltaragi added 6 commits March 15, 2024 14:10
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
@ltaragi ltaragi force-pushed the remote-store-shard-allocation branch from 194aba3 to 20f5207 Compare March 15, 2024 08:41
@gbbafna gbbafna marked this pull request as ready for review March 15, 2024 09:21
Copy link
Contributor

✅ Gradle check result for 20f5207: SUCCESS

Copy link

codecov bot commented Mar 15, 2024

Codecov Report

Attention: Patch coverage is 77.55102% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 71.47%. Comparing base (b15cb0c) to head (20f5207).
Report is 35 commits behind head on main.

Files Patch % Lines
...decider/RemoteStoreMigrationAllocationDecider.java 76.59% 7 Missing and 4 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #12505      +/-   ##
============================================
+ Coverage     71.42%   71.47%   +0.05%     
- Complexity    59978    60089     +111     
============================================
  Files          4985     4990       +5     
  Lines        282275   282625     +350     
  Branches      40946    40996      +50     
============================================
+ Hits         201603   202000     +397     
+ Misses        63999    63849     -150     
- Partials      16673    16776     +103     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@gbbafna gbbafna merged commit 18e5816 into opensearch-project:main Mar 15, 2024
99 checks passed
rayshrey pushed a commit to rayshrey/OpenSearch that referenced this pull request Mar 18, 2024
@gbbafna gbbafna added the backport 2.x Backport to 2.x branch label Mar 19, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 19, 2024
#12505)

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
(cherry picked from commit 18e5816)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
gbbafna pushed a commit that referenced this pull request Mar 19, 2024
#12505) (#12754)

(cherry picked from commit 18e5816)

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
ruai0511 pushed a commit to ruai0511/OpenSearch that referenced this pull request Mar 19, 2024
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
opensearch-project#12505)

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants