Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decouple remote state configuration #11858

Merged
merged 4 commits into from
Mar 18, 2024
Merged

Decouple remote state configuration #11858

merged 4 commits into from
Mar 18, 2024

Conversation

soosinha
Copy link
Member

@soosinha soosinha commented Jan 11, 2024 •
edited
Loading

Description

For remote store, segment and translog repo need to be configured. For remote state, cluster state repo is required. But as per current logic either all three repos should be present or none of them should be preesnt. So remote state is tightly coupled with remote store.
This PR decouples the remote state setting so that it can be enabled or disabled as an independent feature.

Related Issues

#9884

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 11, 2024 •
edited
Loading

Compatibility status:

Checks if related components are compatible with change db7683c

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/alerting.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/sql.git]

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for d8ab19e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@soosinha soosinha force-pushed the decouple_remotestate branch from d8ab19e to 3f67648 Compare January 11, 2024 10:35
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 3f67648: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@soosinha
Copy link
Member Author

Tagging @shwetathareja @psychbot for review.

@shwetathareja
Copy link
Member

No other major comment, lets add new test cover new combination where remote state is enabled without remote store for segments.

@soosinha soosinha force-pushed the decouple_remotestate branch from 33f810d to 8e220e2 Compare March 12, 2024 10:54
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for 8e220e2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@soosinha
Copy link
Member Author

Gradle check failure due to flaky test: #12593

soosinha added 4 commits March 13, 2024 13:52
@soosinha
Decouple remote state configuration
fc831d6 
Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
@soosinha
Rename validation method in RemoteStoreNodeAttribute
2204505 
Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
@soosinha
Add unit tests to cover remote state clusters
d90894a 
Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
@soosinha
Make remote state repo mandatory for remote data store
db7683c 
Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
@soosinha soosinha force-pushed the decouple_remotestate branch from 8e220e2 to db7683c Compare March 13, 2024 08:22
@github-actions GitHub Actions
Copy link
Contributor

❕ Gradle check result for db7683c: UNSTABLE

  • TEST FAILURES:
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@soosinha
Copy link
Member Author

Gradle check failure due to flaky test: #10558

@gbbafna
Copy link
Collaborator

gbbafna commented Mar 15, 2024

@soosinha : Would existing non-remote clusters be able to use remote state after upgrading to whatever the version we are targeting ?

@sachinpkale sachinpkale merged commit 00d4b71 into opensearch-project:main Mar 18, 2024
33 of 39 checks passed
@sachinpkale sachinpkale added the backport 2.x Backport to 2.x branch label Mar 18, 2024
@soosinha
Copy link
Member Author

@gbbafna Yes Existing non-remote store clusters should be able to use remote state after upgrade.

opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 18, 2024
@github-actions
Decouple remote state configuration (#11858)
7bff2c8 
* Decouple remote state configuration

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
(cherry picked from commit 00d4b71)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Mar 18, 2024
Merged
rayshrey pushed a commit to rayshrey/OpenSearch that referenced this pull request Mar 18, 2024
@soosinha @rayshrey
Decouple remote state configuration (opensearch-project#11858)
2e6dc0d 
* Decouple remote state configuration

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
reta added a commit to reta/OpenSearch that referenced this pull request Mar 18, 2024
@reta
Revert "Decouple remote state configuration (opensearch-project#11858)"
a8c44b1 
This reverts commit 00d4b71.
sachinpkale pushed a commit that referenced this pull request Mar 19, 2024
@opensearch-trigger-bot @github-actions
Decouple remote state configuration (#11858) (#12712)
a8fa92a 
* Decouple remote state configuration


(cherry picked from commit 00d4b71)

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
ruai0511 pushed a commit to ruai0511/OpenSearch that referenced this pull request Mar 19, 2024
@soosinha @ruai0511
Decouple remote state configuration (opensearch-project#11858)
28e8b1e 
* Decouple remote state configuration

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
@soosinha soosinha mentioned this pull request Mar 20, 2024
Closed
4 tasks
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
@soosinha @shiv0408
Decouple remote state configuration (opensearch-project#11858)
7532e5c 
* Decouple remote state configuration

Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
@BhumikaSaini-Amazon BhumikaSaini-Amazon mentioned this pull request May 3, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shwetathareja shwetathareja shwetathareja left review comments

@linuxpi linuxpi linuxpi approved these changes

@sachinpkale sachinpkale sachinpkale approved these changes

@abbashus abbashus Awaiting requested review from abbashus

@adnapibar adnapibar Awaiting requested review from adnapibar

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

Assignees

@soosinha soosinha

Labels
backport 2.x Backport to 2.x branch Cluster Manager skip-changelog
Projects
Cluster Manager Project Board
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@soosinha @sohami @shwetathareja @gbbafna @sachinpkale @linuxpi