Fix remote cluster restore for data stream #10777

dhwanilpatel · 2023-10-20T09:56:28Z

Description

Fixing data stream issue for remote cluster state restore.
Updated existing data stream ITs to run with Remote store restore.

Related Issues

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-20T10:21:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28564/
CommitID: 084404a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-20T10:26:37Z

Compatibility status:

Checks if related components are compatible with change 1b77f1b

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-20T11:13:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28568/
CommitID: 62c9458
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

...rc/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreClusterStateRestoreIT.java

...ernalClusterTest/java/org/opensearch/action/admin/indices/datastream/DataStreamTestCase.java

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-22T11:29:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28730/
CommitID: 220d131
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-22T13:52:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28735/
CommitID: 11c7d4d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-22T18:01:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28740/
CommitID: 9ffc943
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…e_DS_fix Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

github-actions · 2023-10-23T07:01:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28773/
CommitID: 1b77f1b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

shwetathareja · 2023-10-23T07:20:29Z

ConcurrentSearch test issue - #10154

shwetathareja · 2023-10-23T07:22:11Z

zone awareness test failure - #3603

github-actions · 2023-10-23T08:12:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/28781/
CommitID: 1b77f1b

codecov · 2023-10-23T08:12:51Z

Codecov Report

Merging #10777 (1b77f1b) into main (51626d0) will decrease coverage by 0.15%.
Report is 9 commits behind head on main.
The diff coverage is 90.90%.

@@             Coverage Diff              @@
##               main   #10777      +/-   ##
============================================
- Coverage     71.31%   71.17%   -0.15%     
- Complexity    58671    58674       +3     
============================================
  Files          4860     4869       +9     
  Lines        276335   276446     +111     
  Branches      40198    40198              
============================================
- Hits         197068   196756     -312     
- Misses        62803    63291     +488     
+ Partials      16464    16399      -65

Files	Coverage Δ
...upport/replication/TransportReplicationAction.java	`79.78% <100.00%> (-0.90%)`	⬇️
...ava/org/opensearch/cluster/node/DiscoveryNode.java	`91.62% <100.00%> (+0.17%)`	⬆️
...a/org/opensearch/common/network/NetworkModule.java	`92.20% <100.00%> (+0.20%)`	⬆️
...rg/opensearch/common/settings/ClusterSettings.java	`92.85% <ø> (ø)`
.../java/org/opensearch/gateway/GatewayMetaState.java	`69.56% <100.00%> (+1.04%)`	⬆️
...earch/index/remote/RemoteStorePressureService.java	`93.22% <ø> (-6.78%)`	⬇️
server/src/main/java/org/opensearch/node/Node.java	`85.31% <100.00%> (+0.09%)`	⬆️
...ting/admissioncontrol/AdmissionControlService.java	`100.00% <100.00%> (ø)`
...issioncontrol/controllers/AdmissionController.java	`100.00% <100.00%> (ø)`
...g/admissioncontrol/enums/AdmissionControlMode.java	`100.00% <100.00%> (ø)`
... and 13 more

... and 476 files with indirect coverage changes

* Fix remote cluster restore for data stream, also added integ test Signed-off-by: Dhwanil Patel <dhwanip@amazon.com> (cherry picked from commit b5ef078) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix remote cluster restore for data stream, also added integ test (cherry picked from commit b5ef078) Signed-off-by: Dhwanil Patel <dhwanip@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix remote cluster restore for data stream, also added integ test Signed-off-by: Dhwanil Patel <dhwanip@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Fix remote cluster restore for data stream

084404a

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Fixed UT

62c9458

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

shwetathareja reviewed Oct 22, 2023

View reviewed changes

linuxpi reviewed Oct 22, 2023

View reviewed changes

...ernalClusterTest/java/org/opensearch/action/admin/indices/datastream/DataStreamTestCase.java Outdated Show resolved Hide resolved

Incorporated comments

220d131

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

Fix test framework spotless

11c7d4d

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

minor fix

9ffc943

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

linuxpi assigned dhwanilpatel Oct 23, 2023

linuxpi added the skip-changelog label Oct 23, 2023

linuxpi approved these changes Oct 23, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/main' into remote_cluster_stat…

1b77f1b

…e_DS_fix Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

shwetathareja approved these changes Oct 23, 2023

View reviewed changes

shwetathareja mentioned this pull request Oct 23, 2023

[BUG] Flaky org.opensearch.search.aggregations.metrics.CardinalityWithRequestBreakerIT test #10154

Closed

shwetathareja mentioned this pull request Oct 23, 2023

[BUG] org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness flaky #3603

Open

shwetathareja merged commit b5ef078 into opensearch-project:main Oct 23, 2023
16 checks passed

shwetathareja added the backport 2.x Backport to 2.x branch label Oct 23, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 23, 2023

[Backport 2.x] Fix remote cluster restore for data stream #10848

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix remote cluster restore for data stream #10777

Fix remote cluster restore for data stream #10777

dhwanilpatel commented Oct 20, 2023

github-actions bot commented Oct 20, 2023

github-actions bot commented Oct 20, 2023 •

edited

Loading

github-actions bot commented Oct 20, 2023

github-actions bot commented Oct 22, 2023

github-actions bot commented Oct 22, 2023

github-actions bot commented Oct 22, 2023

github-actions bot commented Oct 23, 2023

shwetathareja commented Oct 23, 2023

shwetathareja commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

codecov bot commented Oct 23, 2023

Fix remote cluster restore for data stream #10777

Fix remote cluster restore for data stream #10777

Conversation

dhwanilpatel commented Oct 20, 2023

Description

Related Issues

Check List

github-actions bot commented Oct 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 20, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Oct 20, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

shwetathareja commented Oct 23, 2023

shwetathareja commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Oct 23, 2023

Codecov Report

github-actions bot commented Oct 20, 2023 •

edited

Loading