Decrease store refcount on any failure to create NRTReplicationEngine #9796

sachinpkale · 2023-09-06T07:21:07Z

Description

Currently, NRTReplicationEngine calls store.incRef() in the constructor and corresponding store.decRef() is only called on IOException.
But in case of exceptions like TranslogCorruptedException, store is not de-referenced causing ShardLock issues.
In this PR, we make sure to always call store.decRef() if there is a failure in creating NRTReplicationEngine.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-06T07:56:39Z

Compatibility status:

Checks if related components are compatible with change 8d37065

Incompatible components

Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-06T08:07:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

URL: https://build.ci.opensearch.org/job/gradle-check/24568/
CommitID: 8d37065
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-09-06T08:08:52Z

Codecov Report

Merging #9796 (f9ad560) into main (951f051) will increase coverage by 0.01%.
The diff coverage is 85.71%.

@@             Coverage Diff              @@
##               main    #9796      +/-   ##
============================================
+ Coverage     71.02%   71.04%   +0.01%     
- Complexity    58040    58085      +45     
============================================
  Files          4831     4831              
  Lines        273927   273931       +4     
  Branches      39913    39915       +2     
============================================
+ Hits         194567   194604      +37     
+ Misses        63061    62985      -76     
- Partials      16299    16342      +43

Files Changed	Coverage Δ
.../opensearch/index/engine/NRTReplicationEngine.java	`78.77% <85.71%> (+1.62%)`	⬆️

... and 443 files with indirect coverage changes

BhumikaSaini-Amazon

Please could you help add tests for this change?

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

github-actions · 2023-09-07T04:17:52Z

Compatibility status:

Checks if related components are compatible with change 6055d8c

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

server/src/test/java/org/opensearch/index/engine/NRTReplicationEngineTests.java

github-actions · 2023-09-07T04:40:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24812/
CommitID: 7be0ba7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-07T04:45:48Z

Compatibility status:

Checks if related components are compatible with change 7be0ba7

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-07T09:49:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24859/
CommitID: 7be0ba7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Sachin Kale <kalsac@amazon.com>

ashking94

lgtm

github-actions · 2023-09-07T10:03:43Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24865/
CommitID: f9ad560
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-07T10:34:57Z

Compatibility status:

Checks if related components are compatible with change f9ad560

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-07T10:55:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

URL: https://build.ci.opensearch.org/job/gradle-check/24868/
CommitID: f9ad560
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…#9796) * Decrease store refcount on any failure to create NRTReplicationEngine Signed-off-by: Sachin Kale <kalsac@amazon.com> * Add unit tests Signed-off-by: Sachin Kale <kalsac@amazon.com> * Fix spotless Signed-off-by: Sachin Kale <kalsac@amazon.com> --------- Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> (cherry picked from commit 199123d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…opensearch-project#9796) * Decrease store refcount on any failure to create NRTReplicationEngine Signed-off-by: Sachin Kale <kalsac@amazon.com> * Add unit tests Signed-off-by: Sachin Kale <kalsac@amazon.com> * Fix spotless Signed-off-by: Sachin Kale <kalsac@amazon.com> --------- Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>

…#9796) (#9903) Signed-off-by: Sachin Kale <kalsac@amazon.com>

…#9796) (#9902) --------- Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>

…opensearch-project#9796) * Decrease store refcount on any failure to create NRTReplicationEngine Signed-off-by: Sachin Kale <kalsac@amazon.com> * Add unit tests Signed-off-by: Sachin Kale <kalsac@amazon.com> * Fix spotless Signed-off-by: Sachin Kale <kalsac@amazon.com> --------- Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…opensearch-project#9796) * Decrease store refcount on any failure to create NRTReplicationEngine Signed-off-by: Sachin Kale <kalsac@amazon.com> * Add unit tests Signed-off-by: Sachin Kale <kalsac@amazon.com> * Fix spotless Signed-off-by: Sachin Kale <kalsac@amazon.com> --------- Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>

…opensearch-project#9796) * Decrease store refcount on any failure to create NRTReplicationEngine Signed-off-by: Sachin Kale <kalsac@amazon.com> * Add unit tests Signed-off-by: Sachin Kale <kalsac@amazon.com> * Fix spotless Signed-off-by: Sachin Kale <kalsac@amazon.com> --------- Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…opensearch-project#9796) * Decrease store refcount on any failure to create NRTReplicationEngine Signed-off-by: Sachin Kale <kalsac@amazon.com> * Add unit tests Signed-off-by: Sachin Kale <kalsac@amazon.com> * Fix spotless Signed-off-by: Sachin Kale <kalsac@amazon.com> --------- Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

sachinpkale added the skip-changelog label Sep 6, 2023

BhumikaSaini-Amazon suggested changes Sep 6, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java Show resolved Hide resolved

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java Show resolved Hide resolved

gbbafna approved these changes Sep 6, 2023

View reviewed changes

andrross approved these changes Sep 7, 2023

View reviewed changes

server/src/test/java/org/opensearch/index/engine/NRTReplicationEngineTests.java Show resolved Hide resolved

Sachin Kale added 3 commits September 7, 2023 15:20

Decrease store refcount on any failure to create NRTReplicationEngine

7520e4c

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Add unit tests

4f2b32a

Signed-off-by: Sachin Kale <kalsac@amazon.com>

Fix spotless

f9ad560

Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale force-pushed the nrt-engine-translog-corruption branch from 7be0ba7 to f9ad560 Compare September 7, 2023 09:50

ashking94 approved these changes Sep 7, 2023

View reviewed changes

sachinpkale merged commit 199123d into opensearch-project:main Sep 7, 2023

sachinpkale added backport 2.x Backport to 2.x branch backport 2.10 Backport to 2.10 branch labels Sep 7, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 7, 2023

[Backport 2.x] Decrease store refcount on any failure to create NRTReplicationEngine #9874

Closed

opensearch-trigger-bot bot mentioned this pull request Sep 7, 2023

[Backport 2.10] Decrease store refcount on any failure to create NRTReplicationEngine #9875

Closed

sachinpkale mentioned this pull request Sep 7, 2023

[Backport 2.x] Decrease store refcount on any failure to create NRTReplicationEngine #9902

Merged

sachinpkale mentioned this pull request Sep 7, 2023

[Backport 2.10] Decrease store refcount on any failure to create NRTReplicationEngine #9903

Merged

gbbafna pushed a commit that referenced this pull request Sep 7, 2023

Decrease store refcount on any failure to create NRTReplicationEngine (…

1ddb085

…#9796) (#9903) Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale added a commit that referenced this pull request Sep 7, 2023

Decrease store refcount on any failure to create NRTReplicationEngine (…

12546f7

…#9796) (#9902) --------- Signed-off-by: Sachin Kale <kalsac@amazon.com> Co-authored-by: Sachin Kale <kalsac@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decrease store refcount on any failure to create NRTReplicationEngine #9796

Decrease store refcount on any failure to create NRTReplicationEngine #9796

sachinpkale commented Sep 6, 2023 •

edited

Loading

github-actions bot commented Sep 6, 2023

github-actions bot commented Sep 6, 2023

codecov bot commented Sep 6, 2023 •

edited

Loading

BhumikaSaini-Amazon left a comment

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

ashking94 left a comment

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

github-actions bot commented Sep 7, 2023

Decrease store refcount on any failure to create NRTReplicationEngine #9796

Decrease store refcount on any failure to create NRTReplicationEngine #9796

Conversation

sachinpkale commented Sep 6, 2023 • edited Loading

Description

Check List

github-actions bot commented Sep 6, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Sep 6, 2023 • edited Loading

Codecov Report

BhumikaSaini-Amazon left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 7, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 7, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 7, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 7, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 7, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 7, 2023

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Sep 6, 2023 •

edited

Loading

codecov bot commented Sep 6, 2023 •

edited

Loading