[Backport 2.x] Decouple replication lag from logic to fail stale replicas (#9507)

[ankitkala]

Decouple replication lag from replication timer logic used to fail stale replicas

Backport for PR: #9507

---

Signed-off-by: Ankit Kala ankikala@amazon.com
(cherry picked from commit d66df10)

[github-actions]

Compatibility status:

Checks if related components are compatible with change cef2aae

Incompatible components

Skipped components

Compatible components

[github-actions]

Compatibility status:

Checks if related components are compatible with change c1a1eaf

Incompatible components

Skipped components

Compatible components

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23926/
• CommitID: cef2aae
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23927/
• CommitID: c1a1eaf
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 596ea7b

Incompatible components

Skipped components

Compatible components

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23928/
• CommitID: 596ea7b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change be800c8

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23940/
• CommitID: be800c8
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24009/
• CommitID: be800c8
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 0738663

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsOnRemoteStoreRestore

• URL: https://build.ci.opensearch.org/job/gradle-check/24092/
• CommitID: 0738663
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #9656 (0738663) into 2.x (9e41df4) will decrease coverage by 0.07%.
Report is 1 commits behind head on 2.x.
The diff coverage is 69.56%.

@@             Coverage Diff              @@
##                2.x    #9656      +/-   ##
============================================
- Coverage     70.86%   70.79%   -0.07%     
- Complexity    57786    57867      +81     
============================================
  Files          4776     4783       +7     
  Lines        273121   273288     +167     
  Branches      40198    40217      +19     
============================================
- Hits         193537   193472      -65     
- Misses        63032    63268     +236     
+ Partials      16552    16548       -4     

Files Changed	Coverage Δ
...java/org/opensearch/common/crypto/DataKeyPair.java	0.00% <0.00%> (ø)
...rg/opensearch/common/settings/ClusterSettings.java	93.18% <ø> (ø)
...search/encryption/keyprovider/CryptoMasterKey.java	22.72% <22.72%> (ø)
...opensearch/index/SegmentReplicationShardStats.java	39.13% <40.00%> (+0.10%)	⬆️
...replication/common/SegmentReplicationLagTimer.java	40.00% <40.00%> (ø)
...earch/index/SegmentReplicationPressureService.java	76.69% <69.23%> (-0.63%)	⬇️
...rg/opensearch/encryption/CryptoManagerFactory.java	79.41% <79.41%> (ø)
...org/opensearch/index/seqno/ReplicationTracker.java	69.14% <80.95%> (+0.24%)	⬆️
...h/common/crypto/DecryptedRangedStreamProvider.java	83.33% <83.33%> (ø)
...java/org/opensearch/encryption/TrimmingStream.java	84.61% <84.61%> (ø)
... and 4 more

... and 454 files with indirect coverage changes

[mch2]

Closing as duplicate, backported this with #9705