[Segment Replication] Update segrep bwc tests to verify replica checkpoints and skip tests for 1.x bwc versions

[dreamer-89]

Description

1. Adds segment replicatoin stats verification step by asserting checkpoints_behind metric from _cat/segment_replication API should be 0.
2. Skips test execution for 1.x bwc versions

Related

#7490

Testing

Ran test 50 times locally without any failure i.e. test is not flaky

for i in $(seq 0 50); do echo "running $i iteration\n";  ./gradlew :qa:rolling-upgrade:v2.9.0#upgradedClusterTest --tests "org.opensearch.upgrades.IndexingIT.testIndexingWithSegRep" >> /tmp/iteration_$i; done; &
...
ubuntu@ip-172-31-49-145:~$ grep "BUILD SUCCESS" /tmp/iteration_* | wc -l
51

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18120/
• CommitID: e01d3af
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/18126/
• CommitID: e01d3af
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8203 (2bd6457) into main (84dffac) will increase coverage by 0.15%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #8203      +/-   ##
============================================
+ Coverage     70.83%   70.98%   +0.15%     
- Complexity    56652    56768     +116     
============================================
  Files          4733     4733              
  Lines        268125   268125              
  Branches      39283    39283              
============================================
+ Hits         189932   190341     +409     
+ Misses        62154    61758     -396     
+ Partials      16039    16026      -13     

see 458 files with indirect coverage changes

[dreamer-89]

@mch2 : Ping for review

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testStartReplicationListenerSuccess
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/18255/
• CommitID: ca5a4ee
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[dreamer-89]

org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testStartReplicationListenerSuccess

#8136

org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

#3603

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/18264/
• CommitID: bc5e7dc

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18268/
• CommitID: 2bd6457
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dreamer-89]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/18268/
• CommitID: 2bd6457
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

#6287

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting" -Dtests.seed=2A3A8527C963DA82 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sk-SK -Dtests.timezone=Indian/Kerguelen -Druntime.java=20

org.opensearch.cluster.shards.ClusterShardLimitIT > testCreateIndexWithMaxClusterShardSetting FAILED
    java.lang.AssertionError: MaxShardPerCluster 3 should be greater than or equal to MaxShardPerNode 1000
        at __randomizedtesting.SeedInfo.seed([2A3A8527C963DA82:60DC7499643D1C3A]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.cluster.shards.ClusterShardLimitIT.setMaxShardLimit(ClusterShardLimitIT.java:750)
        at org.opensearch.cluster.shards.ClusterShardLimitIT.testCreateIndexWithMaxClusterShardSetting(ClusterShardLimitIT.java:259)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.client.ReindexIT.testReindexTask

• URL: https://build.ci.opensearch.org/job/gradle-check/18275/
• CommitID: 2bd6457
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.