Fix flakyness in RemoteStoreRefreshListenerIT

[ashking94]

Description

Fix flakyness in RemoteStoreRefreshListernerIT -> testRemoteRefreshRetryOnFailure

Related Issues

Resolves #7703

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testPitCreatedOnReplica
      1 org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

• URL: https://build.ci.opensearch.org/job/gradle-check/19504/
• CommitID: 0c46d06
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8547 (75d5987) into main (3928ddc) will increase coverage by 0.13%.
The diff coverage is 25.00%.

@@             Coverage Diff              @@
##               main    #8547      +/-   ##
============================================
+ Coverage     70.81%   70.95%   +0.13%     
- Complexity    56894    56986      +92     
============================================
  Files          4758     4758              
  Lines        269359   269359              
  Branches      39409    39410       +1     
============================================
+ Hits         190740   191114     +374     
+ Misses        62545    62107     -438     
- Partials      16074    16138      +64     

Impacted Files	Coverage Δ
...nsearch/gradle/testclusters/OpenSearchCluster.java	0.00% <0.00%> (ø)
...opensearch/gradle/testclusters/OpenSearchNode.java	0.00% <0.00%> (ø)
.../gradle/testclusters/TestClusterConfiguration.java	0.00% <ø> (ø)
...rch/search/query/ConcurrentQueryPhaseSearcher.java	56.75% <7.69%> (-18.25%)	⬇️
...ensearch/search/internal/ContextIndexSearcher.java	68.04% <28.57%> (-2.95%)	⬇️
.../org/opensearch/search/internal/SearchContext.java	36.17% <33.33%> (-0.20%)	⬇️
...n/java/org/opensearch/search/query/QueryPhase.java	78.14% <50.00%> (-0.67%)	⬇️
...search/index/shard/RemoteStoreRefreshListener.java	82.92% <75.00%> (-0.41%)	⬇️

... and 465 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19505/
• CommitID: 690e293
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

The unmuted IT has been run successfully for 3000 iterations.

[ashking94]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19505/
• CommitID: 690e293
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Found another flaky test -
Test Result (4 failures / +2)
org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnThirdAttemptAttempt
org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnThirdAttemptAttempt
org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnThirdAttemptAttempt
org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnThirdAttemptAttempt

Created issue -

• [BUG] Flaky test org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testRefreshSuccessOnThirdAttemptAttempt #8549
  Will be raising a separate PR for this.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/19516/
• CommitID: 75d5987

[sachinpkale]

codecov/patch failures on files that are not changed as part of this PR.