Handle replication cases where the remote store metadata file is not present

[ankitkala]

Description

Handle the case in segrep with remote where primary hasn't refreshed yet. This is required because we trigger a force sync here

Related Issues

Resolves #8325

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19046/
• CommitID: 36a1062
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19045/
• CommitID: 264bc7b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

• URL: https://build.ci.opensearch.org/job/gradle-check/19048/
• CommitID: 69596ea
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8433 (69596ea) into main (2f9728e) will decrease coverage by 0.15%.
The diff coverage is 100.00%.

❗ Current head 69596ea differs from pull request most recent head 041e614. Consider uploading reports for the commit 041e614 to get more accurate results

@@             Coverage Diff              @@
##               main    #8433      +/-   ##
============================================
- Coverage     71.01%   70.86%   -0.15%     
+ Complexity    57029    56909     -120     
============================================
  Files          4758     4758              
  Lines        269251   269188      -63     
  Branches      39401    39409       +8     
============================================
- Hits         191214   190771     -443     
- Misses        61958    62289     +331     
- Partials      16079    16128      +49     

Impacted Files	Coverage Δ
...ices/replication/RemoteStoreReplicationSource.java	93.93% <100.00%> (+0.39%)	⬆️

... and 494 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19381/
• CommitID: 8fcd17d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/19393/
• CommitID: 041e614

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/19904/
• CommitID: 0191916
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[mch2]

Not sure how this test passed the checks here - i'd expect not having the segrep flag enabled to fail this test 100% of the time. #8704 should fix this.