Fix issue of red index on close for remote enabled clusters

[ashking94]

Description

The close index operation involves following steps -

1. Start closing indices by adding a write block
2. Wait for the operations on the shards to be completed
   1. Acquire all indexing operation permits to ensure that all operations have completed indexing
3. After acquiring all indexing permits, closing a index involves 2 phases -
   1. Sync translog
   2. Flush Index Shard
4. Move index states from OPEN to CLOSE in cluster state for indices that are ready for closing

During a happy index close, we upload translog twice -

• 1st time, as part of the 3.a. Sync Translog step, the indexing operations are uploaded
• 2nd time, as part of the 3.b. Flush Index Shard step, the latest GCP is uploaded.

However, if there is a flush that has happened after the operation landed in the Lucene buffer but before the buffered sync (for sync translog) or the periodic async sync (for async translog), then the steps 3(a) and 3(b) becomes no-op and the GCP uploaded in the checkpoint file would be the one from the last translog sync. This causes the discrepancy between maxSeqNo and GCP and causing exception while creating ReadOnlyEngine leading to red index.

In this PR, we do a force Sync during close index to ensure that we have uploaded the translog with the latest global checkpoint.

Related Issues

Resolves #15989

Check List

• Functionality includes testing.
• [ ] API changes companion pull request created, if applicable.
• [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for a1d5a87: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[ashking94]

❌ Gradle check result for a1d5a87: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

I have added tests for the edge case that is mentioned in the referenced issue. I had added the tests first and then the main code changes -

[org.opensearch.remotestore.RemoteStoreIT.testCloseIndexWithNoOpSyncAndFlushForSyncTranslog](https://build.ci.opensearch.org/job/gradle-check/48058/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testCloseIndexWithNoOpSyncAndFlushForSyncTranslog/)
[org.opensearch.remotestore.RemoteStoreIT.testCloseIndexWithNoOpSyncAndFlushForSyncTranslog](https://build.ci.opensearch.org/job/gradle-check/48058/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testCloseIndexWithNoOpSyncAndFlushForSyncTranslog_2/)
[org.opensearch.remotestore.RemoteStoreIT.testCloseIndexWithNoOpSyncAndFlushForSyncTranslog](https://build.ci.opensearch.org/job/gradle-check/48058/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testCloseIndexWithNoOpSyncAndFlushForSyncTranslog_3/)
[org.opensearch.remotestore.RemoteStoreIT.testCloseIndexWithNoOpSyncAndFlushForSyncTranslog](https://build.ci.opensearch.org/job/gradle-check/48058/testReport/junit/org.opensearch.remotestore/RemoteStoreIT/testCloseIndexWithNoOpSyncAndFlushForSyncTranslog_4/)

[github-actions]

❌ Gradle check result for ac864f0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 29cf87f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[ashking94]

❌ Gradle check result for 29cf87f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Fixed this.

[github-actions]

❕ Gradle check result for 4420487: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review.

Project coverage is 71.86%. Comparing base (3937ccb) to head (4420487).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...rg/opensearch/index/translog/RemoteFsTranslog.java	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #15990      +/-   ##
============================================
- Coverage     71.90%   71.86%   -0.05%     
+ Complexity    64392    64351      -41     
============================================
  Files          5278     5278              
  Lines        300877   300886       +9     
  Branches      43478    43478              
============================================
- Hits         216351   216218     -133     
- Misses        66747    66920     +173     
+ Partials      17779    17748      -31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.