[Remote Translog] Fix translog inconsistent generation assertion failure in ITs

[ashking94]

Description

As seen in #10902, we can see that during test cleanup, we see an assertion failure that happens due to max reader generation being different than supplied generation. This will fix multiple flaky tests that fail due to this reason.

Related Issues

Resolves #10902

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 06df33a

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testIndexReopenClose

• URL: https://build.ci.opensearch.org/job/gradle-check/29219/
• CommitID: 06df33a
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[ashking94]

org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testIndexReopenClose

Created issue for test failure as it does not seem to be related to the code change done here - #10987

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/29515/
• CommitID: 06df33a

[codecov]

Codecov Report

Merging #10985 (06df33a) into main (73bbeb5) will decrease coverage by 0.03%.
Report is 18 commits behind head on main.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main   #10985      +/-   ##
============================================
- Coverage     71.27%   71.25%   -0.03%     
- Complexity    58724    58747      +23     
============================================
  Files          4870     4870              
  Lines        276578   276578              
  Branches      40202    40202              
============================================
- Hits         197137   197069      -68     
- Misses        62986    63073      +87     
+ Partials      16455    16436      -19     

Files	Coverage Δ
...rg/opensearch/index/translog/RemoteFsTranslog.java	75.60% <100.00%> (ø)

... and 493 files with indirect coverage changes