[BUG] org.opensearch.remotestore.RemoteStoreForceMergeIT.testRestoreForceMergeSingleIteration flaky failure

[gbbafna]

Describe the bug

The assertions in delete stale translog data is tripping . This is not a reproducible failure.

https://build.ci.opensearch.org/job/gradle-check/22373/testReport/junit/org.opensearch.remotestore/RemoteStoreForceMergeIT/testRestoreForceMergeSingleIteration/

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2679, name=opensearch[node_s1][flush][T#1], state=RUNNABLE, group=TGRP-RemoteStoreForceMergeIT]
	at __randomizedtesting.SeedInfo.seed([90BCF02533C9A76B:61FEE78AB3B48CC0]:0)
Caused by: java.lang.AssertionError: [remote-store-test-idx-1][0] Expected non-empty readers
	at __randomizedtesting.SeedInfo.seed([90BCF02533C9A76B]:0)
	at org.opensearch.index.translog.RemoteFsTranslog.deleteStaleRemotePrimaryTerms(RemoteFsTranslog.java:430)
	at org.opensearch.index.translog.RemoteFsTranslog.trimUnreferencedReaders(RemoteFsTranslog.java:400)
	at org.opensearch.index.translog.InternalTranslogManager.trimUnreferencedReaders(InternalTranslogManager.java:394)
	at org.opensearch.index.engine.InternalEngine.revisitIndexDeletionPolicyOnTranslogSynced(InternalEngine.java:537)
	at org.opensearch.index.engine.InternalEngine$1.onAfterTranslogSync(InternalEngine.java:265)
	at org.opensearch.index.translog.listener.CompositeTranslogEventListener.onAfterTranslogSync(CompositeTranslogEventListener.java:49)
	at org.opensearch.index.translog.InternalTranslogManager.syncTranslog(InternalTranslogManager.java:193)
	at org.opensearch.index.shard.IndexShard.sync(IndexShard.java:4154)
	at org.opensearch.index.IndexService.maybeFSyncTranslogs(IndexService.java:980)
	at org.opensearch.index.IndexService$AsyncTranslogFSync.runInternal(IndexService.java:1104)
	at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1623)

[sachinpkale]

Looking into it.

[sachinpkale]

This should be fixed with #9458

Running the test in local for ~200 times to verify.