[BUG] org.opensearch.remotestore.multipart.RemoteStoreMultipartIT is flaky

[reta]

Describe the bug

The test case org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testNoSearchIdleForAnyReplicaCount is flaky:

org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testNoSearchIdleForAnyReplicaCount

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=12912, name=Thread-7889, state=RUNNABLE, group=TGRP-RemoteStoreMultipartIT]
Caused by: org.apache.lucene.store.AlreadyClosedException: this Directory is closed
	at __randomizedtesting.SeedInfo.seed([7E1BDEE57B443ADB]:0)
	at app//org.apache.lucene.store.BaseDirectory.ensureOpen(BaseDirectory.java:50)
	at app//org.opensearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:175)
	at app//org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at app//org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at app//org.opensearch.index.store.RemoteDirectory.lambda$uploadBlob$1(RemoteDirectory.java:362)
	at app//org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getMultipartStreamSupplier$1(RemoteTransferContainer.java:167)
	at app//org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getTransferPartStreamSupplier$0(RemoteTransferContainer.java:145)
	at app//org.opensearch.common.StreamContext.provideStream(StreamContext.java:69)
	at app//org.opensearch.remotestore.multipart.mocks.MockFsAsyncBlobContainer.lambda$asyncBlobUpload$0(MockFsAsyncBlobContainer.java:60)
	at java.base@21.0.1/java.lang.Thread.run(Thread.java:1583)

Related component

Other

To Reproduce

./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testNoSearchIdleForAnyReplicaCount" -Dtests.seed=7E1BDEE57B443ADB

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context

• https://build.ci.opensearch.org/job/gradle-check/32513/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testNoSearchIdleForAnyReplicaCount/

[peternied]

[Triage - attendees 1 2 3]
@reta Thanks for filing

[andrross]

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testOverriddenBufferInterval" -Dtests.seed=56DCBC16E3B2A344 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=uk -Dtests.timezone=Asia/Baghdad -Druntime.java=21

org.opensearch.remotestore.multipart.RemoteStoreMultipartIT > testOverriddenBufferInterval FAILED
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10413, name=Thread-7230, state=RUNNABLE, group=TGRP-RemoteStoreMultipartIT]

        Caused by:
        org.apache.lucene.store.AlreadyClosedException: this Directory is closed
            at __randomizedtesting.SeedInfo.seed([56DCBC16E3B2A344]:0)
            at app//org.apache.lucene.store.BaseDirectory.ensureOpen(BaseDirectory.java:50)
            at app//org.opensearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:175)
            at app//org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
            at app//org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
            at app//org.opensearch.index.store.RemoteDirectory.lambda$uploadBlob$1(RemoteDirectory.java:362)
            at app//org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getMultipartStreamSupplier$1(RemoteTransferContainer.java:167)
            at app//org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getTransferPartStreamSupplier$0(RemoteTransferContainer.java:145)
            at app//org.opensearch.common.StreamContext.provideStream(StreamContext.java:69)
            at app//org.opensearch.remotestore.multipart.mocks.MockFsAsyncBlobContainer.lambda$asyncBlobUpload$0(MockFsAsyncBlobContainer.java:60)
            at java.****@21.0.2/java.lang.Thread.run(Thread.java:1583)

https://build.ci.opensearch.org/job/gradle-check/32595/consoleText

[linuxpi]

[Storage Triage - attendees 1 2 3 4 5 6 7 8 9 10 11 12 13]

@vikasvb90 Updating the release target to 2.15. Feel free to get in touch with folks if you need any help

[sohami]

Another test is flaky in the same class:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testAsyncDurabilityThrowsExceptionWhenRestrictSettingTrue" -Dtests.seed=5083B88EA28AD82D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=el-POLYTON -Dtests.timezone=America/Adak -Druntime.java=21


com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4818, name=Thread-1060, state=RUNNABLE, group=TGRP-RemoteStoreMultipartIT]
Caused by: org.apache.lucene.store.AlreadyClosedException: this Directory is closed
	at __randomizedtesting.SeedInfo.seed([5083B88EA28AD82D]:0)
	at app//org.apache.lucene.store.BaseDirectory.ensureOpen(BaseDirectory.java:50)
	at app//org.opensearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:175)
	at app//org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at app//org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
	at app//org.opensearch.index.store.RemoteDirectory.lambda$uploadBlob$2(RemoteDirectory.java:377)
	at app//org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getMultipartStreamSupplier$1(RemoteTransferContainer.java:207)
	at app//org.opensearch.common.blobstore.transfer.RemoteTransferContainer.lambda$getTransferPartStreamSupplier$0(RemoteTransferContainer.java:185)
	at app//org.opensearch.common.StreamContext.provideStream(StreamContext.java:69)
	at app//org.opensearch.remotestore.multipart.mocks.MockFsAsyncBlobContainer.lambda$asyncBlobUpload$0(MockFsAsyncBlobContainer.java:60)
	at java.base@21.0.3/java.lang.Thread.run(Thread.java:1583)

CI: https://build.ci.opensearch.org/job/gradle-check/41593/testReport/junit/org.opensearch.remotestore.multipart/RemoteStoreMultipartIT/testAsyncDurabilityThrowsExceptionWhenRestrictSettingTrue/