[BUG] Test Failure org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded

[opensearch-ci-bot]

@opensearch-project/opensearch-core team, please take a look at Build 5977 in gradle_check log for more information.

[xuezhou25]

> Task :server:internalClusterTest

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded" -Dtests.seed=2B0CB509EFB4F322 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-TN -Dtests.timezone=America/Whitehorse -Druntime.java=17

org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT > testHighWatermarkNotExceeded FAILED
    java.lang.AssertionError: Mismatching shard routings: []
    Expected: a collection with size <1>
         but: collection size was <0>
        at __randomizedtesting.SeedInfo.seed([2B0CB509EFB4F322:C22D54BB6F723ACC]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.lambda$assertBusyWithDiskUsageRefresh$5(DiskThresholdDeciderIT.java:362)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1049)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.assertBusyWithDiskUsageRefresh(DiskThresholdDeciderIT.java:355)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded(DiskThresholdDeciderIT.java:188)

[tlfeng]

Also occurred in #3583 (comment)

[kotwanikunal]

Also in #3568 #3595 #3576

[tlfeng]

The flaky test failure is caused by the commit fc54154 / PR #3537 that upgrades Lucene version to 9.3.
The solution may be related to 055a9fc / #3591

The assertion failed at https://github.com/opensearch-project/OpenSearch/blob/2.0.1/server/src/internalClusterTest/java/org/opensearch/cluster/routing/allocation/decider/DiskThresholdDeciderIT.java#L188

[dreamer-89]

Another occurrence #3646

This test has started failing more often recently.

[sachinpkale]

Failing for #3703

[Rishikesh1159]

This test started failing after upgrading Lucene snapshot verison to "9.3.0-snapshot-b7231bb". Merge on refresh is enabled by default after upgrading. This test passes if Merge on refresh is not enabled by default. Thanks to @reta for already putting out a PR to allow users to enable merge on refresh by a index setting. Once that PR get's merged we can leverage that change in this integ test using index setting. That should fix this test failure