Jenkins Randomized Gradle Check Failed on main

[opensearch-ci-bot]

@opensearch-project/opensearch-core team, please take a look at Build 6216 in gradle_check log for more information.

https://ci.opensearch.org/logs/ci/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/gradle_check_6216.log

[tlfeng]

The first one also occurs in PR #3576 (comment)
The second one is reported in issue #3579

> Task :server:internalClusterTest

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testRestoreSnapshotAllocationDoesNotExceedWatermark" -Dtests.seed=4CEE83E26FF11E19 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-CO -Dtests.timezone=Pacific/Chatham -Druntime.java=17

org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT > testRestoreSnapshotAllocationDoesNotExceedWatermark FAILED
    java.lang.AssertionError: 
    Expected: an empty collection
         but: <[[jxjbkkpxnq][1], node[Pbdf_5ypSOel_JaFT_0AVQ], [P], s[STARTED], a[id=g7LW_iiLQfipKHWXqvH7Zg], [jxjbkkpxnq][0], node[Pbdf_5ypSOel_JaFT_0AVQ], [P], s[STARTED], a[id=LFmMTgY2S8Wi-evP9JzAxQ]]>
        at __randomizedtesting.SeedInfo.seed([4CEE83E26FF11E19:5E562583044801A1]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.lambda$testRestoreSnapshotAllocationDoesNotExceedWatermark$2(DiskThresholdDeciderIT.java:260)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1049)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1022)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testRestoreSnapshotAllocationDoesNotExceedWatermark(DiskThresholdDeciderIT.java:260)

> Task :server:internalClusterTest

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded" -Dtests.seed=4CEE83E26FF11E19 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-CO -Dtests.timezone=Pacific/Chatham -Druntime.java=17

org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT > testHighWatermarkNotExceeded FAILED
    java.lang.AssertionError: Mismatching shard routings: []
    Expected: a collection with size <1>
         but: collection size was <0>
        at __randomizedtesting.SeedInfo.seed([4CEE83E26FF11E19:A5CF6250EF37D7F7]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.lambda$assertBusyWithDiskUsageRefresh$5(DiskThresholdDeciderIT.java:362)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1049)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.assertBusyWithDiskUsageRefresh(DiskThresholdDeciderIT.java:355)
        at org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testHighWatermarkNotExceeded(DiskThresholdDeciderIT.java:188)

[tlfeng]

Close this issue, the test failure of DiskThresholdDeciderIT > testRestoreSnapshotAllocationDoesNotExceedWatermark will be tracked in issue #4162