[CI] Test failure in "org.opensearch.cluster.coordination.Zen1IT > testZen2NodesJoiningZen1Cluster"

[tlfeng]

Describe the bug
It's a flaky test failure found in CI, and not reproducible locally.
First found in the issue #2045 (comment)
The code located in:

OpenSearch/server/src/internalClusterTest/java/org/opensearch/cluster/coordination/Zen1IT.java

Line 110 in 8725061

public void testZen2NodesJoiningZen1Cluster() {

Test log and report:
Log 2201
Reports 2201

Related part from the test log:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.coordination.Zen1IT.testZen2NodesJoiningZen1Cluster" -Dtests.seed=9BEC23D1ED301BA0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-DO -Dtests.timezone=Pacific/Wallis -Druntime.java=17

org.opensearch.cluster.coordination.Zen1IT > testZen2NodesJoiningZen1Cluster FAILED
    java.lang.AssertionError: node_t0 still having pending states:
    cluster uuid: Dzc-NXAnQIej_D7Uyx2ufw [committed: false]
...
    Expected: an empty array
         but: array size was <1>
        at __randomizedtesting.SeedInfo.seed([9BEC23D1ED301BA0:8EB7DC57E5182715]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.opensearch.test.OpenSearchIntegTestCase.lambda$afterInternal$3(OpenSearchIntegTestCase.java:622)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1048)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1021)
        at org.opensearch.test.OpenSearchIntegTestCase.afterInternal(OpenSearchIntegTestCase.java:620)
        at org.opensearch.test.OpenSearchIntegTestCase.cleanUpCluster(OpenSearchIntegTestCase.java:2377)
...
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:833)

To Reproduce
Not reproducible locally.

Expected behavior
The integration test passes.

Plugins
no other plugins installed.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: Amazon Linux
• Version: 2

Additional context
It may not worth to fix the flaky test failure, because "Zen1 Discovery" and the test have been removed in OpenSearch 2.x version in the commit 4db97aa.

[owaiskazi19]

Backport is pending for #1216. Zen1 directory is already been removed on main.

[tlfeng]

Close this issue, since the test has been removed from main branch in PR #1216 as mentioned above. Currently main branch has got code for OpenSearch version 3.x .