[CI] o.o.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks multiple failures

[nknize]

Multiple PR test failures (most recent for the following:

./gradlew ':qa:remote-clusters:integTest' --tests "org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks" -Dtests.seed=403F055E1F14E391 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-TN -Dtests.timezone=Africa/Conakry -Druntime.java=17

2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([403F055E1F14E391:4749D095249295CC]:0)
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertTrue(Assert.java:52)
        at org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks(RemoteClustersIT.java:125)
  1> [2021-12-10T20:14:09,766][INFO ][o.o.c.r.t.RemoteClustersIT] [testProxyModeConnectionWorks] before test
  1> [2021-12-10T20:14:10,385][INFO ][o.o.c.r.t.RemoteClustersIT] [testProxyModeConnectionWorks] Configuring remote cluster [opensearch-2:9300]
  1> [2021-12-10T20:14:10,487][INFO ][o.o.c.r.t.RemoteClustersIT] [testProxyModeConnectionWorks] Connection info: org.opensearch.client.cluster.RemoteConnectionInfo@688b190
  1> [2021-12-10T20:14:10,693][INFO ][o.o.c.r.t.RemoteClustersIT] [testProxyModeConnectionWorks] after test
  1> [2021-12-10T20:14:10,737][INFO ][o.o.c.r.t.RemoteClustersIT] [testSniffModeConnectionFails] before test
  1> [2021-12-10T20:14:11,294][INFO ][o.o.c.r.t.RemoteClustersIT] [testSniffModeConnectionFails] Configuring remote cluster [opensearch-2:9300]
  1> [2021-12-10T20:14:11,363][INFO ][o.o.c.r.t.RemoteClustersIT] [testSniffModeConnectionFails] Connection info: org.opensearch.client.cluster.RemoteConnectionInfo@5aef8603
  1> [2021-12-10T20:14:11,500][INFO ][o.o.c.r.t.RemoteClustersIT] [testSniffModeConnectionFails] after test
  2> NOTE: leaving temporary files on disk at: /var/CITOOL/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/search/qa/remote-clusters/build/testrun/integTest/temp/org.opensearch.cluster.remote.test.RemoteClustersIT_403F055E1F14E391-001
  2> NOTE: test params are: codec=Lucene87, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=ar-TN, timezone=Africa/Conakry
  2> NOTE: Linux 5.4.0-1045-aws amd64/Eclipse Adoptium 17.0.1 (64-bit)/cpus=72,threads=1,free=451487832,total=536870912
  2> NOTE: All tests run in this JVM: [RemoteClustersIT]

Note:

        RemoteConnectionInfo rci = cluster1Client().cluster().remoteInfo(new RemoteInfoRequest(), RequestOptions.DEFAULT).getInfos().get(0);
        logger.info("Connection info: {}", rci);
        assertTrue(rci.isConnected());

RemoteConnectionInfo logging is useless. (todo: add toString support for logging)

[dblock]

failed in #1723 again

[dblock]

#1791 (comment)

[dreamer-89]

Looking into it

[dreamer-89]

In order to repro issue, when I run repro command it doesn't run the defined task.

Command Used:

./gradlew ':qa:remote-clusters:integTest' --tests "org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks" -Dtests.seed=403F055E1F14E391 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-TN -Dtests.timezone=Africa/Conakry -Druntime.java=17

Output:

➜  OpenSearch git:(main) ✗ ./gradlew ':qa:remote-clusters:integTest' --tests "org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks" -Dtests.seed=403F055E1F14E391 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-TN -Dtests.timezone=Africa/Conakry -Druntime.java=11

Starting a Gradle Daemon, 1 incompatible Daemon could not be reused, use --status for details

> Task :buildSrc:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :buildSrc:compileGroovy
Note: /Users/singhnjb/OpenSearch/buildSrc/src/main/groovy/org/opensearch/gradle/test/TestWithSslPlugin.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :buildSrc:compileTestFixturesJava
Note: /Users/singhnjb/OpenSearch/buildSrc/src/testFixtures/java/org/opensearch/gradle/test/GradleIntegrationTestCase.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :buildSrc:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.3.3
  OS Info               : Mac OS X 11.6.2 (x86_64)
  Runtime JDK Version   : 11 (Eclipse Adoptium JDK)
  Runtime java.home     : /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
  Gradle JDK Version    : 11 (Eclipse Adoptium JDK)
  Gradle java.home      : /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
  Random Testing Seed   : 403F055E1F14E391
  In FIPS 140 mode      : false
=======================================

> Task :libs:opensearch-x-content:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :server:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :modules:aggs-matrix-stats:compileJava
Note: /Users/singhnjb/OpenSearch/modules/aggs-matrix-stats/src/main/java/org/opensearch/search/aggregations/matrix/MatrixAggregationPlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:lang-mustache:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:parent-join:compileJava
Note: /Users/singhnjb/OpenSearch/modules/parent-join/src/main/java/org/opensearch/join/ParentJoinPlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :client:rest-high-level:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :test:framework:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

BUILD SUCCESSFUL in 1m 26s
58 actionable tasks: 48 executed, 10 up-to-date

[dreamer-89]

The test in question needs docker-compose to run; this was causing the test to get silently ignored. Passing -i (info) flag reveals this.

[dreamer-89]

Ran tests in multiple iterations (1,10,100) but test did not fail even once.

[dreamer-89]

This issue is still causing gradle check failures; even though this specific test is not reproducible when only this test is run.

@anasalkouz : Reopening this to have correct root cause fix.

[andrross]

Tracked by autocut: #14599