[CI] Failure in org.elasticsearch.indices.state.RareClusterStateIT.testDelayedMappingPropagationOnReplica

[tvernum]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/379/

Does not reproduce

./gradlew :server:integTest -Dtests.seed=F9699D9733C3983F \
  -Dtests.class=org.elasticsearch.indices.state.RareClusterStateIT \
  -Dtests.method="testDelayedMappingPropagationOnReplica" \
  -Dtests.security.manager=true -Dtests.locale=ar-TN -Dtests.timezone=Asia/Yakutsk \
  -Dcompiler.java=11 -Druntime.java=8

02:01:35   1> [2018-12-19T11:01:22,864][INFO ][o.e.i.s.RareClusterStateIT] [testDelayedMappingPropagationOnReplica] [RareClusterStateIT#testDelayedMappingPropagationOnReplica]: cleaned up after test
02:01:35   1> [2018-12-19T11:01:22,864][INFO ][o.e.i.s.RareClusterStateIT] [testDelayedMappingPropagationOnReplica] after test
02:01:35 FAILURE 29.9s J4 | RareClusterStateIT.testDelayedMappingPropagationOnReplica <<< FAILURES!
02:01:35    > Throwable #1: java.lang.AssertionError: 
02:01:35    > Expected: an instance of org.elasticsearch.action.index.IndexResponse
02:01:35    >      but: null
02:01:35    > 	at __randomizedtesting.SeedInfo.seed([F9699D9733C3983F:85175C586324A3DC]:0)
02:01:35    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
02:01:35    > 	at org.elasticsearch.indices.state.RareClusterStateIT.lambda$testDelayedMappingPropagationOnReplica$5(RareClusterStateIT.java:413)
02:01:35    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:847)
02:01:35    > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:821)
02:01:35    > 	at org.elasticsearch.indices.state.RareClusterStateIT.testDelayedMappingPropagationOnReplica(RareClusterStateIT.java:409)
02:01:35    > 	at java.lang.Thread.run(Thread.java:748)
02:01:35    > 	Suppressed: java.lang.AssertionError: 
02:01:35    > Expected: an instance of org.elasticsearch.action.support.master.AcknowledgedResponse
02:01:35    >      but: null

Previous email threads suggest there's a connection to #33673

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

This looks like the test may have just been very slow for some reason. Looking at the fact that it polled 10 times (so more than half a second) for the AcknowledgedResponse when locally this this busy assert takes me less than 10ms.
Also, there's this in the logs:

ERROR: JVM J0 ended with an exception: Forked process returned with error code: 137. Very likely a JVM crash.  See process stderr at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+periodic/server/build/testrun/integTest/temp/junit4-J0-20181219_015515_2601288045382664915697.syserr
	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1542)
	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:123)
	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:997)
	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:994)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

=> seems like the kernel OOM killer killed a test JVM? ... hard to tell but maybe the system was so overloaded that the JVM process almost froze (from swapping or so) and simply timed out.
I also can't find anything obviously wrong in the cluster logs (and the logging is set to globally debug here).

This is not the same failure as in other failing runs of this test that email threads reference. In those a non-polling ack assert failed, this one:

        assertAcked(client().admin().indices().prepareUpdateSettings("index").setSettings(Settings.builder()
                .put("index.routing.allocation.include._name", "")).get());

=> I'd close this and reopen if it comes up again, since a timeout in a test run that eventually ran into the OOM killer is really suspicious? :)

[original-brownbear]

The missing ack failure mode linked to in the initial issue description is happening here still though:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/162/consoleText

=> looking into this.

[alpar-t]

I observed this failure on a Centos 7 VM after ~ 1 hour of running internal cluster tests in a loop. Stack trace is different not sure if it's related. I haven't experienced slowness on CentOS in CI so far.

2> REPRODUCE WITH: ./gradlew :server:integTest -Dtests.seed=BD345D2CF064C409 -Dtests.class=org.elasticsearch.indices.state.RareClusterStateIT -Dtests.method="testDelayedMappingPropagationOnReplica" -Dtests.security.manager=true -Dtests.locale=br-FR -Dtests.timezone=Asia/Jerusalem -Dcompiler.java=11 -Druntime.java=11
FAILURE 3.55s J2 | RareClusterStateIT.testDelayedMappingPropagationOnReplica <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: AcknowledgedResponse failed - not acked
   > Expected: <true>
   >      but: was <false>
   >    at __randomizedtesting.SeedInfo.seed([BD345D2CF064C409:C14A9CE3A083FFEA]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:112)
   >    at org.elasticsearch.indices.state.RareClusterStateIT.testDelayedMappingPropagationOnReplica(RareClusterStateIT.java:340)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >    at java.base/java.lang.Thread.run(Thread.java:834)

[alpar-t]

Here's another recent repro for the one in the description: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/1209/

[pgomulka]

another failure. does not reproduce for me locally as well
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/131/console

./gradlew :server:integTest \
  -Dtests.seed=37FF28E573A0046A \
  -Dtests.class=org.elasticsearch.indices.state.RareClusterStateIT \
  -Dtests.method="testDelayedMappingPropagationOnReplica" \
  -Dtests.security.manager=true \
  -Dtests.locale=ga \
  -Dtests.timezone=Atlantic/Faeroe \
  -Dcompiler.java=11 \
  -Druntime.java=8