ElasticsearchNodeCommandIT#test3MasterNodes2Failed test failure

[talevy]

reproduction command.

note: this is flaky, it does not always fail, but seems to fail a few times in CI

./gradlew :server:integTest \
  -Dtests.seed=A09F6B07D84F6123 \
  -Dtests.class=org.elasticsearch.cluster.coordination.ElasticsearchNodeCommandIT \
  -Dtests.method="test3MasterNodes2Failed" \
  -Dtests.security.manager=true \
  -Dtests.locale=sr-Latn \
  -Dtests.timezone=America/Goose_Bay \
  -Dcompiler.java=11 \
  -Druntime.java=8

CI links:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/2158/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+internalClusterTest/656/console

relevant logs:

ElasticsearchException[no metadata is referenced from the manifest file, cluster has never been bootstrapped?]
	at __randomizedtesting.SeedInfo.seed([F768D758DE5A5A2A:B18278007512FA47]:0)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.loadMetaData(ElasticsearchNodeCommand.java:96)
	at org.elasticsearch.cluster.coordination.UnsafeBootstrapMasterCommand.processNodePaths(UnsafeBootstrapMasterCommand.java:98)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommand.processNodePathsWithLock(ElasticsearchNodeCommand.java:81)
	at org.elasticsearch.cluster.coordination.UnsafeBootstrapMasterCommand.execute(UnsafeBootstrapMasterCommand.java:83)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommandIT.executeCommand(ElasticsearchNodeCommandIT.java:70)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommandIT.unsafeBootstrap(ElasticsearchNodeCommandIT.java:79)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommandIT.unsafeBootstrap(ElasticsearchNodeCommandIT.java:93)
	at org.elasticsearch.cluster.coordination.ElasticsearchNodeCommandIT.test3MasterNodes2Failed(ElasticsearchNodeCommandIT.java:294)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)

attached logs: 20190220171036-41D8C99A-log-5527-851472915786149870.txt

[elasticmachine]

Pinging @elastic/es-distributed

[matriv]

Cannot manage to reproduce locally.

After some investigation and comparison of the logs of successful runs vs the logs of this failure, I see that for successful runs, when creating the index there are entries like:

[node_tm0] processing [ApplyCommitRequest{...

and

[node_tm0] received full cluster state version 

which are missing from the log when the failure occurs.