ClusterDisruptionIT#testCannotJoinIfMasterLostDataFolder can fail with assertion error.

[jtibshirani]

Unfortunately the failure doesn't reproduce for me locally. I wasn't able to find other instances of this failure in past CI runs.

---

Link to the build:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu-14.04&&immutable/324/console

Command to reproduce:

./gradlew :server:integTest --tests "org.elasticsearch.discovery.ClusterDisruptionIT.testCannotJoinIfMasterLostDataFolder" \
  -Dtests.seed=53F47615B7AB0A88 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-ZA \
  -Dtests.timezone=Pacific/Pago_Pago \
  -Dcompiler.java=12 \
  -Druntime.java=11

Relevant excerpt from the logs:

  2> REPRODUCE WITH: ./gradlew :server:integTest --tests "org.elasticsearch.discovery.ClusterDisruptionIT.testCannotJoinIfMasterLostDataFolder" -Dtests.seed=53F47615B7AB0A88 -Dtests.security.manager=true -Dtests.locale=en-ZA -Dtests.timezone=Pacific/Pago_Pago -Dcompiler.java=12 -Druntime.java=11
  2> java.lang.AssertionError
        at org.junit.Assert.fail(Assert.java:86)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.junit.Assert.assertTrue(Assert.java:52)
        at org.elasticsearch.discovery.ClusterDisruptionIT.testCannotJoinIfMasterLostDataFolder(ClusterDisruptionIT.java:411)

    MasterNotDiscoveredException[null]

[elasticmachine]

Pinging @elastic/es-distributed

[ywelsch]

What happens here is that there is a master node and a data-only node forming a cluster. The master is then restarted and its data folder wiped (and re-bootstrapped), which means that after the restart, the master should form a cluster by itself without the data node and should not let the data node join because of cluster uuid join validation. Unfortunately the condition internalCluster().client(masterNode).admin().cluster().prepareHealth().setWaitForNodes("2").get() successfully returns, because the data node joins while the master is still not fully established as master just yet and bypasses the join validation that is done before adding the node to the cluster. Only the join validation when handling the publish request takes place, but at this point the cluster state has been successfully committed, and will subsequently be exposed to the applier. The health check therefore sees the cluster state with the 2 nodes and thinks all is good, even though the data node never accepted this state. What's worse is that it takes 90 seconds for the data node to be kicked out of the cluster (lag detection).

Logs analyzed: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu-14.04&&immutable/324/consoleText

We could fix the test by first taking down the data-only node, then restart and wipe the master node, and bringing the data node back up after the master has successfully formed its 1-node cluster. I don't think we should complicate the join validation logic to handle this case differently, given how rare it might happen. WDYT @DaveCTurner ?

[DaveCTurner]

I agree that this doesn't indicate an issue in production code, because it should resolve itself eventually (after 90 seconds). However I think it'd be good to check that it does indeed resolve itself eventually even if we get into this state. Can we reduce cluster.follower_lag.timeout to a shorter time and use assertBusy() to wait for the lag detector to take effect?

[ywelsch]

sure, do you want to make that change?

[ywelsch]

I've pushed 250973a as a fix