Elastic Search should terminate & restart gracefully

[omgitsads]

From one of our customers John Paul:

check process elasticsearch_9080 
  with pidfile /var/run/elasticsearch.pid
  start program = "/usr/lib/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch.pid" with timeout 90 seconds 
  stop program = "/bin/bash -c '/bin/kill `cat /var/run/elasticsearch.pid`'" with timeout 90 seconds 
  group elasticsearch

kill is just program sends a TERM signal to a process by default. it does not guarantee the shutdown process, so after a Kill, Monit thinks that ES is off. but > in reality it is not so!

Next, monit starts a new process elasticsearch. Together with the old process, they form a cluster. New shard begins to process and download the data from the old shard, but do not have time, as the old process shutdowned by kill.

This can cause vanishing indexes

This situation can be avoided by using standard upstart or systemd as supervisor of demons.

[dvalfre]

@radamanthus in the light of all the changes done to the ElasticSearch recipe, is the issue still relevant?