Modernize ELK example with latest versions of everything

[geerlingguy]

There are a few bits of the ELK stack that have radically changed in the past year—Kibana 4 is an entirely different beast, and Logstash Forwarder is now Filebeat... and the entire architecture seems to have evolved quite a bit.

I just finished porting the Kibana role to 4, in the 2.0.0 version (see: geerlingguy/ansible-role-kibana#5), but I pinned this repo's ELK example to the role's 1.0.8 version, which installs Kibana 3...

Need to test out some other bits and pieces, also rename things like logs server to something more logical (see closed issue #25).

[geerlingguy]

Some debugging tricks:

• Check ES index status: curl http://localhost:9200/_cat/indices?v
• Check ES cluster status: curl -XGET http://localhost:9200/_cluster/health?pretty
• Start Filebeat to monitor: sudo filebeat -e -v -c /etc/filebeat/filebeat.yml (make sure to stop filebeat service first)
• Start Logstash to monitor: sudo /opt/logstash/bin/logstash --verbose -f /etc/logstash/conf.d/ (or --debug instead of --verbose if you wanna go crazy)

I'm almost there, but am hitting a quirk where the filebeat index generated through filebeat->logstash->ES is stuck in red status, and I'm not sure why. It was working yesterday, go figure.

[geerlingguy]

Committed what I had so far.

Everything works... but when it all boots up, I have to first delete the auto-generated filebeats-[date] index in ES with curl -XDELETE 'http://localhost:9200/[index-name-here]/', then restart logstash and filebeat, and everything starts flowing through.

For some reason, on first start, the index created for filebeat through logstash gets stuck in a red/uninitialized state :/

[geerlingguy]

Moving this to geerlingguy/ansible-for-devops#196 for more thorough fixing.