Kafka output LZ4 compression support

[weichuliu]

What

Currently Kafka officially supports LZ4 as compression method. (See producer configs)

It is already supported by Sarama. As I checked beats' vendored code on master, it is already there.

However, filebeat (on version 6.0.0-alpha2) itself doesn't support lz4 compression. It would throw the following error:

2017/07/10 21:08:48.985462 outputs.go:101: ERR failed to initialize kafka plugin as output: compression mode 'lz4' unknown accessing 'output.kafka' (source:'filebeat/lz4_test.yml')
2017/07/10 21:08:48.985477 beat.go:539: CRIT Exiting: error initializing publisher: compression mode 'lz4' unknown accessing 'output.kafka' (source:'filebeat/lz4_test.yml')
Exiting: error initializing publisher: compression mode 'lz4' unknown accessing 'output.kafka' (source:'filebeat/lz4_test.yml')

Why we want LZ4

Since we are using filebeat to forward a large amount of logs to our Kafka, using gzip is very CPU consuming on broker side. On the other hand, Kafka's storage with snappy is poorly implememted. (As tested by myself, on 0.10.x, snappy uses 70% disk as uncompressed while gzip uses 10%. This is improved on 0.11.0 but it still uses 35% as uncompressed.)

From this kafka compression benchmark, and this stackoverflow post LZ4 has good performance, along with very high disk effeciency. That's why we would really want to have beats supporting LZ4.