Logstash Filebeat module

[ph]

This module add support for the differents logs files generated by
Logstash and add simple dashboard to display relevent information.

Depending on the configuration Logstash will generate two differents
log:

1.logstash-(plain|json).log: this log file will contains all the
normal log events generated by logstash.

2. logstash-slowlog-(plain|json).log: This log will contains events
   generated when the slow log feature is enabled in logstash and will contains
   the name of the plugin and the execution time. The recording of theses
   events are based on configurable threshold.

Also Logstash will output by default plain text log events but user can
change that to export structured JSON logs.

TODO

• Add screenshots for the dashboard.

[ph]

If you want to use Logstash's slowlog you have to turn it on in the configuration with something like this.

slowlog.threshold.warn: 2s
slowlog.threshold.info: 1s
slowlog.threshold.debug: 500ms
slowlog.threshold.trace: 100ms

You can use a sleep filter to actually get something slow :)

input {
    generator {}
}

filter {
    sleep {
        time => 3
    }
}

output {
    file {
        path => "/tmp/slow_events.log"
    }
}

[tsg]

This part of the docs is being refactored in #5341. It might make sense to update it already to the new style of module documentation.

[ph]

Thanks for the pointer! I will do that.

[tsg]

The JSON decoding could also be done on the Filebeat side, but I'm not sure if there's any advantage to that.

[ph]

My reasoning was, It's easier to to all the transformation at the same place, easier to tests/debug.

[tsg]

Nice 👍

[ruflin]

It's a bit strange that we call it slow logs in the comments but below it's error?

[ph]

oops, typo good catch :)

[ph]

I will do another complete run, are you volunteering to test it? 💃

[ruflin]

can do, but probably only get to it tomorrow.

[ph]

I have updated the PR, generating a bit more dummy data to create a dash board.

[ph]

@tsg Can you take a look at it?

[tsg]

I didn't see in the PR a module.yml file like this one. It just lists the dashboards which makes it easier to export them (see the -yml option from the export_dashboards program).

[tsg]

@ph This looks great! There seem to be some errors in the tests, though.

[ph]

re: modules.yml I might have followed an older doc, will add it easily, also checking for tests.

[ph]

@tsg I have updated the module.yml and created #5656 for the tests followup.

[ph]

fixing pep8.. ;)

[tsg]

@ph there seems to be still one more error:

06:46:41 ======================================================================
06:46:41 FAIL: Tests all filebeat modules
06:46:41 ----------------------------------------------------------------------
06:46:41 Traceback (most recent call last):
06:46:41   File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_modules.py", line 71, in test_modules
06:46:41     cfgfile=cfgfile)
06:46:41   File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_modules.py", line 133, in run_on_file
06:46:41     assert len(objects) > 0
06:46:41 AssertionError: 
06:46:41 -------------------- >> begin captured stdout << ---------------------