Logstash Filebeat module

CHANGELOG.asciidoc

@@ -101,6 +101,7 @@ https://github.com/elastic/beats/compare/v6.0.0-beta2...master[Check the HEAD di
 - Add experimental Docker `json-file` prospector . {pull}5402[5402]
 - Add experimental Docker autodiscover functionality. {pull}5245[5245]
 - Add option to convert the timestamps to UTC in the system module. {pull}5647[5647]
+- Add Logstash module support for main log and the slow log, support the plain text or structured JSON format {pull}5481[5481]
 
 *Heartbeat*
 

filebeat/docs/fields.asciidoc

@@ -21,6 +21,7 @@ grouped in the following categories:
 * <<exported-fields-kafka>>
 * <<exported-fields-kubernetes-processor>>
 * <<exported-fields-log>>
+* <<exported-fields-logstash>>
 * <<exported-fields-mysql>>
 * <<exported-fields-nginx>>
 * <<exported-fields-postgresql>>
@@ -970,6 +971,161 @@ The Filebeat module that generated this event.
 The Filebeat fileset that generated this event.
 
 
+[[exported-fields-logstash]]
+== logstash fields
+
+logstash Module
+
+
+
+[float]
+== logstash fields
+
+
+
+
+[float]
+== log fields
+
+Fields from the Logstash logs.
+
+
+
+[float]
+=== `logstash.log.message`
+
+type: text
+
+Contains the un-parsed log message
+
+
+[float]
+=== `logstash.log.level`
+
+type: keyword
+
+The log level of the message, this correspond to Log4j levels.
+
+
+[float]
+=== `logstash.log.module`
+
+type: keyword
+
+The module or class where the event originate.
+
+
+[float]
+=== `logstash.log.thread`
+
+type: text
+
+Information about the running thread where the log originate.
+
+
+[float]
+=== `logstash.log.log_event`
+
+type: object
+
+key and value debugging information.
+
+
+[float]
+== slowlog fields
+
+slowlog
+
+
+
+[float]
+=== `logstash.slowlog.message`
+
+type: text
+
+Contains the un-parsed log message
+
+
+[float]
+=== `logstash.slowlog.level`
+
+type: keyword
+
+The log level of the message, this correspond to Log4j levels.
+
+
+[float]
+=== `logstash.slowlog.module`
+
+type: keyword
+
+The module or class where the event originate.
+
+
+[float]
+=== `logstash.slowlog.thread`
+
+type: text
+
+Information about the running thread where the log originate.
+
+
+[float]
+=== `logstash.slowlog.event`
+
+type: text
+
+Raw dump of the original event
+
+
+[float]
+=== `logstash.slowlog.plugin_name`
+
+type: keyword
+
+Name of the plugin
+
+
+[float]
+=== `logstash.slowlog.plugin_type`
+
+type: keyword
+
+Type of the plugin: Inputs, Filters, Outputs or Codecs.
+
+
+[float]
+=== `logstash.slowlog.took_in_millis`
+
+type: long
+
+Execution time for the plugin in milliseconds.
+
+
+[float]
+=== `logstash.slowlog.took_in_nanos`
+
+type: long
+
+Execution time for the plugin in nanoseconds.
+
+
+[float]
+=== `logstash.slowlog.plugin_params`
+
+type: text
+
+String value of the plugin configuration
+
+
+[float]
+=== `logstash.slowlog.plugin_params_object`
+
+type: object
+
+key -> value of the configuration used by the plugin.
+
+
 [[exported-fields-mysql]]
 == MySQL fields
 

filebeat/docs/modules/logstash.asciidoc

@@ -0,0 +1,98 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-logstash]]
+:modulename: logstash
+== logstash module
+
+The +{modulename}+ module parse logstash regular logs and the slow log, it will support the plain text format
+and the JSON format (--log.format json). The default is the plain text format.
+
+include::../include/what-happens.asciidoc[]
+
+The +{modulename}+ module has two filesets:
+
+* The `log` fileset collects and parses the logs that logstash writes to disk.
+* The `slowlog` fileset parses the logstash slowlog.
+
+
+For the `slowlog` fileset, make sure to configure logstash slowlog option
+link:https://www.elastic.co/guide/en/logstash/current/logging.html#_slowlog
+
+[float]
+=== Compatibility
+
+The Logstash `log` fileset was tested with logs from Logstash 5.6 and 6.0.
+
+The Logstash `slowlog` fileset was tested with logs from Logstash 5.6 and 6.0
+
+include::../include/running-modules.asciidoc[]
+
+[float]
+===  This module comes with a two sample dashboard
+
+[role="screenshot"]
+image::./images/kibana-logstash-log.png[]
+
+image::./images/kibana-logstash-slowlog.png[]
+
+include::../include/configuring-intro.asciidoc[]
+
+The following example shows how to set paths in the +modules.d/{modulename}.yml+
+file to override the default paths for Logstash logs and set the format to json
+
+["source","yaml",subs="attributes"]
+-----
+- module: logstash
+  log:
+    enabled: true
+    var.paths: ["/path/to/log/logstash.log*"]
+    format: json
+  slowlog:
+    enabled: true
+    var.paths: ["/path/to/log/logstash-slowlog.log*"]
+    format: json
+-----
+
+To specify the same settings at the command line, you use:
+
+["source","sh",subs="attributes"]
+-----
+./{beatname_lc} -M "logstash.log.var.paths=[/path/to/log/logstash/logstash-server.log*]" -M "logstash.slowlog.var.paths=[/path/to/log/logstash/logstash-slowlog.log*]"
+-----
+
+
+The command in the example assumes that you have already enabled the +{modulename}+ module.
+
+//set the fileset name used in the included example
+:fileset_ex: log
+
+include::../include/config-option-intro.asciidoc[]
+
+
+[float]
+==== `log` fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+*`var.format`*::
+
+The configured Logstash log format, possible values are: `json` or `plain`, by default it will use the `plain` format.
+
+[float]
+==== `slowlog` fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+*`var.format`*::
+
+The configured Logstash log format, possible values are: `json` or `plain`, by default it will use the `plain` format.
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-logstash,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -7,6 +7,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-auditd>>
   * <<filebeat-module-icinga>>
   * <<filebeat-module-kafka>>
+  * <<filebeat-module-logstash>>
   * <<filebeat-module-mysql>>
   * <<filebeat-module-nginx>>
   * <<filebeat-module-postgresql>>
@@ -22,6 +23,7 @@ include::modules/apache2.asciidoc[]
 include::modules/auditd.asciidoc[]
 include::modules/icinga.asciidoc[]
 include::modules/kafka.asciidoc[]
+include::modules/logstash.asciidoc[]
 include::modules/mysql.asciidoc[]
 include::modules/nginx.asciidoc[]
 include::modules/postgresql.asciidoc[]

filebeat/filebeat.reference.yml

@@ -135,6 +135,23 @@ filebeat.modules:
     #var.paths:
 
 
+#------------------------------ logstash Module ------------------------------
+#- module: logstash
+  # logs
+  #log:
+    #enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    # var.paths:
+
+  # Slow logs
+  #slowlog:
+   #enabled: true
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
 #-------------------------------- MySQL Module -------------------------------
 #- module: mysql
   # Error logs

filebeat/module/logstash/_meta/config.reference.yml

@@ -0,0 +1,15 @@
+#- module: logstash
+  # logs
+  #log:
+    #enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    # var.paths:
+
+  # Slow logs
+  #slowlog:
+   #enabled: true
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:

filebeat/module/logstash/_meta/config.yml

@@ -0,0 +1,15 @@
+- module: logstash
+  # logs
+  log:
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
+  # Slow logs
+  slowlog:
+   enabled: true
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths: