Filebeat: osquery module

filebeat/docs/fields.asciidoc

@@ -24,6 +24,7 @@ grouped in the following categories:
 * <<exported-fields-logstash>>
 * <<exported-fields-mysql>>
 * <<exported-fields-nginx>>
+* <<exported-fields-osquery>>
 * <<exported-fields-postgresql>>
 * <<exported-fields-redis>>
 * <<exported-fields-system>>
@@ -1535,6 +1536,64 @@ type: text
 The error message
 
 
+[[exported-fields-osquery]]
+== Osquery fields
+
+Fields exported by the `osquery` module
+
+
+
+[float]
+== osquery fields
+
+
+
+
+[float]
+== result fields
+
+Common fields exported by the result metricset.
+
+
+
+[float]
+=== `osquery.result.name`
+
+type: keyword
+
+The name of the query that generated this event.
+
+
+[float]
+=== `osquery.result.action`
+
+type: keyword
+
+For incremental data, marks whether the entry was added or removed. It can be one of "added", "removed", or "snapshot".
+
+
+[float]
+=== `osquery.result.host_identifier`
+
+type: keyword
+
+The identifier for the host on which the osquery agent is running. Normally the hostname.
+
+
+[float]
+=== `osquery.result.unix_time`
+
+type: long
+
+Unix timestamp of the event, in seconds since the epoch. Used for computing the `@timestamp` column.
+
+
+[float]
+=== `osquery.result.calendar_time`
+
+String representation of the collection time, as formatted by osquery.
+
+
 [[exported-fields-postgresql]]
 == PostgreSQL fields
 

filebeat/docs/modules/osquery.asciidoc

@@ -0,0 +1,80 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-osquery]]
+:modulename: osquery
+
+== Osquery module
+
+The +{modulename}+ module collects and decodes the result logs written by
+https://osquery.readthedocs.io/en/latest/introduction/using-osqueryd/[osqueryd]
+in the JSON format. To set up osqueryd follow the osquery installation
+instructions for your operating system and configure the `filesystem` logging
+driver (the default). Make sure UTC timestamps are enabled.
+
+include::../include/what-happens.asciidoc[]
+
+
+[float]
+=== Compatibility
+
+The  +{modulename}+ module was tested with logs from osquery version 2.10.2.
+Since the results are written in the JSON format, it is likely that this module
+works with any version of osquery.
+
+This module is available on Linux, macOS, and Windows.
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard for visualizing the data collected by
+the "compliance" pack. To collect this data, enable the `id-compliance` pack in
+the osquery configuration file.
+
+[role="screenshot"]
+image::./images/kibana-osquery-compatibility.png[]
+
+include::../include/configuring-intro.asciidoc[]
+
+The following example shows how to set paths in the +modules.d/{modulename}.yml+
+file to override the default paths for the syslog and authorization logs:
+
+["source","yaml",subs="attributes"]
+-----
+- module: osquery
+  result:
+    enabled: true
+    var.paths: ["/path/to/osqueryd.results.log*"]
+-----
+
+
+To specify the same settings at the command line, you use:
+
+["source","sh",subs="attributes"]
+-----
+./{beatname_lc} --modules {modulename} -M "osquery.result.var.paths=[/path/to/osqueryd.results.log*]"
+-----
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `result` fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+*`var.use_namespace`*::
+
+If true, all fields exported by this module are prefixed with `osquery.result`.
+Set to false to copy the fields in the root of the document. If enabled, this
+setting also disables the renaming of some fields (e.g. `hostIdentifier` to
+`host_identifier`).  Note that if you set this to false, the sample dashboards
+coming with this module won't work correctly. The default is true.
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-osquery,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -10,6 +10,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-logstash>>
   * <<filebeat-module-mysql>>
   * <<filebeat-module-nginx>>
+  * <<filebeat-module-osquery>>
   * <<filebeat-module-postgresql>>
   * <<filebeat-module-redis>>
   * <<filebeat-module-system>>
@@ -26,6 +27,7 @@ include::modules/kafka.asciidoc[]
 include::modules/logstash.asciidoc[]
 include::modules/mysql.asciidoc[]
 include::modules/nginx.asciidoc[]
+include::modules/osquery.asciidoc[]
 include::modules/postgresql.asciidoc[]
 include::modules/redis.asciidoc[]
 include::modules/system.asciidoc[]

filebeat/filebeat.reference.yml

@@ -204,6 +204,20 @@ filebeat.modules:
     # can be added under this section.
     #prospector:
 
+#------------------------------- Osquery Module ------------------------------
+- module: osquery
+  result:
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
+    # If true, all fields created by this module are prefixed with
+    # `osquery.result`. Set to false to copy the fields in the root
+    # of the document. The default is true.
+    #var.use_namespace: true
+
 #----------------------------- PostgreSQL Module -----------------------------
 #- module: postgresql
   # Logs

filebeat/module/kafka/log/ingest/pipeline.json

@@ -45,7 +45,7 @@
     {
       "rename": {
         "field": "@timestamp",
-        "target_field": "beat.read_time"
+        "target_field": "read_timestamp"
       }
     },
     {

filebeat/module/osquery/_meta/config.reference.yml

@@ -0,0 +1,12 @@
+- module: osquery
+  result:
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
+    # If true, all fields created by this module are prefixed with
+    # `osquery.result`. Set to false to copy the fields in the root
+    # of the document. The default is true.
+    #var.use_namespace: true

filebeat/module/osquery/_meta/config.yml

@@ -0,0 +1,12 @@
+- module: osquery
+  result:
+    enabled: true
+
+    # Set custom paths for the log files. If left empty,
+    # Filebeat will choose the paths depending on your OS.
+    #var.paths:
+
+    # If true, all fields created by this module are prefixed with
+    # `osquery.result`. Set to false to copy the fields in the root
+    # of the document. The default is true.
+    #var.use_namespace: true

filebeat/module/osquery/_meta/docs.asciidoc

@@ -0,0 +1,67 @@
+:modulename: osquery
+
+== Osquery module
+
+The +{modulename}+ module collects and decodes the result logs written by
+https://osquery.readthedocs.io/en/latest/introduction/using-osqueryd/[osqueryd]
+in the JSON format. To set up osqueryd follow the osquery installation
+instructions for your operating system and configure the `filesystem` logging
+driver (the default). Make sure UTC timestamps are enabled.
+
+include::../include/what-happens.asciidoc[]
+
+
+[float]
+=== Compatibility
+
+The  +{modulename}+ module was tested with logs from osquery version 2.10.2.
+Since the results are written in the JSON format, it is likely that this module
+works with any version of osquery.
+
+This module is available on Linux, macOS, and Windows.
+
+[float]
+=== Example dashboard
+
+This module comes with a sample dashboard for visualizing the data collected by
+the "compliance" pack. To collect this data, enable the `id-compliance` pack in
+the osquery configuration file.
+
+[role="screenshot"]
+image::./images/kibana-osquery-compatibility.png[]
+
+include::../include/configuring-intro.asciidoc[]
+
+The following example shows how to set paths in the +modules.d/{modulename}.yml+
+file to override the default paths for the syslog and authorization logs:
+
+["source","yaml",subs="attributes"]
+-----
+- module: osquery
+  result:
+    enabled: true
+    var.paths: ["/path/to/osqueryd.results.log*"]
+-----
+
+
+To specify the same settings at the command line, you use:
+
+["source","sh",subs="attributes"]
+-----
+./{beatname_lc} --modules {modulename} -M "osquery.result.var.paths=[/path/to/osqueryd.results.log*]"
+-----
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `result` fileset settings
+
+include::../include/var-paths.asciidoc[]
+
+*`var.use_namespace`*::
+
+If true, all fields exported by this module are prefixed with `osquery.result`.
+Set to false to copy the fields in the root of the document. If enabled, this
+setting also disables the renaming of some fields (e.g. `hostIdentifier` to
+`host_identifier`).  Note that if you set this to false, the sample dashboards
+coming with this module won't work correctly. The default is true.

filebeat/module/osquery/_meta/fields.yml

@@ -0,0 +1,9 @@
+- key: osquery
+  title: "Osquery"
+  description: >
+    Fields exported by the `osquery` module
+  fields:
+    - name: osquery
+      type: group
+      description: >
+      fields: