diff --git a/content/chapter-correlating-log-messages/grouping-by-parser/_index.md b/content/chapter-correlating-log-messages/grouping-by-parser/_index.md index 85735634..333a8aa5 100644 --- a/content/chapter-correlating-log-messages/grouping-by-parser/_index.md +++ b/content/chapter-correlating-log-messages/grouping-by-parser/_index.md @@ -74,6 +74,3 @@ These lines are connected by their second field: `msg=audit(1440927434.124:40347 ); }; ``` - -For another example, see [The grouping-by() parser blog post](https://www.syslog-ng.com/community/b/blog/posts/the-grouping-by-parser-in-syslog-ng-3-8) - diff --git a/content/chapter-destinations/configuring-destinations-graphite/_index.md b/content/chapter-destinations/configuring-destinations-graphite/_index.md index 83f3868e..94c6d576 100644 --- a/content/chapter-destinations/configuring-destinations-graphite/_index.md +++ b/content/chapter-destinations/configuring-destinations-graphite/_index.md @@ -8,9 +8,6 @@ short_description: "Send metrics to Graphite" The `graphite()` destination can send metrics to a [Graphite](http://graphite.readthedocs.io/en/latest/index.html) server to store numeric time-series data. There are many ways to feed the Graphite template function with name value pairs. The {{% param "product.abbrev" %}} CSV and PatternDB parsers (for details, see {{% xref "/chapter-parsers/chapter-patterndb/reference-parsers-pattern-databases/syslog-ng-patterndb-using-parsers/_index.md" %}}) can parse log messages and generate name value pairs based on message content. The CSV parser (for details, see {{% xref "/chapter-parsers/csv-parser/_index.md" %}}) can be used for logs that have a constant field based structure, like the Apache web server access logs. The [patterndb() parser] can parse information and can extract important fields from free form log messages, as long as patterns describing the log messages are available. Another way is to send JSON-based log messages (for details, see {{% xref "/chapter-parsers/json-parser/_index.md" %}}) to {{% param "product.abbrev" %}}, like running a simple shell script collecting metrics and running it from cron regularly. -To see an example of how the `graphite()` destination is used to collect statistics coming from `syslog-ng`, see the blog post [Collecting syslog-ng statistics to Graphite](https://syslog-ng.com/blog/collecting-syslog-ng-statistics-to-graphite/). - - ## Declaration: ```shell @@ -21,7 +18,7 @@ To see an example of how the `graphite()` destination is used to collect statist ## Example: Using the graphite() driver {#example-destination-graphite} -To use the graphite() destination, the only mandatory parameter is payload, which specifies the value pairs to send to graphite. In the following example any value pairs starting with "monitor." are forwarded to graphite. +To use the `graphite()` destination, the only mandatory parameter is payload, which specifies the value pairs to send to Graphite. In the following example any value pairs starting with `"monitor."` are forwarded to Graphite. ```shell destination d_graphite { graphite(payload("--key monitor.*")); }; diff --git a/content/chapter-destinations/python-destination/_index.md b/content/chapter-destinations/python-destination/_index.md index f233b962..c374a1e0 100644 --- a/content/chapter-destinations/python-destination/_index.md +++ b/content/chapter-destinations/python-destination/_index.md @@ -229,8 +229,6 @@ The following code is similar to the previous example, but it opens and closes t }; ``` -For a more detailed example about sending log messages to an MQTT (Message Queuing Telemetry Transport) server, see the [Writing Python destination in syslog-ng: how to send log messages to MQTT blog post](https://www.syslog-ng.com/community/b/blog/posts/writing-python-destination-in-syslog-ng-how-to-send-log-messages-to-mqtt). - ## Example: Print logs in batch mode {#python-destination-batch-mode-example} The following is a simple destination that uses the `flush()` method to print the messages in batch mode. diff --git a/content/chapter-sources/syslog-ng-source-osquery/_index.md b/content/chapter-sources/syslog-ng-source-osquery/_index.md index 57b210c4..f31d217a 100644 --- a/content/chapter-sources/syslog-ng-source-osquery/_index.md +++ b/content/chapter-sources/syslog-ng-source-osquery/_index.md @@ -8,13 +8,11 @@ short_description: "Collect and parse osquery result logs" The [osquery](https://osquery.io) application allows you to ask questions about your machine using an SQL-like language. For example, you can query running processes, logged in users, installed packages and syslog messages as well. You can make queries on demand, and also schedule them to run regularly. -The `osquery()` source of {{% param "product.abbrev" %}} allows you read the results of periodical osquery queries (from the `/var/log/osquery/osqueryd.results.log` file) and automatically parse the messages (if you want to use {{% param "product.abbrev" %}} to [send log messages to osquery, read this blogpost](https://syslog-ng.com/blog/endpoint-visibility-and-monitoring-using-osquery-and-syslog-ng/)). For example, you can: +The `osquery()` source of {{% param "product.abbrev" %}} allows you read the results of periodical osquery queries (from the `/var/log/osquery/osqueryd.results.log` file) and automatically parse the messages. For example, you can: - - Create filters from the fields of the messages. - - - Limit which fields to store, or create additional fields (combine multiple fields into one field, and so on). - - - Send the messages to a central location, for example, to Elasticsearch, directly from {{% param "product.abbrev" %}}. +- Create filters from the fields of the messages. +- Limit which fields to store, or create additional fields (combine multiple fields into one field, and so on). +- Send the messages to a central location, for example, to Elasticsearch, directly from {{% param "product.abbrev" %}}. The {{% param "product.abbrev" %}} application automatically adds the `.osquery.` prefix to the name of the fields the extracted from the message. @@ -23,14 +21,13 @@ The `osquery()` source is available in {{% param "product.abbrev" %}} version 3. ## Prerequisites: - - To use the `osquery()` driver, the `scl.conf` file must be included in your {{% param "product.abbrev" %}} configuration: - +- To use the `osquery()` driver, the `scl.conf` file must be included in your {{% param "product.abbrev" %}} configuration: + ```shell - @include "scl.conf" - + @include "scl.conf" ``` - - {{% param "product.abbrev" %}} must be compiled with JSON-support enabled. +- {{% param "product.abbrev" %}} must be compiled with JSON-support enabled. The `osquery()` driver is actually a reusable configuration snippet configured to read the osquery log file using the `file()` driver, and process its JSON contents. For details on using or writing such configuration snippets, see {{% xref "/chapter-configuration-file/large-configs/config-blocks/_index.md" %}}. You can find the source of this configuration snippet on [GitHub](https://github.com/axoflow/axosyslog/blob/master/scl/osquery/plugin.conf). @@ -41,83 +38,74 @@ The `osquery()` driver is actually a reusable configuration snippet configured t The following {{% param "product.abbrev" %}} configuration sample uses the default settings of the driver, reading osquery result logs from the `/var/log/osquery/osqueryd.results.log` file, and writes the log messages generated from the traps into a file. ```shell - @version: 3.10 - @include "scl.conf" - source s_osquery { - osquery(); - }; - log { - source(s_osquery); - destination { - file("/var/log/example.log"); - }; +@version: 3.10 +@include "scl.conf" +source s_osquery { + osquery(); +}; +log { + source(s_osquery); + destination { + file("/var/log/example.log"); }; +}; ``` Filter for messages related to loading Linux kernel modules: ```shell - @version: 3.10 - @include "scl.conf" - source s_osquery { - osquery(); +@version: 3.10 +@include "scl.conf" +source s_osquery { + osquery(); +}; +log { + source(s_osquery); + filter f_modules { + "${.osquery.name}" eq "pack_incident-response_kernel_modules" }; - log { - source(s_osquery); - filter f_modules { - "${.osquery.name}" eq "pack_incident-response_kernel_modules" - }; - destination { - file("/var/log/example.log"); - }; + destination { + file("/var/log/example.log"); }; +}; ``` - - ## Example: Using the osquery() driver with custom configuration The following {{% param "product.abbrev" %}} configuration sample reads osquery result logs from the `/tmp/osquery_input.log` file, and writes the log messages generated from the traps into a file. Using the `format-json` template, the outgoing message will be a well-formed JSON message. - -## Input message: +### Input message ```shell {"name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","calendarTime":"Fri Jul 21 10:04:41 2017 UTC","unixTime":"1500631481","decorations":{"host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","username":"myuser"},"columns":{"build_distro":"xenial","build_platform":"ubuntu","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","config_valid":"1","counter":"4","extensions":"active","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","pid":"19764","resident_size":"26416000","start_time":"1500629552","system_time":"223","user_time":"476","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","version":"2.5.0","watcher":"19762"},"action":"added"} ``` - - -## {{% param "product.abbrev" %}} configuration: +### {{% param "product.abbrev" %}} configuration ```shell - @version: 3.10 - @include "scl.conf" - source s_osquery { - osquery( - file(/tmp/osquery_input.log) - prefix(.osquery.) - ); - }; - destination d_file { - file( - "/tmp/output.txt" - template("$(format_json --key .osquery.*)\n") - ); - }; - log { - source(s_osquery); - destination(d_file); - flags(flow-control); - }; +@version: 3.10 +@include "scl.conf" +source s_osquery { + osquery( + file(/tmp/osquery_input.log) + prefix(.osquery.) + ); +}; +destination d_file { + file( + "/tmp/output.txt" + template("$(format_json --key .osquery.*)\n") + ); +}; +log { + source(s_osquery); + destination(d_file); + flags(flow-control); +}; ``` -## Outgoing message: +### Outgoing message ```shell - Outgoing message; message='{"_osquery":{"unixTime":"1500631481","name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"},"columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","user_time":"476","system_time":"223","start_time":"1500629552","resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","extensions":"active","counter":"4","config_valid":"1","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu","build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a' +Outgoing message; message='{"_osquery":{"unixTime":"1500631481","name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"},"columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","user_time":"476","system_time":"223","start_time":"1500629552","resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","extensions":"active","counter":"4","config_valid":"1","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu","build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a' ``` - - - -To configure a destination to send the log messages to Elasticsearch, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch2/_index.md" %}}. For other destinations, see {{% xref "/chapter-destinations/_index.md" %}}. diff --git a/content/chapter-troubleshooting-syslog-ng/topic-troubleshooting-unusual-directory/_index.md b/content/chapter-troubleshooting-syslog-ng/topic-troubleshooting-unusual-directory/_index.md deleted file mode 100644 index da05e288..00000000 --- a/content/chapter-troubleshooting-syslog-ng/topic-troubleshooting-unusual-directory/_index.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: "No local logs after specifying an unusual storage directory" -weight: 1500 ---- - - -{{% include-headless "chunk/para-selinux.md" %}} - -When you choose to save logs of a central {{% param "product.abbrev" %}} server to a directory other than the `/var/log` directory, logs will not start appearing on the newly configured directory. For details on how to fix this issue, see section "Using a different storage directory" in the blog post titled [Using syslog-ng with SELinux in enforcing mode](https://syslog-ng.com/blog/using-syslog-ng-with-selinux-in-enforcing-mode/). - diff --git a/content/chapter-troubleshooting-syslog-ng/topic-troubleshooting-unusual-port-number/_index.md b/content/chapter-troubleshooting-syslog-ng/topic-troubleshooting-unusual-port-number/_index.md deleted file mode 100644 index 479ce783..00000000 --- a/content/chapter-troubleshooting-syslog-ng/topic-troubleshooting-unusual-port-number/_index.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: "No logs after specifying an unusual port number" -weight: 1700 ---- - - -{{% include-headless "chunk/para-selinux.md" %}} - -By default, SELinux only allows connections to the default syslog ports. When you have to use any other port for some reason, sending logs to that port will not work. For details on how to fix this issue, see section "Using a different port" in the blog post titled [Using syslog-ng with SELinux in enforcing mode](https://syslog-ng.com/blog/using-syslog-ng-with-selinux-in-enforcing-mode/). - diff --git a/content/headless/chunk/example-grouping-by-referencing-earlier-values.md b/content/headless/chunk/example-grouping-by-referencing-earlier-values.md index 4b9f7b23..cf6a90c4 100644 --- a/content/headless/chunk/example-grouping-by-referencing-earlier-values.md +++ b/content/headless/chunk/example-grouping-by-referencing-earlier-values.md @@ -11,6 +11,3 @@ The following action can be used to log the length of an SSH session (the time d value('value name="MESSAGE" An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}') ) ``` - -For another example, see [The grouping-by() parser in syslog-ng blog post](https://www.syslog-ng.com/community/b/blog/posts/the-grouping-by-parser-in-syslog-ng-3-8) - diff --git a/content/headless/chunk/topic-troubleshooting-unusual-directory.md b/content/headless/chunk/topic-troubleshooting-unusual-directory.md deleted file mode 100644 index 1f731633..00000000 --- a/content/headless/chunk/topic-troubleshooting-unusual-directory.md +++ /dev/null @@ -1,10 +0,0 @@ ---- ---- - - -# No local logs after specifying an unusual storage directory - -{{% include-headless "chunk/para-selinux.md" %}} - -When you choose to save logs of a central {{% param "product.abbrev" %}} server to a directory other than the `/var/log` directory, logs will not start appearing on the newly configured directory. For details on how to fix this issue, see sectionUsing a different storage directoryin the blog post titled [Using syslog-ng with SELinux in enforcing mode](https://syslog-ng.com/blog/using-syslog-ng-with-selinux-in-enforcing-mode/). - diff --git a/content/headless/chunk/topic-troubleshooting-unusual-port-number.md b/content/headless/chunk/topic-troubleshooting-unusual-port-number.md deleted file mode 100644 index 5d3a7a9f..00000000 --- a/content/headless/chunk/topic-troubleshooting-unusual-port-number.md +++ /dev/null @@ -1,10 +0,0 @@ ---- ---- - - -# No logs after specifying an unusual port number - -{{% include-headless "chunk/para-selinux.md" %}} - -By default, SELinux only allows connections to the default syslog ports. When you have to use any other port for some reason, sending logs to that port will not work. For details on how to fix this issue, see sectionUsing a different portin the blog post titled [Using syslog-ng with SELinux in enforcing mode](https://syslog-ng.com/blog/using-syslog-ng-with-selinux-in-enforcing-mode/). -