Merge pull request #81 from axoflow/filterx-4.9-updates

Filterx 4.9 updates

.github/workflows/ci.yaml

@@ -18,7 +18,7 @@ jobs:
       - name: Set up Hugo
         uses: peaceiris/actions-hugo@v2
         with:
-          hugo-version: '0.110.0'
+          hugo-version: '0.122.0'
           extended: true
 
       - name: Set up Node

.github/workflows/publish.yaml

@@ -38,7 +38,7 @@ jobs:
       - name: Set up Hugo
         uses: peaceiris/actions-hugo@v2
         with:
-          hugo-version: '0.119.0'
+          hugo-version: '0.122.0'
           extended: true
 
       - name: Set up Node

.github/workflows/staging.yaml

@@ -33,7 +33,7 @@ jobs:
       - name: Set up Hugo
         uses: peaceiris/actions-hugo@v2
         with:
-          hugo-version: '0.110.0'
+          hugo-version: '0.122.0'
           extended: true
 
       - name: Set up Node

content/chapter-parsers/metrics-probe/_index.md

@@ -8,7 +8,7 @@ Available in {{% param "product_name" %}} version 4.1.1 and newer.
 
 `metrics-probe()` is a special parser that counts the messages that pass through the log path, and creates labeled stats counters based on the fields of the passing messages.
 
-You can configure the name of the keys and the labels. Note that the keys are automatically prefixes with the `syslogng_` string. You can use templates in the values of the labels.
+You can configure the name of the keys and the labels. Note that the keys are automatically prefixed with the `syslogng_` string. You can use templates in the values of the labels.
 
 The minimal configuration creates counters with the key `syslogng_classified_events_total` and labels `app`, `host`, `program` and `source`. For example:
 

content/filterx/_index.md

@@ -64,6 +64,10 @@ FilterX statements can be one of the following:
 - Existence of a variable of field. For example, the `${HOST};` expression is true only if the `${HOST}` macro exists and isn't empty.
 - A conditional statement ( `if (expr) { ... } elif (expr) {} else { ... };`) which allows you to evaluate complex decision trees.
 - A declaration of a [pipeline variable](#variable-scope), for example, `declare my_pipeline_variable = "something";`.
+- A FilterX action. This can be one of the following:
+
+    - `drop;`: Intentionally drop the message. This means that the message was successfully processed, but discarded. Processing the dropped message stops at the `drop` statement, subsequent sections or other branches of the FilterX block won't process the message. For example, you can use this to discard unneeded messages, like debug logs. Available in {{< product >}} 4.9 and later.
+    - `done;`: Return truthy and don't execute the rest of the FilterX block, returns with true. This is an early return that you can use to avoid unnecessary processing, for example, when the message matches an early classification in the block. Available in {{< product >}} 4.9 and later.
 
 {{% alert title="Note" color="info" %}}
 
@@ -235,13 +239,9 @@ To unset every empty field of an object, use the [`unset-empties`]({{< relref "/
 
 {{< include-headless "chunk/filterx-unset-hard-macros.md" >}}
 
-## Concatenate strings
+## Add two values
 
-You can concatenate strings by adding them with the `+` operator. Note that if you want to have spaces between the added elements, you have to add them manually, like in Python, for example:
-
-```shell
-${MESSAGE} = ${HOST} + " first part of the message," + " second part of the message" + "\n";
-```
+{{< include-headless "chunk/filterx-plus-operator.md" >}}
 
 ## Complex types: lists, dicts, and JSON {#json}
 
@@ -331,11 +331,7 @@ Within a FilterX block, you can access the fields of complex data types by using
 
 When referring to the field of a name-value pair (which begins with the `$` character), place the dot or the square bracket outside the curly bracket surrounding the name of the name-value pair, for example: `${MY-LIST}[2]` or `${MY-OBJECT}.mykey`. If the name of the key contains characters that are not permitted in FilterX variable names, for example, a hyphen (`-`), use the bracketed syntax and enclose the key in double quotes: `${MY-LIST}["my-key-name"]`.
 
-<!-- FIXME Clarify when/what can be accessed from the destination templates
-    # Set the important elements as name-value pairs so they can be referenced in the destination template
- -->
-
-<!-- FIXME more examples for lists/dicts if needed -->
+You can add two lists or two dicts using the {{% xref "/filterx/operator-reference.md#plus-operator" %}}.
 
 <!--
 ### Type casting
@@ -352,7 +348,7 @@ FilterX has the following operators.
 - [Boolean operators]({{< relref "/filterx/filterx-boolean/_index.md" >}}): `not`, `or`, `and`.
 - [Dot operator (`.`)](#json) to access fields of an object, like JSON.
 - [Indexing operator `[]`](#json) to access fields of an object, like JSON.
-- [Plus (`+`) operator](#concatenate-strings) to concatenate strings.
+- [Plus (`+`) operator]({{< relref "/filterx/operator-reference.md#plus-operator" >}}) to add values and concatenate strings.
 - [Plus equal (`+=`) operator]({{< relref "/filterx/operator-reference.md#plus-equal-operator" >}}) to add the right operand to the left.
 - [Ternary conditional operator]({{< relref "/filterx/operator-reference.md#ternary-conditional-operator" >}}): `?:`.
 - [Null coalescing operator]({{< relref "/filterx/operator-reference.md#null-coalescing-operator" >}}): `??`.
@@ -365,23 +361,32 @@ For details, see {{% xref "/filterx/operator-reference.md" %}}.
 FilterX has the following built-in functions.
 
 - [`cache_json_file`]({{< relref "/filterx/function-reference.md#cache-json-file" >}}): Loads an external JSON file to lookup contextual information.
+- [`endswith`]({{< relref "/filterx/filterx-string-search/_index.md" >}}): Checks if a string ends with the specified value.
 - [`flatten`]({{< relref "/filterx/function-reference.md#flatten" >}}): Flattens the nested elements of an object.
 - [`format_csv`]({{< relref "/filterx/function-reference.md#format-csv" >}}): Formats a dictionary or a list into a comma-separated string.
 - [`format_json`]({{< relref "/filterx/function-reference.md#format-json" >}}): Dumps a JSON object into a string.
 - [`format_kv`]({{< relref "/filterx/function-reference.md#format-kv" >}}): Formats a dictionary into key=value pairs.
+- [`get_sdata`]({{< relref "/filterx/filterx-sdata/_index.md" >}}): Returns the SDATA part of an RFC5424-formatted syslog message as a JSON object.
+- [`has_sdata`]({{< relref "/filterx/filterx-sdata/_index.md" >}}): Checks if a string ends with the specified value.
+- [`includes`]({{< relref "/filterx/filterx-string-search/_index.md" >}}): Checks if a string contains a specific substring.
 - [`isodate`]({{< relref "/filterx/function-reference.md#isodate" >}}): Parses a string as a date in ISODATE format.
+- [`is_sdata_from_enterprise`]({{< relref "/filterx/filterx-sdata/_index.md" >}}): Checks if the message contains the specified organization ID.
 - [`isset`]({{< relref "/filterx/function-reference.md#isset" >}}): Checks that argument exists and its value is not empty or null.
 - [`istype`]({{< relref "/filterx/function-reference.md#istype" >}}): Checks the type of an object.
 - [`len`]({{< relref "/filterx/function-reference.md#len" >}}): Returns the length of an object.
 - [`lower`]({{< relref "/filterx/function-reference.md#lower" >}}): Converts a string into lowercase characters.
 - [`parse_csv`]({{< relref "/filterx/filterx-parsing/csv/_index.md" >}}): Parses a comma-separated or similar string.
 - [`parse_kv`]({{< relref "/filterx/filterx-parsing/key-value-parser/_index.md" >}}): Parses a string consisting of whitespace or comma-separated `key=value` pairs.
-    <!-- - [`parse_xml`](FIXME): Parses an XML object into a JSON object. -->
+- [`parse_leef`]({{< relref "/filterx/filterx-parsing/leef/_index.md" >}}): Parses LEEF-formatted string.
+- [`parse_xml`]({{< relref "/filterx/filterx-parsing/xml/_index.md" >}}): Parses an XML object into a JSON object.
+- [`parse_windows_eventlog_xml`]({{< relref "/filterx/filterx-parsing/windows-eventlog/_index.md" >}}): Parses a Windows Event Log XML object into a JSON object.
 - [`regexp_search`]({{< relref "/filterx/function-reference.md#regexp-search" >}}): Searches a string using regular expressions.
 - [`regexp_subst`]({{< relref "/filterx/function-reference.md#regexp-subst" >}}): Rewrites a string using regular expressions.
+- [`startswith`]({{< relref "/filterx/filterx-string-search/_index.md" >}}): Checks if a string begins with the specified value.
 - [`strptime`]({{< relref "/filterx/function-reference.md#strptime" >}}): Converts a string containing a date/time value, using a specified format string.
 - [`unset`]({{< relref "/filterx/function-reference.md#unset" >}}): Deletes a name-value pair, or a field from an object.
 - [`unset_empties`]({{< relref "/filterx/function-reference.md#unset-empties" >}}): Deletes empty fields from an object.
+- [`update_metric`]({{< relref "/filterx/filterx-metrics/_index.md" >}}): Updates a labeled metric counter.
 - [`upper`]({{< relref "/filterx/function-reference.md#upper" >}}): Converts a string into uppercase characters.
 - [`vars`]({{< relref "/filterx/function-reference.md#vars" >}}): Lists the variables defined in the FilterX block.
 

content/filterx/filterx-metrics/_index.md

@@ -0,0 +1,85 @@
+---
+title: "Metrics"
+weight: 1000
+---
+<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->
+
+{{< include-headless "chunk/filterx-experimental-banner.md" >}}
+
+Available in {{< product >}} 4.9 and later.
+
+Updates a labeled metric counter, similarly to the [`metrics-probe()` parser]({{< relref "/chapter-parsers/metrics-probe/_index.md" >}}). For details, see {{% xref "/filterx/filterx-metrics/_index.md" %}}.
+
+You can use `update_metric` to count the processed messages, and create labeled metric counters based on the fields of the processed messages.
+
+You can configure the name of the counter to update and the labels to add. The name of the counter is an unnamed, mandatory option. Note that the name is automatically prefixed with the `syslogng_` string. For example:
+
+```json
+update_metric(
+    "my_counter_name",
+    labels={
+        "host": ${HOST},
+        "app": ${PROGRAM},
+        "id": ${SOURCE}
+    }
+);
+```
+
+This results in counters like:
+
+```shell
+syslogng_my_counter_name{app="example-app", host="localhost", source="s_local_1"} 3
+```
+
+## Options
+
+### increment
+
+|          |         |
+| -------- | ------- |
+| Type:    | integer or variable |
+| Default: | 1 |
+
+An integer, or an expression that resolves to an integer that defines the increment of the counter. The following example defines a counter called `syslogng_input_event_bytes_total`, and increases its value with the size of the incoming message (in bytes).
+
+```shell
+update_metric(
+    "input_event_bytes_total",
+    labels={
+        "host": ${HOST},
+        "app": ${PROGRAM},
+        "id": ${SOURCE}
+    },
+    increment=${RAWMSG_SIZE}
+);
+```
+
+### labels
+
+|          |         |
+| -------- | ------- |
+| Type:    | dict |
+| Default: | `{}` |
+
+The labels used to create separate counters, based on the fields of the messages processed by `update_metric`. Use the following format:
+
+```shell
+labels(
+    {
+      "name-of-label1": "value-of-the-label1",
+      ... ,
+      "name-of-labelx": "value-of-the-labelx"
+    }
+)
+```
+
+## level
+
+|          |         |
+| -------- | ------- |
+| Type:    | integer (0-3) |
+| Default: | 0 |
+
+Sets the stats level of the generated metrics.
+
+> Note: Drivers configured with `internal(yes)` register their metrics on level 3. That way if you are creating an SCL, you can disable the built-in metrics of the driver, and create metrics manually using `update_metric`.

content/filterx/filterx-parsing/cef/_index.md

@@ -0,0 +1,54 @@
+---
+title: "CEF"
+weight: 100
+---
+<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->
+
+{{< include-headless "chunk/filterx-experimental-banner.md" >}}
+
+Available in {{< product >}} 4.9 and later.
+
+The `parse_cef` FilterX function parses messages formatted in the [Common Event Format (CEF)](https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-8.3/cef-implementation-standard/Content/CEF/Chapter%201%20What%20is%20CEF.htm).
+
+## Declaration
+
+Usage: `parse_cef(<input-string>, value_separator="=", pair_separator="|")`
+
+The first argument is the input message. Optionally, you can set the `pair_separator` and `value_separator` arguments to override their default values.
+
+The `value_separator` must be a single-character string. The `pair_separator` can be a regular string.
+
+## Example
+
+The following is a CEF-formatted message including mandatory and custom (extension) fields:
+
+```shell
+CEF:0|KasperskyLab|SecurityCenter|13.2.0.1511|KLPRCI_TaskState|Completed successfully|1|foo=foo bar=bar baz=test
+```
+
+The following FilterX expression parses it and converts it into JSON format:
+
+```shell
+filterx {
+    ${PARSED_MESSAGE} = json(parse_cef(${MESSAGE}));
+};
+```
+
+The content of the JSON object for this message will be:
+
+```json
+{
+"version":"0",
+"device_vendor":"KasperskyLab",
+"device_product":"SecurityCenter",
+"device_version":"13.2.0.1511",
+"device_event_class_id":"KLPRCI_TaskState",
+"name":"Completed successfully",
+"agent_severity":"1",
+"extensions": {
+    "foo":"foo=bar",
+    "bar":"bar=baz",
+    "baz":"test"
+    }
+}
+```

content/filterx/filterx-parsing/cef/cef-parser-options/_index.md

@@ -0,0 +1,15 @@
+---
+title: "Options of CEF parsers"
+weight:  100
+---
+<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->
+
+The `parse_cef` FilterX function has the following options.
+
+## pair_separator
+
+Specifies the character or string that separates the key-value pairs in the extensions. Default value: ` ` (space).
+
+## value_separator
+
+Specifies the character that separates the keys from the values in the extensions. Default value: `=`.

content/filterx/filterx-parsing/csv/_index.md

@@ -64,7 +64,7 @@ block filterx p_apache() {
     "CONTENT_LENGTH", "REFERER", "USER_AGENT",
     "PROCESS_TIME", "SERVER_NAME"
     ];
-    ${APACHE} = parse_csv(${MESSAGE}, columns=cols, delimiter=(" "), strip_whitespaces=true, dialect="escape-double-char");
+    ${APACHE} = parse_csv(${MESSAGE}, columns=cols, delimiter=(" "), strip_whitespace=true, dialect="escape-double-char");
 
     # Set the important elements as name-value pairs so they can be referenced in the destination template
     ${APACHE_USER_NAME} = ${APACHE.USER_NAME};

content/filterx/filterx-parsing/csv/reference-parsers-csv/_index.md

@@ -76,7 +76,7 @@ my-parsed-values = parse_csv(${MESSAGE}, columns=["COLUMN1", "COLUMN2", "COLUMN3
 | Synopsis: | `strip_whitespace=true` |
 | Default value: | `false` |
 
-*Description:* Remove leading and trailing whitespaces from all columns. The `strip_whitespaces` option is an alias for `strip_whitespace`.
+*Description:* Remove leading and trailing whitespaces from all columns. The `strip_whitespace` option is an alias for `strip_whitespace`.
 
 ## string_delimiters {#string-delimiters}
 

content/filterx/filterx-parsing/leef/_index.md

@@ -0,0 +1,57 @@
+---
+title: "LEEF"
+weight: 1100
+---
+<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->
+
+{{< include-headless "chunk/filterx-experimental-banner.md" >}}
+
+Available in {{< product >}} 4.9 and later.
+
+The `parse_leef` FilterX function parses messages formatted in the [Log Event Extended Format (LEEF)](https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.ibm.com/docs/en/dsm%3Ftopic%3Doverview-leef-event-components&ved=2ahUKEwj87cLOjs-JAxUjgf0HHfxyM6AQFnoECBkQAQ&usg=AOvVaw1-YjjgdcnHjZLcJtzB3t6X).
+
+Both LEEF versions (1.0 and 2.0) are supported.
+
+## Declaration
+
+Usage: `parse_leef(<input-string>, value_separator="=", pair_separator="|")`
+
+The first argument is the input message. Optionally, you can set the `pair_separator` and `value_separator` arguments to override their default values.
+
+The `value_separator` must be a single-character string. The `pair_separator` can be a regular string.
+
+## Example
+
+The following is a LEEF-formatted message including mandatory and custom (extension) fields:
+
+```shell
+LEEF:1.0|Microsoft|MSExchange|4.0 SP1|15345|src=192.0.2.0 dst=172.50.123.1 sev=5cat=anomaly srcPort=81 dstPort=21 usrName=john.smith
+```
+
+The following FilterX expression parses it and converts it into JSON format:
+
+```shell
+filterx {
+    ${PARSED_MESSAGE} = json(parse_leef(${MESSAGE}));
+};
+```
+
+The content of the JSON object for this message will be:
+
+```json
+{
+"version":"1.0",
+"vendor":"Microsoft",
+"product_name":"MSExchange",
+"product_version":"4.0 SP1",
+"event_id":"15345",
+"extensions": {
+    "src":"192.0.2.0",
+    "dst":"172.50.123.1",
+    "sev":"5cat=anomaly",
+    "srcPort":"81",
+    "dstPort":"21",
+    "usrName":"john.smith"
+    }
+}
+```

content/filterx/filterx-parsing/leef/leef-parser-options/_index.md

@@ -0,0 +1,17 @@
+---
+title: "Options of LEEF parsers"
+weight:  100
+---
+<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->
+
+The `parse_leef` FilterX function has the following options.
+
+## pair_separator
+
+Specifies the character or string that separates the key-value pairs in the extensions. Default value: `\t` (tab).
+
+LEEF v2 can specify the separator per message. Omitting this option uses the LEEF v2 provided separator, setting this value overrides it during parsing.
+
+## value_separator
+
+Specifies the character that separates the keys from the values in the extensions. Default value: `=`.