diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 5b726e60..cc6b55e4 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -18,7 +18,7 @@ jobs: - name: Set up Hugo uses: peaceiris/actions-hugo@v2 with: - hugo-version: '0.110.0' + hugo-version: '0.122.0' extended: true - name: Set up Node diff --git a/.github/workflows/publish.yaml b/.github/workflows/publish.yaml index f169107d..8adba1f8 100644 --- a/.github/workflows/publish.yaml +++ b/.github/workflows/publish.yaml @@ -38,7 +38,7 @@ jobs: - name: Set up Hugo uses: peaceiris/actions-hugo@v2 with: - hugo-version: '0.119.0' + hugo-version: '0.122.0' extended: true - name: Set up Node diff --git a/.github/workflows/staging.yaml b/.github/workflows/staging.yaml index f40e6fc2..98fd3ca3 100644 --- a/.github/workflows/staging.yaml +++ b/.github/workflows/staging.yaml @@ -33,7 +33,7 @@ jobs: - name: Set up Hugo uses: peaceiris/actions-hugo@v2 with: - hugo-version: '0.110.0' + hugo-version: '0.122.0' extended: true - name: Set up Node diff --git a/content/chapter-parsers/metrics-probe/_index.md b/content/chapter-parsers/metrics-probe/_index.md index 490d2922..0fbcd129 100644 --- a/content/chapter-parsers/metrics-probe/_index.md +++ b/content/chapter-parsers/metrics-probe/_index.md @@ -8,7 +8,7 @@ Available in {{% param "product_name" %}} version 4.1.1 and newer. `metrics-probe()` is a special parser that counts the messages that pass through the log path, and creates labeled stats counters based on the fields of the passing messages. -You can configure the name of the keys and the labels. Note that the keys are automatically prefixes with the `syslogng_` string. You can use templates in the values of the labels. +You can configure the name of the keys and the labels. Note that the keys are automatically prefixed with the `syslogng_` string. You can use templates in the values of the labels. The minimal configuration creates counters with the key `syslogng_classified_events_total` and labels `app`, `host`, `program` and `source`. For example: diff --git a/content/filterx/_index.md b/content/filterx/_index.md index 221fdfd5..0535d8f9 100644 --- a/content/filterx/_index.md +++ b/content/filterx/_index.md @@ -64,6 +64,10 @@ FilterX statements can be one of the following: - Existence of a variable of field. For example, the `${HOST};` expression is true only if the `${HOST}` macro exists and isn't empty. - A conditional statement ( `if (expr) { ... } elif (expr) {} else { ... };`) which allows you to evaluate complex decision trees. - A declaration of a [pipeline variable](#variable-scope), for example, `declare my_pipeline_variable = "something";`. +- A FilterX action. This can be one of the following: + + - `drop;`: Intentionally drop the message. This means that the message was successfully processed, but discarded. Processing the dropped message stops at the `drop` statement, subsequent sections or other branches of the FilterX block won't process the message. For example, you can use this to discard unneeded messages, like debug logs. Available in {{< product >}} 4.9 and later. + - `done;`: Return truthy and don't execute the rest of the FilterX block, returns with true. This is an early return that you can use to avoid unnecessary processing, for example, when the message matches an early classification in the block. Available in {{< product >}} 4.9 and later. {{% alert title="Note" color="info" %}} @@ -235,13 +239,9 @@ To unset every empty field of an object, use the [`unset-empties`]({{< relref "/ {{< include-headless "chunk/filterx-unset-hard-macros.md" >}} -## Concatenate strings +## Add two values -You can concatenate strings by adding them with the `+` operator. Note that if you want to have spaces between the added elements, you have to add them manually, like in Python, for example: - -```shell -${MESSAGE} = ${HOST} + " first part of the message," + " second part of the message" + "\n"; -``` +{{< include-headless "chunk/filterx-plus-operator.md" >}} ## Complex types: lists, dicts, and JSON {#json} @@ -331,11 +331,7 @@ Within a FilterX block, you can access the fields of complex data types by using When referring to the field of a name-value pair (which begins with the `$` character), place the dot or the square bracket outside the curly bracket surrounding the name of the name-value pair, for example: `${MY-LIST}[2]` or `${MY-OBJECT}.mykey`. If the name of the key contains characters that are not permitted in FilterX variable names, for example, a hyphen (`-`), use the bracketed syntax and enclose the key in double quotes: `${MY-LIST}["my-key-name"]`. - - - +You can add two lists or two dicts using the {{% xref "/filterx/operator-reference.md#plus-operator" %}}. +- [`parse_leef`]({{< relref "/filterx/filterx-parsing/leef/_index.md" >}}): Parses LEEF-formatted string. +- [`parse_xml`]({{< relref "/filterx/filterx-parsing/xml/_index.md" >}}): Parses an XML object into a JSON object. +- [`parse_windows_eventlog_xml`]({{< relref "/filterx/filterx-parsing/windows-eventlog/_index.md" >}}): Parses a Windows Event Log XML object into a JSON object. - [`regexp_search`]({{< relref "/filterx/function-reference.md#regexp-search" >}}): Searches a string using regular expressions. - [`regexp_subst`]({{< relref "/filterx/function-reference.md#regexp-subst" >}}): Rewrites a string using regular expressions. +- [`startswith`]({{< relref "/filterx/filterx-string-search/_index.md" >}}): Checks if a string begins with the specified value. - [`strptime`]({{< relref "/filterx/function-reference.md#strptime" >}}): Converts a string containing a date/time value, using a specified format string. - [`unset`]({{< relref "/filterx/function-reference.md#unset" >}}): Deletes a name-value pair, or a field from an object. - [`unset_empties`]({{< relref "/filterx/function-reference.md#unset-empties" >}}): Deletes empty fields from an object. +- [`update_metric`]({{< relref "/filterx/filterx-metrics/_index.md" >}}): Updates a labeled metric counter. - [`upper`]({{< relref "/filterx/function-reference.md#upper" >}}): Converts a string into uppercase characters. - [`vars`]({{< relref "/filterx/function-reference.md#vars" >}}): Lists the variables defined in the FilterX block. diff --git a/content/filterx/filterx-metrics/_index.md b/content/filterx/filterx-metrics/_index.md new file mode 100644 index 00000000..ce7515c3 --- /dev/null +++ b/content/filterx/filterx-metrics/_index.md @@ -0,0 +1,85 @@ +--- +title: "Metrics" +weight: 1000 +--- + + +{{< include-headless "chunk/filterx-experimental-banner.md" >}} + +Available in {{< product >}} 4.9 and later. + +Updates a labeled metric counter, similarly to the [`metrics-probe()` parser]({{< relref "/chapter-parsers/metrics-probe/_index.md" >}}). For details, see {{% xref "/filterx/filterx-metrics/_index.md" %}}. + +You can use `update_metric` to count the processed messages, and create labeled metric counters based on the fields of the processed messages. + +You can configure the name of the counter to update and the labels to add. The name of the counter is an unnamed, mandatory option. Note that the name is automatically prefixed with the `syslogng_` string. For example: + +```json +update_metric( + "my_counter_name", + labels={ + "host": ${HOST}, + "app": ${PROGRAM}, + "id": ${SOURCE} + } +); +``` + +This results in counters like: + +```shell +syslogng_my_counter_name{app="example-app", host="localhost", source="s_local_1"} 3 +``` + +## Options + +### increment + +| | | +| -------- | ------- | +| Type: | integer or variable | +| Default: | 1 | + +An integer, or an expression that resolves to an integer that defines the increment of the counter. The following example defines a counter called `syslogng_input_event_bytes_total`, and increases its value with the size of the incoming message (in bytes). + +```shell +update_metric( + "input_event_bytes_total", + labels={ + "host": ${HOST}, + "app": ${PROGRAM}, + "id": ${SOURCE} + }, + increment=${RAWMSG_SIZE} +); +``` + +### labels + +| | | +| -------- | ------- | +| Type: | dict | +| Default: | `{}` | + +The labels used to create separate counters, based on the fields of the messages processed by `update_metric`. Use the following format: + +```shell +labels( + { + "name-of-label1": "value-of-the-label1", + ... , + "name-of-labelx": "value-of-the-labelx" + } +) +``` + +## level + +| | | +| -------- | ------- | +| Type: | integer (0-3) | +| Default: | 0 | + +Sets the stats level of the generated metrics. + +> Note: Drivers configured with `internal(yes)` register their metrics on level 3. That way if you are creating an SCL, you can disable the built-in metrics of the driver, and create metrics manually using `update_metric`. diff --git a/content/filterx/filterx-parsing/cef/_index.md b/content/filterx/filterx-parsing/cef/_index.md new file mode 100644 index 00000000..b825b521 --- /dev/null +++ b/content/filterx/filterx-parsing/cef/_index.md @@ -0,0 +1,54 @@ +--- +title: "CEF" +weight: 100 +--- + + +{{< include-headless "chunk/filterx-experimental-banner.md" >}} + +Available in {{< product >}} 4.9 and later. + +The `parse_cef` FilterX function parses messages formatted in the [Common Event Format (CEF)](https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors-8.3/cef-implementation-standard/Content/CEF/Chapter%201%20What%20is%20CEF.htm). + +## Declaration + +Usage: `parse_cef(, value_separator="=", pair_separator="|")` + +The first argument is the input message. Optionally, you can set the `pair_separator` and `value_separator` arguments to override their default values. + +The `value_separator` must be a single-character string. The `pair_separator` can be a regular string. + +## Example + +The following is a CEF-formatted message including mandatory and custom (extension) fields: + +```shell +CEF:0|KasperskyLab|SecurityCenter|13.2.0.1511|KLPRCI_TaskState|Completed successfully|1|foo=foo bar=bar baz=test +``` + +The following FilterX expression parses it and converts it into JSON format: + +```shell +filterx { + ${PARSED_MESSAGE} = json(parse_cef(${MESSAGE})); +}; +``` + +The content of the JSON object for this message will be: + +```json +{ +"version":"0", +"device_vendor":"KasperskyLab", +"device_product":"SecurityCenter", +"device_version":"13.2.0.1511", +"device_event_class_id":"KLPRCI_TaskState", +"name":"Completed successfully", +"agent_severity":"1", +"extensions": { + "foo":"foo=bar", + "bar":"bar=baz", + "baz":"test" + } +} +``` diff --git a/content/filterx/filterx-parsing/cef/cef-parser-options/_index.md b/content/filterx/filterx-parsing/cef/cef-parser-options/_index.md new file mode 100644 index 00000000..66f0c44c --- /dev/null +++ b/content/filterx/filterx-parsing/cef/cef-parser-options/_index.md @@ -0,0 +1,15 @@ +--- +title: "Options of CEF parsers" +weight: 100 +--- + + +The `parse_cef` FilterX function has the following options. + +## pair_separator + +Specifies the character or string that separates the key-value pairs in the extensions. Default value: ` ` (space). + +## value_separator + +Specifies the character that separates the keys from the values in the extensions. Default value: `=`. diff --git a/content/filterx/filterx-parsing/csv/_index.md b/content/filterx/filterx-parsing/csv/_index.md index a1a0ea58..1fb97198 100644 --- a/content/filterx/filterx-parsing/csv/_index.md +++ b/content/filterx/filterx-parsing/csv/_index.md @@ -64,7 +64,7 @@ block filterx p_apache() { "CONTENT_LENGTH", "REFERER", "USER_AGENT", "PROCESS_TIME", "SERVER_NAME" ]; - ${APACHE} = parse_csv(${MESSAGE}, columns=cols, delimiter=(" "), strip_whitespaces=true, dialect="escape-double-char"); + ${APACHE} = parse_csv(${MESSAGE}, columns=cols, delimiter=(" "), strip_whitespace=true, dialect="escape-double-char"); # Set the important elements as name-value pairs so they can be referenced in the destination template ${APACHE_USER_NAME} = ${APACHE.USER_NAME}; diff --git a/content/filterx/filterx-parsing/csv/reference-parsers-csv/_index.md b/content/filterx/filterx-parsing/csv/reference-parsers-csv/_index.md index ee29b87b..6c0b6d1a 100644 --- a/content/filterx/filterx-parsing/csv/reference-parsers-csv/_index.md +++ b/content/filterx/filterx-parsing/csv/reference-parsers-csv/_index.md @@ -76,7 +76,7 @@ my-parsed-values = parse_csv(${MESSAGE}, columns=["COLUMN1", "COLUMN2", "COLUMN3 | Synopsis: | `strip_whitespace=true` | | Default value: | `false` | -*Description:* Remove leading and trailing whitespaces from all columns. The `strip_whitespaces` option is an alias for `strip_whitespace`. +*Description:* Remove leading and trailing whitespaces from all columns. The `strip_whitespace` option is an alias for `strip_whitespace`. ## string_delimiters {#string-delimiters} diff --git a/content/filterx/filterx-parsing/leef/_index.md b/content/filterx/filterx-parsing/leef/_index.md new file mode 100644 index 00000000..32ebf2a6 --- /dev/null +++ b/content/filterx/filterx-parsing/leef/_index.md @@ -0,0 +1,57 @@ +--- +title: "LEEF" +weight: 1100 +--- + + +{{< include-headless "chunk/filterx-experimental-banner.md" >}} + +Available in {{< product >}} 4.9 and later. + +The `parse_leef` FilterX function parses messages formatted in the [Log Event Extended Format (LEEF)](https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.ibm.com/docs/en/dsm%3Ftopic%3Doverview-leef-event-components&ved=2ahUKEwj87cLOjs-JAxUjgf0HHfxyM6AQFnoECBkQAQ&usg=AOvVaw1-YjjgdcnHjZLcJtzB3t6X). + +Both LEEF versions (1.0 and 2.0) are supported. + +## Declaration + +Usage: `parse_leef(, value_separator="=", pair_separator="|")` + +The first argument is the input message. Optionally, you can set the `pair_separator` and `value_separator` arguments to override their default values. + +The `value_separator` must be a single-character string. The `pair_separator` can be a regular string. + +## Example + +The following is a LEEF-formatted message including mandatory and custom (extension) fields: + +```shell +LEEF:1.0|Microsoft|MSExchange|4.0 SP1|15345|src=192.0.2.0 dst=172.50.123.1 sev=5cat=anomaly srcPort=81 dstPort=21 usrName=john.smith +``` + +The following FilterX expression parses it and converts it into JSON format: + +```shell +filterx { + ${PARSED_MESSAGE} = json(parse_leef(${MESSAGE})); +}; +``` + +The content of the JSON object for this message will be: + +```json +{ +"version":"1.0", +"vendor":"Microsoft", +"product_name":"MSExchange", +"product_version":"4.0 SP1", +"event_id":"15345", +"extensions": { + "src":"192.0.2.0", + "dst":"172.50.123.1", + "sev":"5cat=anomaly", + "srcPort":"81", + "dstPort":"21", + "usrName":"john.smith" + } +} +``` diff --git a/content/filterx/filterx-parsing/leef/leef-parser-options/_index.md b/content/filterx/filterx-parsing/leef/leef-parser-options/_index.md new file mode 100644 index 00000000..03e4215d --- /dev/null +++ b/content/filterx/filterx-parsing/leef/leef-parser-options/_index.md @@ -0,0 +1,17 @@ +--- +title: "Options of LEEF parsers" +weight: 100 +--- + + +The `parse_leef` FilterX function has the following options. + +## pair_separator + +Specifies the character or string that separates the key-value pairs in the extensions. Default value: `\t` (tab). + +LEEF v2 can specify the separator per message. Omitting this option uses the LEEF v2 provided separator, setting this value overrides it during parsing. + +## value_separator + +Specifies the character that separates the keys from the values in the extensions. Default value: `=`. diff --git a/content/filterx/filterx-parsing/windows-eventlog/_index.md b/content/filterx/filterx-parsing/windows-eventlog/_index.md new file mode 100644 index 00000000..a38d726a --- /dev/null +++ b/content/filterx/filterx-parsing/windows-eventlog/_index.md @@ -0,0 +1,105 @@ +--- +title: "Windows Event Log" +weight: 1100 +--- + + +{{< include-headless "chunk/filterx-experimental-banner.md" >}} + +Available in {{< product >}} 4.9 and later. + +The `parse_windows_eventlog_xml()` FilterX function parses Windows Event Logs XMLs. It's a specialized version of the [`parse_xml()` parser]({{< relref "/filterx/filterx-parsing/xml/_index.md" >}}). + +The parser returns false in the following cases: + +- The input isn't valid XML. +- The root element doesn't reference the [Windows Event Log schema](https://learn.microsoft.com/en-us/windows/win32/wes/eventschema-schema) (``). Note that the parser doesn't validate the input data to the schema. + +For example, the following converts the input XML into a JSON object: + +```shell +filterx { + xml = "" + $MSG = json(parse_windows_eventlog_xml(xml)); +}; +``` + +Given the following input: + +```xml + + + + 999 + 0 + 2 + 0 + 0 + 0x80000000000000 + + 934 + + + Application + DESKTOP-2MBFIV7 + + + + foobar + Error + + Info + + + + Classic + + + + foo + bar + + +``` + +The parser creates the following JSON object: + +```json +{ + "Event": { + "@xmlns": "http://schemas.microsoft.com/win/2004/08/events/event", + "System": { + "Provider": {"@Name": "EventCreate"}, + "EventID": {"@Qualifiers": "0", "#text": "999"}, + "Version": "0", + "Level": "2", + "Task": "0", + "Opcode": "0", + "Keywords": "0x80000000000000", + "TimeCreated": {"@SystemTime": "2024-01-12T09:30:12.1566754Z"}, + "EventRecordID": "934", + "Correlation": "", + "Execution": {"@ProcessID": "0", "@ThreadID": "0"}, + "Channel": "Application", + "Computer": "DESKTOP-2MBFIV7", + "Security": {"@UserID": "S-1-5-21-3714454296-2738353472-899133108-1001"}, + }, + "RenderingInfo": { + "@Culture": "en-US", + "Message": "foobar", + "Level": "Error", + "Task": "", + "Opcode": "Info", + "Channel": "", + "Provider": "", + "Keywords": {"Keyword": "Classic"}, + }, + "EventData": { + "Data": { + "param1": "foo", + "param2": "bar", + }, + }, + }, +} +``` diff --git a/content/filterx/filterx-parsing/xml/_index.md b/content/filterx/filterx-parsing/xml/_index.md index 90e823d1..ae8582a3 100644 --- a/content/filterx/filterx-parsing/xml/_index.md +++ b/content/filterx/filterx-parsing/xml/_index.md @@ -1,10 +1,54 @@ --- title: "XML" -weight: 1100 -draft: true +weight: 1300 --- {{< include-headless "chunk/filterx-experimental-banner.md" >}} -The `parse_xml` FilterX function can ... \ No newline at end of file +Available in {{< product >}} 4.9 and later. + +The `parse_xml()` FilterX function parses raw XMLs into dictionaries. This is a new implementation, so the limitations and options of the [legacy `xml-parser()`]({{< relref "/chapter-parsers/xml-parser/_index.md" >}}) do not apply. + +There is no standardized way of converting XML into a dict. {{< product >}} creates the most compact dict possible. This means certain nodes will have different types and structures depending on the input XML element. Note the following points: + +1. Empty XML elements become empty strings. + + ```yaml + XML: + JSON: {"foo": ""} + ``` + +1. Attributions are stored in `@attr` key-value pairs, similarly to other converters (like python xmltodict). + + ```yaml + XML: + JSON: {"foo": {"@bar": "123", "@baz": "bad"}} + ``` + +1. If an XML element has both attributes and a value, we need to store them in a dict, and the value needs a key. We store the text value under the `#text` key. + + ```yaml + XML: baz + JSON: {"foo": {"@bar": "123", "#text": "baz"}} + ``` + +1. An XML element can have both a value and inner elements. We use the `#text` key here, too. + + ```yaml + XML: bar123 + JSON: {"foo": {"#text": "bar", "baz": "123"}} + ``` + +1. An XML element can have multiple values separated by inner elements. In that case we concatenate the values. + + ```yaml + XML: barbaz + JSON: {"foo": {"#text": "barbaz", "a": ""}} + ``` + +## Usage + +```shell +my_structured_data = parse_xml(raw_xml); +``` diff --git a/content/filterx/filterx-parsing/xml/xml-parser-options/_index.md b/content/filterx/filterx-parsing/xml/xml-parser-options/_index.md deleted file mode 100644 index ef1d6738..00000000 --- a/content/filterx/filterx-parsing/xml/xml-parser-options/_index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: "Options of key=value parsers" -weight: 100 ---- - - -The `parse_xml` FilterX function has the following options. - diff --git a/content/filterx/filterx-sdata/_index.md b/content/filterx/filterx-sdata/_index.md new file mode 100644 index 00000000..6206dc6f --- /dev/null +++ b/content/filterx/filterx-sdata/_index.md @@ -0,0 +1,48 @@ +--- +title: "Handle SDATA in RFC5424 log records" +linkTitle: "SDATA in syslog" +weight: 900 +--- + + +{{< include-headless "chunk/filterx-experimental-banner.md" >}} + +Available in {{< product >}} 4.9 and later. + +{{< product >}} FilterX has a few functions to handle the [structured data (SDATA) part of RFC5424-formatted log messages]({{< relref "/chapter-concepts/concepts-message-structure/concepts-message-ietfsyslog/_index.md#the-structured-data-message-part" >}}). These functions allow you to filter messages based on their SDATA fields. + + + +## get_sdata() + +Extracts the SDATA part of the message into a two-level dictionary, for example: + +```json +{"Originator@6876": {"sub": "Vimsvc.ha-eventmgr", "opID": "esxui-13c6-6b16"}} +``` + +```shell +filterx { + sdata_json = get_sdata(); +}; +``` + +## has_sdata() + +Returns `true` if the SDATA field of the current message is not empty: + +```shell +filterx { + has_sdata(); +}; +``` + +## is_sdata_from_enterprise + +Filter messages based on enterprise ID in the SDATA field. For example: + +```shell +filterx { + is_sdata_from_enterprise("6876"); +}; +``` diff --git a/content/filterx/filterx-string-search/_index.md b/content/filterx/filterx-string-search/_index.md new file mode 100644 index 00000000..a5465590 --- /dev/null +++ b/content/filterx/filterx-string-search/_index.md @@ -0,0 +1,36 @@ +--- +title: "String search in FilterX" +linkTitle: "String search" +weight: 550 +--- + + +{{< include-headless "chunk/filterx-experimental-banner.md" >}} + +Available in {{< product >}} 4.9 and later. + +You can check if a string contains a specified string using the `includes` FilterX function. The `startswith` and `endswith` functions check the beginning and ending of the strings, respectively. For example, the following expression checks if the message (`$MESSAGE`) begins with the `%ASA-` string: + +```shell +startswith($MESSAGE, '%ASA-') +``` + +By default, matches are case sensitive. For case insensitive matches, use the `ignorecase=true` option: + +```shell +startswith($MESSAGE, '%ASA-', ignorecase=true) +``` + +All three functions (`includes`, `startswith`, and `endswith`) can take a list with multiple search strings and return true if any of them match. This is equivalent with using combining the individual searches with logical OR operators. For example: + +```shell +${MESSAGE} = "%ASA-5-111010: User ''john'', running ''CLI'' from IP 0.0.0.0, executed ''dir disk0:/dap.xml" +includes($MESSAGE, ['%ASA-','john','CLI']) + +includes($MESSAGE, ['%ASA-','john','CLI']) +includes($MESSAGE, '%ASA-') or includes($MESSAGE, 'john') or includes($MESSAGE, 'CLI') +``` + +For more complex searches, or if you need to match a regular expression, use the [`regexp_search` FilterX function]({{< relref "/filterx/filterx-string-search/_index.md#regexp-search" >}}). + + \ No newline at end of file diff --git a/content/filterx/function-reference.md b/content/filterx/function-reference.md index c1c90072..375fc1a1 100644 --- a/content/filterx/function-reference.md +++ b/content/filterx/function-reference.md @@ -65,11 +65,24 @@ Usually, you use the [strptime](#strptime) FilterX function to create datetime v - When casting from a double, the double is the number of seconds elapsed since the UNIX epoch (00:00:00 UTC on 1 January 1970). (The part before the floating points is the seconds, the part after the floating point is the microseconds.) - When casting from a string, the string (for example, `1701350398.123000+01:00`) is interpreted as: `.+` +## endswith + +Available in {{< product >}} 4.9 and later. + +Returns true if the input string ends with the specified substring. By default, matches are case sensitive. Usage: + +```shell +endswith(input-string, substring); +endswith(input-string, [substring_1, substring_2], ignorecase=true); +``` + +For details, see {{% xref "/filterx/filterx-string-search/_index.md" %}}. + ## flatten Flattens the nested elements of an object using the specified separator, similarly to the [`format-flat-json()` template function]({{< relref "/chapter-manipulating-messages/customizing-message-format/reference-template-functions/_index.md#template-function-format-flat-json" >}}). For example, you can use it to flatten nested JSON objects in the output if the receiving application cannot handle nested JSON objects. -Usage: `flatten(dict, separator=".")` +Usage: `flatten(dict_or_list, separator=".")` You can use multi-character separators, for example, `=>`. If you omit the separator, the default dot (`.`) separator is used. @@ -118,10 +131,35 @@ Formats any value into a raw JSON string. Usage: `format_json($data)` +## get_sdata + +See {{% xref "/filterx/filterx-sdata/_index.md" %}}. + +## has_sdata + +See {{% xref "/filterx/filterx-sdata/_index.md" %}}. + +## includes + +Available in {{< product >}} 4.9 and later. + +Returns true if the input string contains the specified substring. By default, matches are case sensitive. Usage: + +```shell +includes(input-string, substring); +includes(input-string, [substring_1, substring_2], ignorecase=true); +``` + +For details, see {{% xref "/filterx/filterx-string-search/_index.md" %}}. + ## isodate Parses a string as a date in ISODATE format: `%Y-%m-%dT%H:%M:%S%z` +## is_sdata_from_enterprise() + +See {{% xref "/filterx/filterx-sdata/_index.md" %}}. + ## isset Returns true if the argument exists and its value is not empty or null. @@ -155,7 +193,13 @@ Usage: `json()` For example: ```shell -js = json({"key": "value"}); +js_dict = json({"key": "value"}); +``` + +Starting with version 4.9, you can use `{}` without the `json()` keyword as well. For example, the following creates an empty JSON object: + +```shell +js_dict = {}; ``` ## json_array {#json-array} @@ -167,7 +211,13 @@ Usage: `json_array()` For example: ```shell -list = json_array(["first_element", "second_element", "third_element"]); +js_list = json_array(["first_element", "second_element", "third_element"]); +``` + +Starting with version 4.9, you can use `[]` without the `json_array()` keyword as well. For example, the following creates an empty JSON list: + +```shell +js_dict = []; ``` ## len @@ -220,6 +270,30 @@ The `value_separator` must be a single character. The `pair_separator` can consi For details, see {{% xref "/filterx/filterx-parsing/key-value-parser/_index.md" %}}. +## parse_leef {#parse-leef} + +Parse a LEEF-formatted string. + +Usage: `parse_leef(msg)` + +For details, see {{% xref "/filterx/filterx-parsing/leef/_index.md" %}}. + +## parse_xml {#parse-xml} + +Parse an XML object into a JSON object. + +Usage: `parse_xml(msg)` + +For details, see {{< relref "/filterx/filterx-parsing/xml/_index.md" >}} + +## parse_windows_eventlog_xml {#parse-windows} + +Parses a Windows Event Log XML object into a JSON object. + +Usage: `parse_xml(msg)` + +For details, see {{< relref "/filterx/filterx-parsing/xml/_index.md" >}} + ## regexp_search {#regexp-search} Searches a string and returns the matches of a regular expression as a list or a dictionary. If there are no matches, the list or dictionary is empty. @@ -308,6 +382,19 @@ You can use the following flags with the `regexp_subst` function: - `utf8=true`: {{< include-headless "chunk/regex-flag-utf8.md" >}} +## startswith + +Available in {{< product >}} 4.9 and later. + +Returns true if the input string begins with the specified substring. By default, matches are case sensitive. Usage: + +```shell +startswith(input-string, substring); +startswith(input-string, [substring_1, substring_2], ignorecase=true); +``` + +For details, see {{% xref "/filterx/filterx-string-search/_index.md" %}}. + ## string Cast a value into a string. Note that currently {{< product >}} evaluates strings and executes [template functions]({{< relref "/filterx/_index.md#template-functions" >}}) and template expressions within the strings. In the future, template evaluation will be moved to a separate FilterX function. @@ -358,34 +445,26 @@ See also {{% xref "/filterx/_index.md#delete-values" %}}. ## unset_empties {#unset-empties} -Deletes ([unsets](#unset)) the empty fields of an object, for example, a JSON object or list. Use the `recursive=true` parameter to delete empty values of inner dicts' and lists' values. +Deletes ([unsets](#unset)) the empty fields of an object, for example, a JSON object or list. By default, the object is processed recursively, so the empty values are deleted from inner dicts and lists as well. If you set the `replacement` option, you can also use this function to replace fields of the object to custom values. -Usage: `unset_empties(object, recursive=true)` +Usage: `unset_empties(object, options)` - +Updates a labeled metric counter, similarly to the [`metrics-probe()` parser]({{< relref "/chapter-parsers/metrics-probe/_index.md" >}}). For details, see {{% xref "/filterx/filterx-metrics/_index.md" %}}. ## upper diff --git a/content/filterx/operator-reference.md b/content/filterx/operator-reference.md index f0ba3798..ace63531 100644 --- a/content/filterx/operator-reference.md +++ b/content/filterx/operator-reference.md @@ -37,11 +37,7 @@ ${MESSAGE} = json["BODY"] ?? "Empty message" ## Plus operator -The plus operator (`+`) adds two arguments, if possible. (For example, you can't add two datetime values.) - -You can use it to add two numbers (two integers, two double values). If you add a double to an integer, the result is a double. - -Adding two strings [concatenates the strings]({{< relref "/filterx/_index.md#concatenate-strings" >}}). +{{< include-headless "chunk/filterx-plus-operator.md" >}} ## Plus equal operator diff --git a/content/headless/chunk/filterx-plus-operator.md b/content/headless/chunk/filterx-plus-operator.md new file mode 100644 index 00000000..064fbd21 --- /dev/null +++ b/content/headless/chunk/filterx-plus-operator.md @@ -0,0 +1,19 @@ +The plus operator (`+`) adds two arguments, if possible. (For example, you can't add two datetime values.) + +- You can use it to add two numbers (two integers, two double values). If you add a double to an integer, the result is a double. +- Adding two strings concatenates the strings. Note that if you want to have spaces between the added elements, you have to add them manually, like in Python, for example: + + ```shell + ${MESSAGE} = ${HOST} + " first part of the message," + " second part of the message" + "\n"; + ``` + +- Adding two lists merges the lists. Available in {{< product >}} 4.9 and later. +- Adding two dicts updates the dict with the values of the second operand. For example: + + ```shell + x = {"key1": "value1", "key2": "value1"}; + y = {"key3": "value1", "key2": "value2"}; + ${MESSAGE} = x + y; # ${MESSAGE} value is {"key1": "value1", "key3": "value1", "key2": "value2"}; + ``` + + Available in {{< product >}} 4.9 and later.