consolidate multiline parsers info

Signed-off-by: Alexa Kreizinger <alexakreizinger@gmail.com>

Author: alexakreizinger

.gitbook.yaml

@@ -102,3 +102,4 @@ redirects:
     installation/downloads/amazon-ec2: ./installation/downloads/linux/amazon-linux.md
     administration/configuring-fluent-bit/yaml/configuration-file: ./administration/configuring-fluent-bit/yaml.md
     administration/configuring-fluent-bit/unit-sizes: ./administration/configuring-fluent-bit.md#unit-sizes
+    administration/configuring-fluent-bit/multiline-parsing: ./pipeline/parsers/multiline-parsing.md

SUMMARY.md

@@ -54,7 +54,6 @@
     * [Record accessor syntax](administration/configuring-fluent-bit/classic-mode/record-accessor.md)
     * [Upstream servers](administration/configuring-fluent-bit/classic-mode/upstream-servers.md)
     * [Variables](administration/configuring-fluent-bit/classic-mode/variables.md)
-  * [Multiline parsing](administration/configuring-fluent-bit/multiline-parsing.md)
 * [AWS credentials](administration/aws-credentials.md)
 * [Backpressure](administration/backpressure.md)
 * [Buffering and storage](administration/buffering-and-storage.md)
@@ -126,6 +125,7 @@
   * [Windows System Statistics (winstat)](pipeline/inputs/windows-system-statistics.md)
 * [Parsers](pipeline/parsers.md)
   * [Configuring custom parsers](pipeline/parsers/configuring-parser.md)
+  * [Multiline parsing](pipeline/parsers/multiline-parsing.md)
   * [JSON format](pipeline/parsers/json.md)
   * [Logfmt format](pipeline/parsers/logfmt.md)
   * [LTSV format](pipeline/parsers/ltsv.md)

administration/configuring-fluent-bit/yaml/multiline-parsers-section.md

@@ -1,10 +1,19 @@
 # Multiline parsers
 
-Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. Use this parser for handling stack traces, error logs, or any log entry that contains multiple lines of information.
+You can define custom [multiline parsers](../pipeline/parsers/multiline-parsing.md) in the `multiline_parsers` section of YAML configuration files.
 
-In YAML configuration, the syntax for defining multiline parsers differs slightly from the classic configuration format introducing minor breaking changes, specifically on how the rules are defined.
+{% hint style="info" %}
 
-The following example demonstrates how to define a multiline parser directly in the main configuration file, and how to include additional definitions from external files:
+To define standard custom parsers, use [the `parsers` section](../administration/configuring-fluent-bit/yaml/parsers-section.md) of YAML configuration files.
+
+{% endhint %}
+
+## Syntax
+
+To define customers parsers in the `multiline_parsers` section of a YAML configuration file, use the following syntax:
+
+{% tabs %}
+{% tab title="fluent-bit.yaml" %}
 
 ```yaml
 multiline_parsers:
@@ -20,6 +29,9 @@ multiline_parsers:
         next_state: cont
 ```
 
+{% endtab %}
+{% endtabs %}
+
 This example defines a multiline parser named `multiline-regex-test` that uses regular expressions to handle multi-event logs. The parser contains two rules: the first rule transitions from `start_state` to cont when a matching log entry is detected, and the second rule continues to match subsequent lines.
 
-For more detailed information on configuring multiline parsers, including advanced options and use cases, refer to the Configuring Multiline Parsers documentation.
+For information about supported configuration options for custom multiline parsers, see [configuring multiline parsers](./pipeline/parsers/multiline-parsing.md#configuring-multiline-parsers).

pipeline/filters/multiline-stacktrace.md

@@ -15,7 +15,7 @@ Along with multiline filters, you can enable one of the following built-in Fluen
 
 When using this filter:
 
-- The usage of this filter depends on a previous configuration of a [multiline parser](../../administration/configuring-fluent-bit/multiline-parsing.md) definition.
+- The usage of this filter depends on a previous configuration of a [multiline parser](../pipeline/parsers/multiline-parsing.md) definition.
 - To concatenate messages read from a log file, it's highly recommended to use the multiline support in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) itself. This is because performing concatenation while reading the log file is more performant. Concatenating messages that were originally one line, but split by Docker or CRI container engines because of their size, is supported in the [Tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-support) in combination with the `docker` or `cri` parser. To concatenate application logs like stacktraces on top of that, you can use this multiline filter.
 
 {% hint style="warning" %}
@@ -40,7 +40,7 @@ The plugin supports the following configuration parameters:
 
 | Property | Description |
 | -------- | ----------- |
-| `multiline.parser` | Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma. |
+| `multiline.parser` | Specify one or multiple [Multiline Parser definitions](../pipeline/parsers/multiline-parsing.md) to apply to the content. You can specify multiple multiline parsers to detect different formats by separating them with a comma. |
 | `multiline.key_content` | Key name that holds the content to process. A multiline parser definition can specify the `key_content` This option allows for overwriting that value for the purpose of the filter. |
 | `mode` | Mode can be `parser` for regular expression concatenation, or `partial_message` to concatenate split Docker logs. |
 | `buffer` | Enable buffered mode. In buffered mode, the filter can concatenate multiple lines from inputs that ingest records one by one (like Forward), rather than in chunks, re-emitting them into the beginning of the pipeline (with the same tag) using the `in_emitter` instance. With buffer off, this filter won't work with most inputs, except Tail. |
@@ -68,7 +68,7 @@ service:
   flush: 1
   log_level: info
   parsers_file: parsers_multiline.yaml
-    
+
 pipeline:
   inputs:
     - name: tail
@@ -157,10 +157,10 @@ This file defines a multiline parser for the example. A second multiline parser
   # Regex rules for multiline parsing
   # ---------------------------------
   #
-  # configuration hints:  
+  # configuration hints:
   #
   #  - first state always has the name: start_state
-  #  - every field in the rule must be inside double quotes 
+  #  - every field in the rule must be inside double quotes
   #
   # rules |   state name  | regex pattern                  | next state
   # ------|---------------|--------------------------------------------
@@ -314,7 +314,7 @@ Lines that don't match a pattern aren't considered as part of the multiline mess
 
 ## Docker partial message use case
 
-When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16&nbspKB. 
+When Fluent Bit is consuming logs from a container runtime, such as Docker, these logs will be split when larger than a certain limit, usually 16&nbspKB.
 If your application emits a 100K log line, it will be split into seven partial messages. The docker parser will merge these back to one line. If instead you are using the [Fluentd Docker Log Driver](https://docs.docker.com/config/containers/logging/fluentd/) to send the logs to Fluent Bit, they might look like this:
 
 ```text

pipeline/inputs/tail.md

@@ -158,9 +158,9 @@ Multiline core is exposed by the following configuration:
 
 | Key                | Description    |
 |:-------------------|:---------------|
-| `multiline.parser` | Specify one or multiple [Multiline Parser definitions](../../administration/configuring-fluent-bit/multiline-parsing.md) to apply to the content. |
+| `multiline.parser` | Specify one or multiple [Multiline Parser definitions](../pipeline/parsers/multiline-parsing.md) to apply to the content. |
 
-[Multiline Parser](../../administration/configuring-fluent-bit/multiline-parsing.md) provides built-in configuration modes. When using a new `multiline.parser` definition, you must disable the old configuration from your tail section like:
+[Multiline Parser](../pipeline/parsers/multiline-parsing.md) provides built-in configuration modes. When using a new `multiline.parser` definition, you must disable the old configuration from your tail section like:
 
 - `parser`
 - `parser_firstline`

…figuring-fluent-bit/multiline-parsing.md pipeline/parsers/multiline-parsing.mdadministration/configuring-fluent-bit/multiline-parsing.md renamed to pipeline/parsers/multiline-parsing.md administration/configuring-fluent-bit/multiline-parsing.md renamed to pipeline/parsers/multiline-parsing.md

@@ -4,11 +4,11 @@
 
 In an ideal world, applications might log their messages within a single line, but in reality applications generate multiple log messages that sometimes belong to the same context. Processing this information can be complex, like in application stack traces, which always have multiple log lines.
 
-Fluent Bit v1.8 implemented a unified Multiline core capability to solve corner cases.
+Multiline parsers are used to combine logs that span multiple events into a single, cohesive message. Use this parser for handling stack traces, error logs, or any log entry that contains multiple lines of information.
 
 ## Concepts
 
-The Multiline parser engine exposes two ways to configure and use the feature:
+The multiline parser engine exposes two ways to configure and use the feature:
 
 - Built-in multiline parser
 - Configurable multiline parser
@@ -26,13 +26,15 @@ Fluent Bit exposes certain pre-configured parsers (built-in) to solve specific m
 | `python` | Process log entries generated by a Python-based language application and perform concatenation if multiline messages are detected.      |
 | `ruby`   | Process log entries generated by a Ruby-based language application and perform concatenation if multiline messages are detected.        |
 
-### Configurable multiline parsers
+### Configuring multiline parsers
 
-You can define your own Multiline parsers with their own rules, using a configuration file.
+Custom multiline parsers support the following configuration parameters.
 
-A multiline parser is defined in a `parsers configuration file` by using a `[MULTILINE_PARSER]` section definition. The multiline parser must have a unique name and a type, plus other configured properties associated with each type.
+{% hint style="info" %}
+
+To define a custom multiline parser, add an entry to the [`multiline_parsers` section](../configuring-fluent-bit/yaml/multiline-parsers-section.md) of your YAML configuration file.
 
-To understand which multiline parser type is required for your use case you have to know the conditions in the content that determine the beginning of a multiline message, and the continuation of subsequent lines. Fluent Bit provides a regular expression-based configuration that supports states to handle from the most cases.
+{% endhint %}
 
 | Property | Description | Default |
 | -------- | ----------- | ------- |
@@ -43,7 +45,7 @@ To understand which multiline parser type is required for your use case you have
 | `negate`        | Negate the pattern matching result. When set to `true`, a non-matching line is treated as matching. | `false` |
 | `parser`        | Name of a pre-defined parser that must be applied to the incoming content before applying the regular expression rule. If no parser is defined, it's assumed that's a raw text and not a structured message. When a parser is applied to a raw text, the regular expression is applied against a specific key of the structured message by using the `key_content` configuration property. | _none_ |
 | `rule`          | Configure a rule to match a multiline pattern. The rule has a [specific format](#rules-definition). Multiple rules can be defined. Only used with `regex` type. | _none_|
-| `type`          | Set the multiline mode. Fluent Bit supports `regex`, `endswith`, and `equal` (or `eq`). | _none_ |
+| `type`          | Set the multiline mode. Fluent Bit supports `regex`, `endswith`, and `equal` (or `eq`). To understand which multiline parser type is required for your use case you have to know the conditions in the content that determine the beginning of a multiline message, and the continuation of subsequent lines. Fluent Bit provides a regular expression-based configuration that supports states to handle from the most cases. | _none_ |
 
 #### Lines and states
 
@@ -836,4 +838,4 @@ app/models/user.rb:42:in `validate_email'
 ```
 
 {% endtab %}
-{% endtabs %}
+{% endtabs %}