Docs for v1.6.0

README.md

@@ -10,12 +10,14 @@ This project is unofficial and community-driven.
 Please proceed to the microsite for more information:
 
 - [Quick Start](https://loki4j.github.io/loki-logback-appender/#quick-start)
-- [Configuration Guide](https://loki4j.github.io/loki-logback-appender/docs/configuration)
+- [Configuration Reference](https://loki4j.github.io/loki-logback-appender/docs/configuration)
+- [Migration Guide](https://loki4j.github.io/loki-logback-appender/docs/migration)
 
 If you have found this project helpful, please drop a :star:!
 
 ## Key features
 
+- Structured metadata support
 - Flexible management of Loki labels using MDC and SLF4J Markers
 - Out-of-the-box JSON layout support for log message formatting
 - Logback plain text formatting patterns can be used for both labels and messages

docs/docus/docs/apacheclient.md

@@ -4,11 +4,11 @@ title: Using Apache HttpClient
 sidebar_label: Apache HttpClient
 ---
 
-By default, Loki4j uses `JavaHttpSender`, backed by `java.net.http.HttpClient` available in Java 11 and later.
+By default, Loki4j uses `JavaHttpSender`, backed by `java.net.http.HttpClient`.
 This sender does not require any extra dependencies.
 So, it should be a good fit for most users.
 
-However, you may want to switch to `ApacheHttpSender`, backed by `org.apache.http.client.HttpClient` available for Java 8+ projects.
+However, you may want to switch to `ApacheHttpSender`, backed by `org.apache.http.client.HttpClient`.
 In this case, you need to ensure you have added the required dependencies to your project:
 
 <!--DOCUSAURUS_CODE_TABS-->

docs/docus/docs/compatibility.md

@@ -1,16 +1,17 @@
 ---
 id: compatibility
-title: Loki4j Compatibility Matrix
+title: Loki4j compatibility matrix
 sidebar_label: Compatibility Matrix
 ---
 
 The versions of Loki4j that had introduced backward-incompatible platform upgrades are listed in the table below.
 
-|Loki4j|Java|Logback|
-|-------|-------|-----------|
-|v1.5.0||v1.3.x|
-|v0.3.0|[Java 8](#java-8-support), Java 11+||
-|v0.1.0|Java 11+|v1.2.x|
+|Loki4j|Java|Logback|Loki|
+|------|----|-------|----|
+|v1.6.0|Java 11+|v1.4.x|v2.8.0|
+|v1.5.0||v1.3.x||
+|v0.3.0|[Java 8](#java-8-support), Java 11+|||
+|v0.1.0|Java 11+|v1.2.x|v1.6.1|
 
 
 ### Java 8 support

docs/docus/docs/configuration.md

@@ -1,6 +1,6 @@
 ---
 id: configuration
-title: Loki4j Configuration
+title: Loki4j configuration
 sidebar_label: Configuration
 ---
 
@@ -65,13 +65,12 @@ Format settings do not depend on the encoding you use.
 |Setting|Default|Description|
 |-------|-------|-----------|
 |format.label.pattern||**Required**. Logback pattern to use for log record's label|
-|format.label.pairSeparator|,|Character sequence to use as a separator between labels. If it starts with the "regex:" prefix, the remainder is applied as a regular expression separator. Otherwise, the provided char sequence is used as a separator literally|
+|format.label.structuredMetadataPattern||Logback pattern to use for log record's structured metadata|
+|format.label.pairSeparator|,|Character sequence to use as a separator between labels|
 |format.label.keyValueSeparator|=|Character to use as a separator between label's name and its value|
 |format.label.readMarkers|false|If true, Loki4j scans each log record for the attached LabelMarker to add its values to the record's labels|
-|format.label.nopex|true|If true, exception info is not added to labels. If false, you should take care of proper formatting|
 |format.label.streamCache|BoundAtomicMapCache|An implementation of a stream cache to use. By default, caches up to 1000 unique label sets|
-|format.staticLabels|false|If you use only a constant label set (e.g., same keys and values) for all log records, you can set this flag to true and save some CPU time on grouping records by label|
-|format.sortByTime|false|If true, log records in batch are sorted by timestamp. If false, records will be sent to Loki in arrival order. Enable this if you see an 'entry out of order' error from Loki|
+|format.staticLabels|false|If you use only a constant label set (e.g., same keys and values) for all log records, you can set this flag to true and save some CPU and RAM|
 
 #### Plain text message layout
 
@@ -95,8 +94,15 @@ This layout has the following settings:
 |format.message.loggerName.targetLength|-1|The desired target length of logger name: `-1` to disable abbreviation, `0` to print class name only, >`0` to abbreviate to the target length|
 |format.message.logLevel.enabled|true|Enable logLevel provider|
 |format.message.logLevel.fieldName|level|A JSON field name to use for logLevel|
+|format.message.kvp.enabled|true|Enable keyValuePair provider|
+|format.message.kvp.prefix|kvp_|A prefix added to each JSON field name written by this provider|
+|format.message.kvp.noPrefix|false|Whether to omit prefix for this provider|
+|format.message.kvp.fieldSerializer||An implementation of field JSON serializer. By default, `writeObjectField()` is used|
+|format.message.kvp.include||A set of keys to include in JSON payload. If not specified, all keys are included|
+|format.message.kvp.exclude||A set of keys to exclude from JSON payload. The exclude list has precedence over the include list. If not specified, all keys are included|
 |format.message.mdc.enabled|true|Enable MDC provider|
-|format.message.mdc.fieldName|mdc_|A prefix added to each JSON field name written by this provider|
+|format.message.mdc.prefix|mdc_|A prefix added to each JSON field name written by this provider|
+|format.message.mdc.noPrefix|false|Whether to omit prefix for this provider|
 |format.message.mdc.include||A set of MDC keys to include in JSON payload. If not specified, all keys are included|
 |format.message.mdc.exclude||A set of MDC keys to exclude from JSON payload. The exclude list has precedence over the include list. If not specified, all keys are included|
 |format.message.message.enabled|true|Enable message provider|

docs/docus/docs/index.md

@@ -6,7 +6,7 @@ configuration and enjoy.
 
 ### Quick Start
 
-The current stable version of Loki4j requires Java 11+ and Logback v1.3.x.
+The current stable version of Loki4j requires Java 11+ and Logback v1.4.x.
 See the [compatibility matrix](docs/compatibility) for more information about older versions' support.
 
 Add the following dependency to your project:
@@ -40,10 +40,21 @@ Then add Loki appender to your `logback.xml`:
     </http>
     <format>
         <label>
-            <pattern>app=my-app,host=${HOSTNAME}</pattern>
+            <!-- Labels -->
+            <pattern>
+                app = my-app,
+                host = ${HOSTNAME}
+            </pattern>
+            <!-- Structured metadata (since Loki v2.9.0) -->
+            <structuredMetadataPattern>
+                level = %level,
+                thread = %thread,
+                class = %logger,
+                traceId = %mdc{traceId:-none}
+            </structuredMetadataPattern>
         </label>
         <message>
-            <pattern>%-5level [%.5(${HOSTNAME})] %.10thread %logger{20} | %msg %ex</pattern>
+            <pattern>%-5level %logger{20} %msg %ex</pattern>
         </message>
     </format>
 </appender>
@@ -80,8 +91,11 @@ Migrating from the previous Loki4j version? Read the [Migration Guide](docs/migr
 
 ### Key Features:
 
-- **Flexible management of Loki labels using MDC and SLF4J Markers.**
-You can specify Loki labels dynamically for any set of log records and even on a per-record basis.
+- **Structured metadata support.**
+Pass any non-label metadata along with your log lines using [structured metadata](docs/metadata).
+
+- **Flexible management of Loki labels and metadata using MDC and SLF4J Markers.**
+You can specify Loki labels as well as structured metadata dynamically, even on a per-record basis.
 [Learn more...](docs/labels)
 
 - **Out-of-the-box JSON layout support for log message formatting.**

docs/docus/docs/jsonlayout.md

@@ -1,9 +1,26 @@
 ---
 id: jsonlayout
-title: Configuring JSON Message Layout
+title: Configuring JSON message layout
 sidebar_label: JSON Message Layout
 ---
 
+## When to use JSON message layout
+
+Loki4j supports JSON layout for log messages, however it is not Loki's native format for logs.
+Producing and parsing log lines in JSON imposes performance penalties and extra cost comparing to the plain text layout.
+Also worth mentioning that JSON is harder to read for humans than the plain text as well.
+
+Having that said, Loki4j's JSON message layout has no alternatives if you:
+
+- heavily use MDC or KV in multiple dynamic contexts and need more flexibility to configure them;
+- need nested structures in your metadata;
+- have tools in your logging stack (apart from Loki) that require JSON format.
+
+So before you decide to use JSON layout, we recommend you to take a look at other, more straight-forward ways for attaching metadata to log records: [labels](labels.md) and [structured metadata](metadata.md).
+Both of them work perfectly with plain text layout.
+
+Now, if you are still sure JSON layout is the right option for you, please proceed to the next section.
+
 ## Enabling the JSON layout
 
 You can enable JSON layout for log messages by specifying a corresponding `class` attribute for a `message` section:
@@ -118,3 +135,117 @@ You can add as many custom providers as you want:
     <customProvider class="io.my.ConstantProvider3" />
 </message>
 ```
+
+For more insights on how `JsonEventWriter` works, please check the next section.
+
+## Serializing an arbitrary object
+
+Our recommendation is to use plain structure of the log message (i.e., no nested objects), and only some basic types for fields (e.g.: string, number, boolean).
+Following this recommendation will give you both maximum performance and simplicity.
+
+However, Logback allows you to attach arbitrary objects to you log record as key-value pairs:
+
+```java
+log.atInfo().setMessage("Test KVP")
+    .addKeyValue("flow", "ingest")
+    .addKeyValue("msgId", 123456)
+    .addKeyValue("bool", true)
+    .addKeyValue("list", List.of(0, 1, 2))
+    .addKeyValue("date", ZonedDateTime.now())
+    .addKeyValue("obj", new TestJsonKvData(1001L, "admin", UUID.randomUUID()))
+    .log();
+```
+
+Loki4j uses fast and compact JSON serialization algorithm that is [3.5x](https://github.com/loki4j/loki-logback-appender/pull/210#issue-2113540494) faster than logstash-logback-encoder backed by jackson.
+But that speed comes with the cost: no runtime reflection is used, no complex type conversion is implemented.
+Which means we don't provide a built-in mechanism to serialize arbitrary object into JSON right away.
+Instead, we offer you several options, so you can choose the one that better fits your use case.
+
+Loki4j has `KeyValuePairsJsonProvider`, that by default uses `writeObjectField()` for all key-value pairs.
+
+#### Default approach: `writeObjectField()`
+
+Method `writeObjectField()` in class `JsonEventWriter` has the following logic:
+
+- if value is `String`, it's rendered as JSON string: `"flow":"ingest"`, similar to `writeStringField()`;
+- if value is `Integer` or `Long`, it's rendered as JSON number: `"msgId":123456`, similar to `writeNumericField()`;
+- if value is `Boolean`, it's rendered as JSON boolean: `"bool":true`;
+- if value is `Iterable`, it's rendered as JSON array, `writeObjectField()` to render each element: `"list":[0,1,2]`, similar to `writeArrayField()`;
+- if value is `RawJsonString`, its value is rendered as raw JSON without any escaping (see the section below), similar to `writeRawJsonField()`;
+- for any other type of value, the result of `toString()` is rendered as JSON string: `"obj":"TestJsonKvData@1de5f259"`.
+
+#### Direct access to `JsonEventWriter`
+
+If you write a custom provider, `JsonEventWriter` is exposed to you directly.
+You can also configure `KeyValuePairsJsonProvider` to intercept writer calls as well by setting `fieldSerializer`:
+
+```xml
+<message class="com.github.loki4j.logback.JsonLayout">
+    ...
+    <kvp>
+        <fieldSerializer class="io.my.TestFieldSerializer" />
+    </kvp>
+</message>
+```
+
+Your custom serializer must implement interface `JsonFieldSerializer<Object>`:
+
+```java
+import com.github.loki4j.logback.json.JsonEventWriter;
+import com.github.loki4j.logback.json.JsonFieldSerializer;
+
+public class TestFieldSerializer implements JsonFieldSerializer<Object> {
+    @Override
+    public void writeField(JsonEventWriter writer, String fieldName, Object fieldValue) {
+        if (fieldValue instanceof TestJsonKvData) {
+            writer.writeCustomField(fieldName, w -> {
+                    var td = (TestJsonKvData)fieldValue;
+                    w.writeBeginObject();
+                    w.writeObjectField("userId", td.userId);
+                    w.writeFieldSeparator();
+                    w.writeObjectField("userName", td.userName);
+                    w.writeFieldSeparator();
+                    w.writeObjectField("sessionId", td.sessionId);
+                    w.writeEndObject();
+                }
+            );
+        } else {
+            writer.writeObjectField(fieldName, fieldValue);
+        }
+    }
+}
+```
+
+Instead of `"obj":"TestJsonKvData@1de5f259"`, now you will see `"obj":{"userId":1001,"userName":"admin","sessionId":"92a26b23-9f10-47d8-bb0a-df2b9b15c374"}`.
+
+#### `RawJsonString`
+
+If for some reason you can not statically define serialization algorithm for your objects, you can use reflection-based frameworks and put the resulting string into `RawJsonString`.
+Writer will render it as is, without any escaping.
+
+```java
+import com.github.loki4j.logback.json.RawJsonString;
+
+public class TestFieldSerializer implements JsonFieldSerializer<Object> {
+    @Override
+    public void writeField(JsonEventWriter writer, String fieldName, Object fieldValue) {
+        writer.writeObjectField(fieldName, new RawJsonString(MyJson.serialize(fieldValue)));
+        // or
+        writer.writeRawJsonField(fieldName, MyJson.serialize(fieldValue));
+    }
+}
+```
+
+It is your responsibility to make sure that you always put valid JSON into `RawJsonString`.
+
+#### Custom message layout
+
+You can use any implementation of `Layout<ILoggingEvent>` as a message layout, including third-party JSON layouts:
+
+```xml
+<message class="net.logstash.logback.layout.LoggingEventCompositeJsonLayout">
+    <!-- your logstash configuration goes here -->
+</message>
+```
+
+Please note that Loki4j provides **no support** for any third-party components.

docs/docus/docs/labels.md

@@ -33,26 +33,20 @@ By default labels are defined as `key=value` pairs separated by commas.
 </label>
 ```
 
-But you can override `pairSeparator` to organize them in a different way.
-For example, if you have many labels, it's better to have each of them on a separate line:
+If you have many labels, you can put them in multiple lines (a trailing comma is optional):
 
 ```xml
 <label>
     <pattern>
-        job=loki4j
-        app=my-app
-        // you even can write comments here
-        namespace_name=${NAMESCAPE_NAME}
-        pod_name=${POD_NAME}
-        level=%level
+        job = loki4j,
+        app = my-app,
+        namespace_name = ${NAMESCAPE_NAME},
+        pod_name = ${POD_NAME},
+        level=%level,
     </pattern>
-    <pairSeparator>regex:(\n|//[^\n]+)+</pairSeparator>
 </label>
 ```
 
-Please note that in the example above the regular expression in `pairSeparator` defines lines starting with `//` a part of a separator.
-So now we have a `// comment` feature here as well.
-
 ## Using MDC in labels
 
 `label.pattern` is nothing but Logback's [pattern layout](https://logback.qos.ch/manual/layouts.html#ClassicPatternLayout), which means it supports [MDC](https://logback.qos.ch/manual/mdc.html):
@@ -88,4 +82,23 @@ void handleException(Exception ex) {
     var marker = LabelMarker.of("exceptionClass", () -> ex.getClass().getSimpleName());
     log.error(marker, "Unexpected error", ex);
 }
-```
+```
+
+## Best practices
+
+We encourage you to follow the [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/) collected by Grafana Loki team. Loki4j provides several settings to facilitate these recommendations.
+
+First, make sure you have `format.staticLabels` flag enabled.
+This will prevent Loki4j from calculating labels for each particular log record:
+
+```xml
+<appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
+    ...
+    <format>
+        <staticLabels>true</staticLabels>
+        ...
+     </format>
+</appender>
+```
+
+Second, make sure you put all the high-cardinality metadata to [structured metadata](metadata.md) instead of labels.

docs/docus/docs/metadata.md

@@ -0,0 +1,50 @@
+---
+id: metadata
+title: Structured metadata
+sidebar_label: Structured Metadata
+---
+
+[Structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) is a way to attach high-cardinality metadata to logs without indexing them or including them in the log line content itself.
+Structured metadata is a set of key-value pairs; both keys and values must be plain strings.
+This feature was introduced in Loki v2.9.0.
+
+In Loki4j you can configure structured metadata as a Logback pattern similar to [labels](labels.md):
+
+```xml
+<appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
+    ...
+    <format>
+        <label>
+            <!-- Logback pattern for labels -->
+            <pattern>
+                app = my-app,
+                host = ${HOSTNAME}
+            </pattern>
+            <!-- Logback pattern for structured metadata -->
+            <structuredMetadataPattern>
+                level = %level,
+                thread = %thread,
+                class = %logger,
+                traceId = %mdc{traceId:-none}
+            </structuredMetadataPattern>
+        </label>
+        <staticLabels>true</staticLabels>
+        ...
+    </format>
+</appender>
+```
+
+Settings from label section, such as `pairSeparator`, `keyValueSeparator`, and `readMarkers` apply to both labels and structure metadata.
+
+Similarly to labels, if `readMarkers` flag is enabled, you can attach structured metadata to a particular log message dynamically from your Java code using SLF4J marker:
+
+```java
+import com.github.loki4j.slf4j.marker.StructuredMetadataMarker;
+
+...
+
+void handleException(Exception ex) {
+    var marker = StructuredMetadataMarker.of("exceptionClass", () -> ex.getClass().getSimpleName());
+    log.error(marker, "Unexpected error", ex);
+}
+```