From 83e2692574fa554640626d507b8a08a0b6159d18 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Fri, 17 May 2024 01:12:48 +0300 Subject: [PATCH 01/15] Protobuf-related changes --- docs/docus/docs/compatibility.md | 2 +- docs/docus/docs/configuration.md | 2 +- docs/docus/docs/jsonlayout.md | 2 +- docs/docus/docs/migration.md | 9 ++++++++- docs/docus/docs/performance.md | 2 +- docs/docus/docs/protobuf.md | 4 ++-- 6 files changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/docus/docs/compatibility.md b/docs/docus/docs/compatibility.md index 453d9a96..1668c002 100644 --- a/docs/docus/docs/compatibility.md +++ b/docs/docus/docs/compatibility.md @@ -1,6 +1,6 @@ --- id: compatibility -title: Loki4j Compatibility Matrix +title: Loki4j compatibility matrix sidebar_label: Compatibility Matrix --- diff --git a/docs/docus/docs/configuration.md b/docs/docus/docs/configuration.md index f074fcf1..80938460 100644 --- a/docs/docus/docs/configuration.md +++ b/docs/docus/docs/configuration.md @@ -1,6 +1,6 @@ --- id: configuration -title: Loki4j Configuration +title: Loki4j configuration sidebar_label: Configuration --- diff --git a/docs/docus/docs/jsonlayout.md b/docs/docus/docs/jsonlayout.md index 6cd96d55..22f53b26 100644 --- a/docs/docus/docs/jsonlayout.md +++ b/docs/docus/docs/jsonlayout.md @@ -1,6 +1,6 @@ --- id: jsonlayout -title: Configuring JSON Message Layout +title: Configuring JSON message layout sidebar_label: JSON Message Layout --- diff --git a/docs/docus/docs/migration.md b/docs/docus/docs/migration.md index 12af0a93..9fc98664 100644 --- a/docs/docus/docs/migration.md +++ b/docs/docus/docs/migration.md @@ -1,9 +1,16 @@ --- id: migration -title: Loki4j Migration Guide +title: Loki4j migration guide sidebar_label: Migration Guide --- +## Upgrading from 1.5.x to 1.6.x + +#### loki-protobuf version updated to 0.0.2 + +Loki `.proto` files were updated to the latest version from the upstream. +If you use protobuf API for sending logs to Loki, please switch to loki-protobuf v0.0.2. + ## Upgrading from 1.4.x to 1.5.x The most significant breaking change in Loki4j v1.5.0 is an upgrade to Logback v1.3.x. diff --git a/docs/docus/docs/performance.md b/docs/docus/docs/performance.md index 9329d3ab..7cf6faf9 100644 --- a/docs/docus/docs/performance.md +++ b/docs/docus/docs/performance.md @@ -1,6 +1,6 @@ --- id: performance -title: Performance Monitoring +title: Performance monitoring sidebar_label: Monitoring --- diff --git a/docs/docus/docs/protobuf.md b/docs/docus/docs/protobuf.md index 82bf8313..44c48b61 100644 --- a/docs/docus/docs/protobuf.md +++ b/docs/docus/docs/protobuf.md @@ -16,14 +16,14 @@ If you want to use `ProtobufEncoder`, you need to add the following dependency t com.github.loki4j loki-protobuf - 0.0.1_pb3.24.0 + 0.0.2_pb4.26.0 ``` ```groovy -implementation 'com.github.loki4j:loki-protobuf:0.0.1_pb3.24.0' +implementation 'com.github.loki4j:loki-protobuf:0.0.2_pb4.26.0' ``` From fef66b4335922c8366a05c70ed871f36a11feefb Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Fri, 17 May 2024 21:01:53 +0300 Subject: [PATCH 02/15] No Java 8 support --- docs/docus/docs/apacheclient.md | 4 ++-- docs/docus/docs/compatibility.md | 1 + docs/docus/docs/migration.md | 5 +++++ 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/docus/docs/apacheclient.md b/docs/docus/docs/apacheclient.md index e5864897..f84ecd00 100644 --- a/docs/docus/docs/apacheclient.md +++ b/docs/docus/docs/apacheclient.md @@ -4,11 +4,11 @@ title: Using Apache HttpClient sidebar_label: Apache HttpClient --- -By default, Loki4j uses `JavaHttpSender`, backed by `java.net.http.HttpClient` available in Java 11 and later. +By default, Loki4j uses `JavaHttpSender`, backed by `java.net.http.HttpClient`. This sender does not require any extra dependencies. So, it should be a good fit for most users. -However, you may want to switch to `ApacheHttpSender`, backed by `org.apache.http.client.HttpClient` available for Java 8+ projects. +However, you may want to switch to `ApacheHttpSender`, backed by `org.apache.http.client.HttpClient`. In this case, you need to ensure you have added the required dependencies to your project: diff --git a/docs/docus/docs/compatibility.md b/docs/docus/docs/compatibility.md index 1668c002..b2a2aac9 100644 --- a/docs/docus/docs/compatibility.md +++ b/docs/docus/docs/compatibility.md @@ -8,6 +8,7 @@ The versions of Loki4j that had introduced backward-incompatible platform upgrad |Loki4j|Java|Logback| |-------|-------|-----------| +|v1.6.0|Java 11+|v1.4.x| |v1.5.0||v1.3.x| |v0.3.0|[Java 8](#java-8-support), Java 11+|| |v0.1.0|Java 11+|v1.2.x| diff --git a/docs/docus/docs/migration.md b/docs/docus/docs/migration.md index 9fc98664..e956f5a2 100644 --- a/docs/docus/docs/migration.md +++ b/docs/docus/docs/migration.md @@ -6,6 +6,11 @@ sidebar_label: Migration Guide ## Upgrading from 1.5.x to 1.6.x +#### No Java 8 support + +Starting from v1.6.0, Loki4j no longer provides `-jdk8` artifact. +Please upgrade your project at least to Java 11 before switching to Loki4j v1.6.0. + #### loki-protobuf version updated to 0.0.2 Loki `.proto` files were updated to the latest version from the upstream. From 9d75a6c2705a9b48d858d84e7a524b0620803efc Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Fri, 17 May 2024 21:16:56 +0300 Subject: [PATCH 03/15] Logback version upgrade --- docs/docus/docs/index.md | 2 +- docs/docus/docs/migration.md | 4 ++++ docs/docus/docs/performance.md | 4 ++-- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/docus/docs/index.md b/docs/docus/docs/index.md index 8f760f77..98e3ca8b 100644 --- a/docs/docus/docs/index.md +++ b/docs/docus/docs/index.md @@ -6,7 +6,7 @@ configuration and enjoy. ### Quick Start -The current stable version of Loki4j requires Java 11+ and Logback v1.3.x. +The current stable version of Loki4j requires Java 11+ and Logback v1.4.x. See the [compatibility matrix](docs/compatibility) for more information about older versions' support. Add the following dependency to your project: diff --git a/docs/docus/docs/migration.md b/docs/docus/docs/migration.md index e956f5a2..868b4585 100644 --- a/docs/docus/docs/migration.md +++ b/docs/docus/docs/migration.md @@ -11,6 +11,10 @@ sidebar_label: Migration Guide Starting from v1.6.0, Loki4j no longer provides `-jdk8` artifact. Please upgrade your project at least to Java 11 before switching to Loki4j v1.6.0. +#### Logback version switched to 1.4.x + +If your project depends on other external Logback appenders, please make sure all of them are compatible with Logback v1.4.x before upgrading. + #### loki-protobuf version updated to 0.0.2 Loki `.proto` files were updated to the latest version from the upstream. diff --git a/docs/docus/docs/performance.md b/docs/docus/docs/performance.md index 7cf6faf9..48dbe75b 100644 --- a/docs/docus/docs/performance.md +++ b/docs/docus/docs/performance.md @@ -15,14 +15,14 @@ First, you need to make sure that Micrometer dependency is added to your project io.micrometer micrometer-core - 1.12.2 + 1.13.0 ``` ```groovy -implementation 'io.micrometer:micrometer-core:1.12.2' +implementation 'io.micrometer:micrometer-core:1.13.0' ``` From 6359cff45597961b7afae85c274bda5b1ce537fb Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Thu, 3 Oct 2024 01:28:10 +0300 Subject: [PATCH 04/15] Lower supported version of Loki is updated --- docs/docus/docs/compatibility.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/docus/docs/compatibility.md b/docs/docus/docs/compatibility.md index e6e0b7d2..aae80fd2 100644 --- a/docs/docus/docs/compatibility.md +++ b/docs/docus/docs/compatibility.md @@ -6,12 +6,12 @@ sidebar_label: Compatibility Matrix The versions of Loki4j that had introduced backward-incompatible platform upgrades are listed in the table below. -|Loki4j|Java|Logback| -|-------|-------|-----------| -|v1.6.0|Java 11+|v1.4.x| -|v1.5.0||v1.3.x| -|v0.3.0|[Java 8](#java-8-support), Java 11+|| -|v0.1.0|Java 11+|v1.2.x| +|Loki4j|Java|Logback|Loki| +|------|----|-------|----| +|v1.6.0|Java 11+|v1.4.x|v2.8.0| +|v1.5.0||v1.3.x|| +|v0.3.0|[Java 8](#java-8-support), Java 11+||| +|v0.1.0|Java 11+|v1.2.x|v1.6.1| ### Java 8 support From ecad632e1569fadb99daad9005479bc321369112 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Thu, 3 Oct 2024 01:45:27 +0300 Subject: [PATCH 05/15] Migration instruction updated --- docs/docus/docs/migration.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/docus/docs/migration.md b/docs/docus/docs/migration.md index 868b4585..cdef9f1b 100644 --- a/docs/docus/docs/migration.md +++ b/docs/docus/docs/migration.md @@ -20,6 +20,12 @@ If your project depends on other external Logback appenders, please make sure al Loki `.proto` files were updated to the latest version from the upstream. If you use protobuf API for sending logs to Loki, please switch to loki-protobuf v0.0.2. +#### Loki lowest supported version updated to 2.8.0 + +Loki v2.8.0 is the first version that supports label drop functionality that is now mandatory for Loki4j's integration tests. +Because of that Loki v1.6.1 was excluded from the integration tests and the compatibility matrix was updated. +However, Loki4j should still work fine with versions prior to 2.8.0. + ## Upgrading from 1.4.x to 1.5.x The most significant breaking change in Loki4j v1.5.0 is an upgrade to Logback v1.3.x. From 277669134273b90ecc4fa94b6f46c32b0c6d2271 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Thu, 3 Oct 2024 01:48:05 +0300 Subject: [PATCH 06/15] Artefact version updated --- docs/docus/website/siteConfig.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docus/website/siteConfig.js b/docs/docus/website/siteConfig.js index 79112b98..ec8724c6 100644 --- a/docs/docus/website/siteConfig.js +++ b/docs/docus/website/siteConfig.js @@ -112,7 +112,7 @@ const siteConfig = { repoUrl: ghUrl, startDoc: defaultDoc, - artifactVersion: '1.5.2', + artifactVersion: '1.6.0', }; module.exports = siteConfig; From c6141eb68c4a4aaf62bca2a6a72b649816af0630 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Fri, 1 Nov 2024 23:20:36 +0200 Subject: [PATCH 07/15] KV rework --- docs/docus/docs/configuration.md | 3 +-- docs/docus/docs/labels.md | 18 ++++++------------ docs/docus/docs/migration.md | 11 +++++++++++ docs/docus/docs/olddocs.md | 1 + docs/docus/docs/performance.md | 4 ++-- 5 files changed, 21 insertions(+), 16 deletions(-) diff --git a/docs/docus/docs/configuration.md b/docs/docus/docs/configuration.md index 80938460..4743211e 100644 --- a/docs/docus/docs/configuration.md +++ b/docs/docus/docs/configuration.md @@ -65,10 +65,9 @@ Format settings do not depend on the encoding you use. |Setting|Default|Description| |-------|-------|-----------| |format.label.pattern||**Required**. Logback pattern to use for log record's label| -|format.label.pairSeparator|,|Character sequence to use as a separator between labels. If it starts with the "regex:" prefix, the remainder is applied as a regular expression separator. Otherwise, the provided char sequence is used as a separator literally| +|format.label.pairSeparator|,|Character sequence to use as a separator between labels| |format.label.keyValueSeparator|=|Character to use as a separator between label's name and its value| |format.label.readMarkers|false|If true, Loki4j scans each log record for the attached LabelMarker to add its values to the record's labels| -|format.label.nopex|true|If true, exception info is not added to labels. If false, you should take care of proper formatting| |format.label.streamCache|BoundAtomicMapCache|An implementation of a stream cache to use. By default, caches up to 1000 unique label sets| |format.staticLabels|false|If you use only a constant label set (e.g., same keys and values) for all log records, you can set this flag to true and save some CPU time on grouping records by label| |format.sortByTime|false|If true, log records in batch are sorted by timestamp. If false, records will be sent to Loki in arrival order. Enable this if you see an 'entry out of order' error from Loki| diff --git a/docs/docus/docs/labels.md b/docs/docus/docs/labels.md index 1e553151..8f004735 100644 --- a/docs/docus/docs/labels.md +++ b/docs/docus/docs/labels.md @@ -33,26 +33,20 @@ By default labels are defined as `key=value` pairs separated by commas. ``` -But you can override `pairSeparator` to organize them in a different way. -For example, if you have many labels, it's better to have each of them on a separate line: +If you have many labels, you can put them in multiple lines (a trailing comma is optional): ```xml ``` -Please note that in the example above the regular expression in `pairSeparator` defines lines starting with `//` a part of a separator. -So now we have a `// comment` feature here as well. - ## Using MDC in labels `label.pattern` is nothing but Logback's [pattern layout](https://logback.qos.ch/manual/layouts.html#ClassicPatternLayout), which means it supports [MDC](https://logback.qos.ch/manual/mdc.html): diff --git a/docs/docus/docs/migration.md b/docs/docus/docs/migration.md index cdef9f1b..e6366832 100644 --- a/docs/docus/docs/migration.md +++ b/docs/docus/docs/migration.md @@ -26,6 +26,17 @@ Loki v2.8.0 is the first version that supports label drop functionality that is Because of that Loki v1.6.1 was excluded from the integration tests and the compatibility matrix was updated. However, Loki4j should still work fine with versions prior to 2.8.0. +#### Regex pair separators deprecated + +Now you can use multiline strings to declare your key-value pairs in label and structured metadata patterns. +From now on, this will work even if the pair separator is a character (by default it's `,`) or a literal string. +A prefix `regex:` is still supported in `format.label.pairSeparator` for compatibility, but it might be removed in future versions. + +#### 'nopex' label setting removed + +Previously `format.label.nopex` was used to suppress exception output into label pattern. +In v1.6.0 we have re-worked label formatting code, so that this setting is no longer needed. + ## Upgrading from 1.4.x to 1.5.x The most significant breaking change in Loki4j v1.5.0 is an upgrade to Logback v1.3.x. diff --git a/docs/docus/docs/olddocs.md b/docs/docus/docs/olddocs.md index f0c37d2d..dcc09c3a 100644 --- a/docs/docus/docs/olddocs.md +++ b/docs/docus/docs/olddocs.md @@ -9,6 +9,7 @@ It should not take much time, just follow the [Migration Guide](migration). However, the docs for previous Loki4j versions are still available on GitHub: +- [v1.5.0](https://github.com/loki4j/loki-logback-appender/tree/v1.5.0/docs/docus/docs) - [v1.4.0](https://github.com/loki4j/loki-logback-appender/tree/v1.4.0/docs/docus/docs) - [v1.3.0](https://github.com/loki4j/loki-logback-appender/tree/v1.3.0/docs/docus/docs) - [v1.2.0](https://github.com/loki4j/loki-logback-appender/tree/v1.2.0/docs/docus/docs) diff --git a/docs/docus/docs/performance.md b/docs/docus/docs/performance.md index 48dbe75b..62a68c5e 100644 --- a/docs/docus/docs/performance.md +++ b/docs/docus/docs/performance.md @@ -15,14 +15,14 @@ First, you need to make sure that Micrometer dependency is added to your project io.micrometer micrometer-core - 1.13.0 + 1.13.6 ``` ```groovy -implementation 'io.micrometer:micrometer-core:1.13.0' +implementation 'io.micrometer:micrometer-core:1.13.6' ``` From fa9961a73db7819760f1fa4f81f1d95dea07f585 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Mon, 4 Nov 2024 00:39:43 +0200 Subject: [PATCH 08/15] sortByTime removed --- docs/docus/docs/configuration.md | 1 - docs/docus/docs/migration.md | 13 ++++++++++--- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/docus/docs/configuration.md b/docs/docus/docs/configuration.md index 4743211e..b8c9f67e 100644 --- a/docs/docus/docs/configuration.md +++ b/docs/docus/docs/configuration.md @@ -70,7 +70,6 @@ Format settings do not depend on the encoding you use. |format.label.readMarkers|false|If true, Loki4j scans each log record for the attached LabelMarker to add its values to the record's labels| |format.label.streamCache|BoundAtomicMapCache|An implementation of a stream cache to use. By default, caches up to 1000 unique label sets| |format.staticLabels|false|If you use only a constant label set (e.g., same keys and values) for all log records, you can set this flag to true and save some CPU time on grouping records by label| -|format.sortByTime|false|If true, log records in batch are sorted by timestamp. If false, records will be sent to Loki in arrival order. Enable this if you see an 'entry out of order' error from Loki| #### Plain text message layout diff --git a/docs/docus/docs/migration.md b/docs/docus/docs/migration.md index e6366832..b3539d65 100644 --- a/docs/docus/docs/migration.md +++ b/docs/docus/docs/migration.md @@ -20,11 +20,18 @@ If your project depends on other external Logback appenders, please make sure al Loki `.proto` files were updated to the latest version from the upstream. If you use protobuf API for sending logs to Loki, please switch to loki-protobuf v0.0.2. -#### Loki lowest supported version updated to 2.8.0 +#### Minimal supported Loki version is 2.8.0 Loki v2.8.0 is the first version that supports label drop functionality that is now mandatory for Loki4j's integration tests. Because of that Loki v1.6.1 was excluded from the integration tests and the compatibility matrix was updated. -However, Loki4j should still work fine with versions prior to 2.8.0. +However, in most cases Loki4j should still work fine with versions prior to 2.8.0. + +#### "sortByTime" format setting removed + +Since Loki v2.4.0 'entry out of order' is no longer an issue. +Now that minimal supported Loki version is 2.8.0, there is no point in keeping `format.sortByTime` setting. +Furthermore, having this property set to `true` might impose negative effect on performance. +That's why `format.sortByTime` is removed in v1.6.0. #### Regex pair separators deprecated @@ -32,7 +39,7 @@ Now you can use multiline strings to declare your key-value pairs in label and s From now on, this will work even if the pair separator is a character (by default it's `,`) or a literal string. A prefix `regex:` is still supported in `format.label.pairSeparator` for compatibility, but it might be removed in future versions. -#### 'nopex' label setting removed +#### "nopex" label setting removed Previously `format.label.nopex` was used to suppress exception output into label pattern. In v1.6.0 we have re-worked label formatting code, so that this setting is no longer needed. From b3c4ecd0a058a0ecab00847dd7a6a3eca339ae82 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Wed, 13 Nov 2024 23:20:40 +0200 Subject: [PATCH 09/15] Structured metadata --- docs/docus/docs/configuration.md | 3 +- docs/docus/docs/index.md | 15 ++++++++-- docs/docus/docs/jsonlayout.md | 18 ++++++++++++ docs/docus/docs/labels.md | 21 +++++++++++++- docs/docus/docs/metadata.md | 50 ++++++++++++++++++++++++++++++++ docs/docus/docs/migration.md | 2 +- docs/docus/website/sidebars.json | 1 + 7 files changed, 105 insertions(+), 5 deletions(-) create mode 100644 docs/docus/docs/metadata.md diff --git a/docs/docus/docs/configuration.md b/docs/docus/docs/configuration.md index b8c9f67e..96c7b610 100644 --- a/docs/docus/docs/configuration.md +++ b/docs/docus/docs/configuration.md @@ -65,11 +65,12 @@ Format settings do not depend on the encoding you use. |Setting|Default|Description| |-------|-------|-----------| |format.label.pattern||**Required**. Logback pattern to use for log record's label| +|format.label.structuredMetadataPattern||Logback pattern to use for log record's structured metadata| |format.label.pairSeparator|,|Character sequence to use as a separator between labels| |format.label.keyValueSeparator|=|Character to use as a separator between label's name and its value| |format.label.readMarkers|false|If true, Loki4j scans each log record for the attached LabelMarker to add its values to the record's labels| |format.label.streamCache|BoundAtomicMapCache|An implementation of a stream cache to use. By default, caches up to 1000 unique label sets| -|format.staticLabels|false|If you use only a constant label set (e.g., same keys and values) for all log records, you can set this flag to true and save some CPU time on grouping records by label| +|format.staticLabels|false|If you use only a constant label set (e.g., same keys and values) for all log records, you can set this flag to true and save some CPU and RAM| #### Plain text message layout diff --git a/docs/docus/docs/index.md b/docs/docus/docs/index.md index 98e3ca8b..56a7f780 100644 --- a/docs/docus/docs/index.md +++ b/docs/docus/docs/index.md @@ -40,10 +40,21 @@ Then add Loki appender to your `logback.xml`: - %-5level [%.5(${HOSTNAME})] %.10thread %logger{20} | %msg %ex + %-5level %logger{20} %msg %ex diff --git a/docs/docus/docs/jsonlayout.md b/docs/docus/docs/jsonlayout.md index 22f53b26..0e2b0df0 100644 --- a/docs/docus/docs/jsonlayout.md +++ b/docs/docus/docs/jsonlayout.md @@ -4,6 +4,24 @@ title: Configuring JSON message layout sidebar_label: JSON Message Layout --- +## When to use JSON message layout + +JSON is not Loki's native format for logs. +Producing and parsing log lines in JSON format imposes significant performance penalties comparing to the plain text layout. +JSON is harder to read for humans as well. + +Having that said, Loki4j's JSON message layout probably has no alternatives if you: + +- can not specify what exact keys to extract from MDC or KV; +- need nested structures in your metadata; +- have to export your log messages from Loki to another tool that requires JSON. + +If you are not dealing with something from the list above or similar, +we recommend you to take a look at other ways for attaching metadata to log records: [labels](labels.md) and [structured metadata](metadata.md). +Both of them work perfectly with plain text layout. + +Now, if you are still sure JSON layout is the right option for you, please proceed to the next section. + ## Enabling the JSON layout You can enable JSON layout for log messages by specifying a corresponding `class` attribute for a `message` section: diff --git a/docs/docus/docs/labels.md b/docs/docus/docs/labels.md index 8f004735..afb849b0 100644 --- a/docs/docus/docs/labels.md +++ b/docs/docus/docs/labels.md @@ -82,4 +82,23 @@ void handleException(Exception ex) { var marker = LabelMarker.of("exceptionClass", () -> ex.getClass().getSimpleName()); log.error(marker, "Unexpected error", ex); } -``` \ No newline at end of file +``` + +## Best practices + +We encourage you to follow the [Label best practices](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/) collected by Grafana Loki team. Loki4j provides several settings to facilitate these recommendations. + +First, make sure you have `format.staticLabels` flag enabled. +This will prevent Loki4j from calculating labels for each particular log record: + +```xml + + ... + + true + ... + + +``` + +Second, make sure you put all the high-cardinality metadata to [structured metadata](metadata.md) instead of labels. diff --git a/docs/docus/docs/metadata.md b/docs/docus/docs/metadata.md new file mode 100644 index 00000000..504450d7 --- /dev/null +++ b/docs/docus/docs/metadata.md @@ -0,0 +1,50 @@ +--- +id: metadata +title: Structured metadata +sidebar_label: Structured Metadata +--- + +[Structured metadata](https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/) is a way to attach high-cardinality metadata to logs without indexing them or including them in the log line content itself. +Structured metadata is a set of key-value pairs; both keys and values must be plain strings. +This feature was introduced in Loki v2.9.0. + +In Loki4j you can configure structured metadata as a Logback pattern similar to [labels](labels.md): + +```xml + + ... + + + true + ... + + +``` + +Settings from label section, such as `pairSeparator`, `keyValueSeparator`, and `readMarkers` apply to both labels and structure metadata. + +Similarly to labels, if `readMarkers` flag is enabled, you can attach structured metadata to a particular log message dynamically from your Java code using SLF4J marker: + +```java +import com.github.loki4j.slf4j.marker.StructuredMetadataMarker; + +... + +void handleException(Exception ex) { + var marker = StructuredMetadataMarker.of("exceptionClass", () -> ex.getClass().getSimpleName()); + log.error(marker, "Unexpected error", ex); +} +``` diff --git a/docs/docus/docs/migration.md b/docs/docus/docs/migration.md index b3539d65..e7b16a10 100644 --- a/docs/docus/docs/migration.md +++ b/docs/docus/docs/migration.md @@ -30,7 +30,7 @@ However, in most cases Loki4j should still work fine with versions prior to 2.8. Since Loki v2.4.0 'entry out of order' is no longer an issue. Now that minimal supported Loki version is 2.8.0, there is no point in keeping `format.sortByTime` setting. -Furthermore, having this property set to `true` might impose negative effect on performance. +Furthermore, having this property set to `true` in some cases might impose negative effect on performance. That's why `format.sortByTime` is removed in v1.6.0. #### Regex pair separators deprecated diff --git a/docs/docus/website/sidebars.json b/docs/docus/website/sidebars.json index 5cbe54c6..85e50227 100644 --- a/docs/docus/website/sidebars.json +++ b/docs/docus/website/sidebars.json @@ -3,6 +3,7 @@ "Logback": [ "configuration", "labels", + "metadata", "jsonlayout", "protobuf", "apacheclient", From d66861c73492dd14343b53ea3da533df7a1cff38 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Sat, 14 Dec 2024 23:46:52 +0200 Subject: [PATCH 10/15] When to use json improved --- README.md | 3 ++- docs/docus/docs/jsonlayout.md | 15 +++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index dc69b6d9..6c800d3e 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,8 @@ This project is unofficial and community-driven. Please proceed to the microsite for more information: - [Quick Start](https://loki4j.github.io/loki-logback-appender/#quick-start) -- [Configuration Guide](https://loki4j.github.io/loki-logback-appender/docs/configuration) +- [Configuration Reference](https://loki4j.github.io/loki-logback-appender/docs/configuration) +- [Migration Guide](https://loki4j.github.io/loki-logback-appender/docs/migration) If you have found this project helpful, please drop a :star:! diff --git a/docs/docus/docs/jsonlayout.md b/docs/docus/docs/jsonlayout.md index 0e2b0df0..a956cf5d 100644 --- a/docs/docus/docs/jsonlayout.md +++ b/docs/docus/docs/jsonlayout.md @@ -6,18 +6,17 @@ sidebar_label: JSON Message Layout ## When to use JSON message layout -JSON is not Loki's native format for logs. -Producing and parsing log lines in JSON format imposes significant performance penalties comparing to the plain text layout. -JSON is harder to read for humans as well. +Loki4j supports JSON layout for log messages, however it is not Loki's native format for logs. +Producing and parsing log lines in JSON imposes performance penalties and extra cost comparing to the plain text layout. +Also worth mentioning that JSON is harder to read for humans than the plain text as well. -Having that said, Loki4j's JSON message layout probably has no alternatives if you: +Having that said, Loki4j's JSON message layout has no alternatives if you: -- can not specify what exact keys to extract from MDC or KV; +- heavily use MDC or KV in multiple dynamic contexts and need more flexibility to configure them; - need nested structures in your metadata; -- have to export your log messages from Loki to another tool that requires JSON. +- have tools in your logging stack (apart from Loki) that require JSON format. -If you are not dealing with something from the list above or similar, -we recommend you to take a look at other ways for attaching metadata to log records: [labels](labels.md) and [structured metadata](metadata.md). +So before you decide to use JSON layout, we recommend you to take a look at other, more straight-forward ways for attaching metadata to log records: [labels](labels.md) and [structured metadata](metadata.md). Both of them work perfectly with plain text layout. Now, if you are still sure JSON layout is the right option for you, please proceed to the next section. From 0946e98dc99171399b693c48a703f0db4dbc9040 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Sun, 15 Dec 2024 00:03:06 +0200 Subject: [PATCH 11/15] Structured metadata added to features --- README.md | 1 + docs/docus/docs/index.md | 7 +++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6c800d3e..3e2dceec 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,7 @@ If you have found this project helpful, please drop a :star:! ## Key features +- Structured metadata support - Flexible management of Loki labels using MDC and SLF4J Markers - Out-of-the-box JSON layout support for log message formatting - Logback plain text formatting patterns can be used for both labels and messages diff --git a/docs/docus/docs/index.md b/docs/docus/docs/index.md index 56a7f780..57013ba2 100644 --- a/docs/docus/docs/index.md +++ b/docs/docus/docs/index.md @@ -91,8 +91,11 @@ Migrating from the previous Loki4j version? Read the [Migration Guide](docs/migr ### Key Features: -- **Flexible management of Loki labels using MDC and SLF4J Markers.** -You can specify Loki labels dynamically for any set of log records and even on a per-record basis. +- **Structured metadata support.** +Pass any non-label metadata along with your log lines using [structured metadata](docs/metadata). + +- **Flexible management of Loki labels and metadata using MDC and SLF4J Markers.** +You can specify Loki labels as well as structured metadata dynamically, even on a per-record basis. [Learn more...](docs/labels) - **Out-of-the-box JSON layout support for log message formatting.** From d729b76939311eef3460d5ce8ba13b9f82610a54 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Sun, 15 Dec 2024 00:35:05 +0200 Subject: [PATCH 12/15] Deps upgraded --- docs/docus/docs/performance.md | 4 ++-- docs/docus/docs/protobuf.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/docus/docs/performance.md b/docs/docus/docs/performance.md index 62a68c5e..53a6b1a9 100644 --- a/docs/docus/docs/performance.md +++ b/docs/docus/docs/performance.md @@ -15,14 +15,14 @@ First, you need to make sure that Micrometer dependency is added to your project io.micrometer micrometer-core - 1.13.6 + 1.14.2 ``` ```groovy -implementation 'io.micrometer:micrometer-core:1.13.6' +implementation 'io.micrometer:micrometer-core:1.14.2' ``` diff --git a/docs/docus/docs/protobuf.md b/docs/docus/docs/protobuf.md index 44c48b61..cd43d566 100644 --- a/docs/docus/docs/protobuf.md +++ b/docs/docus/docs/protobuf.md @@ -16,14 +16,14 @@ If you want to use `ProtobufEncoder`, you need to add the following dependency t com.github.loki4j loki-protobuf - 0.0.2_pb4.26.0 + 0.0.2_pb4.29.0 ``` ```groovy -implementation 'com.github.loki4j:loki-protobuf:0.0.2_pb4.26.0' +implementation 'com.github.loki4j:loki-protobuf:0.0.2_pb4.29.0' ``` From 411ab7fc5b82f415c1f2313bac3d8267b791fc20 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Mon, 16 Dec 2024 01:26:35 +0200 Subject: [PATCH 13/15] MDC properties migration --- docs/docus/docs/configuration.md | 3 ++- docs/docus/docs/migration.md | 4 ++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/docus/docs/configuration.md b/docs/docus/docs/configuration.md index 96c7b610..c0733900 100644 --- a/docs/docus/docs/configuration.md +++ b/docs/docus/docs/configuration.md @@ -95,7 +95,8 @@ This layout has the following settings: |format.message.logLevel.enabled|true|Enable logLevel provider| |format.message.logLevel.fieldName|level|A JSON field name to use for logLevel| |format.message.mdc.enabled|true|Enable MDC provider| -|format.message.mdc.fieldName|mdc_|A prefix added to each JSON field name written by this provider| +|format.message.mdc.prefix|mdc_|A prefix added to each JSON field name written by this provider| +|format.message.mdc.noPrefix|false|Whether to omit prefix for this provider| |format.message.mdc.include||A set of MDC keys to include in JSON payload. If not specified, all keys are included| |format.message.mdc.exclude||A set of MDC keys to exclude from JSON payload. The exclude list has precedence over the include list. If not specified, all keys are included| |format.message.message.enabled|true|Enable message provider| diff --git a/docs/docus/docs/migration.md b/docs/docus/docs/migration.md index e7b16a10..4f3dad0d 100644 --- a/docs/docus/docs/migration.md +++ b/docs/docus/docs/migration.md @@ -44,6 +44,10 @@ A prefix `regex:` is still supported in `format.label.pairSeparator` for compati Previously `format.label.nopex` was used to suppress exception output into label pattern. In v1.6.0 we have re-worked label formatting code, so that this setting is no longer needed. +#### "mdc.fieldName" setting replaced with "mdc.prefix" + +JSON message layout setting `format.message.mdc.fieldName` is replaced with `format.message.mdc.prefix` as it better reflects its semantics. + ## Upgrading from 1.4.x to 1.5.x The most significant breaking change in Loki4j v1.5.0 is an upgrade to Logback v1.3.x. From 59a1379f5f4da83cf5cb13f2d5de8a2fa4ad50e1 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Wed, 18 Dec 2024 00:04:16 +0200 Subject: [PATCH 14/15] KVP is added --- docs/docus/docs/configuration.md | 6 ++ docs/docus/docs/jsonlayout.md | 114 +++++++++++++++++++++++++++++++ 2 files changed, 120 insertions(+) diff --git a/docs/docus/docs/configuration.md b/docs/docus/docs/configuration.md index c0733900..9cd4b64a 100644 --- a/docs/docus/docs/configuration.md +++ b/docs/docus/docs/configuration.md @@ -94,6 +94,12 @@ This layout has the following settings: |format.message.loggerName.targetLength|-1|The desired target length of logger name: `-1` to disable abbreviation, `0` to print class name only, >`0` to abbreviate to the target length| |format.message.logLevel.enabled|true|Enable logLevel provider| |format.message.logLevel.fieldName|level|A JSON field name to use for logLevel| +|format.message.kvp.enabled|true|Enable keyValuePair provider| +|format.message.kvp.prefix|kvp_|A prefix added to each JSON field name written by this provider| +|format.message.kvp.noPrefix|false|Whether to omit prefix for this provider| +|format.message.kvp.fieldSerializer||An implementation of field JSON serializer. By default, `writeObjectField()` is used| +|format.message.kvp.include||A set of keys to include in JSON payload. If not specified, all keys are included| +|format.message.kvp.exclude||A set of keys to exclude from JSON payload. The exclude list has precedence over the include list. If not specified, all keys are included| |format.message.mdc.enabled|true|Enable MDC provider| |format.message.mdc.prefix|mdc_|A prefix added to each JSON field name written by this provider| |format.message.mdc.noPrefix|false|Whether to omit prefix for this provider| diff --git a/docs/docus/docs/jsonlayout.md b/docs/docus/docs/jsonlayout.md index a956cf5d..3ae4979a 100644 --- a/docs/docus/docs/jsonlayout.md +++ b/docs/docus/docs/jsonlayout.md @@ -135,3 +135,117 @@ You can add as many custom providers as you want: ``` + +For more insights on how `JsonEventWriter` works, please check the next section. + +## Serializing an arbitrary object + +Our recommendation is to use plain structure of the log message (i.e., no nested objects), and only some basic types for fields (e.g.: string, number, boolean). +Following this recommendation will give you both maximum performance and simplicity. + +However, Logback allows you to attach arbitrary objects to you log record as key-value pairs: + +```java +log.atInfo().setMessage("Test KVP") + .addKeyValue("flow", "ingest") + .addKeyValue("msgId", 123456) + .addKeyValue("bool", true) + .addKeyValue("list", List.of(0, 1, 2)) + .addKeyValue("date", ZonedDateTime.now()) + .addKeyValue("obj", new TestJsonKvData(1001L, "admin", UUID.randomUUID())) + .log(); +``` + +Loki4j uses fast and compact JSON serialization algorithm that is [3.5x](https://github.com/loki4j/loki-logback-appender/pull/210#issue-2113540494) faster than logstash-logback-encoder backed by jackson. +But that speed comes with the cost: no runtime reflection is used, no complex type conversion is implemented. +Which means we don't provide a built-in mechanism to serialize arbitrary object into JSON right away. +Instead, we offer you several options, so you can choose the one that better fits your use case. + +Loki4j has `KeyValuePairsJsonProvider`, that by default uses `writeObjectField()` for all key-value pairs. + +#### Default approach: `writeObjectField()` + +Method `writeObjectField()` in class `JsonEventWriter` has the following logic: + +- if value is `String`, it's rendered as JSON string: `"flow":"ingest"`, similar to `writeStringField()`; +- if value is `Integer` or `Long`, it's rendered as JSON number: `"msgId":123456`, similar to `writeNumericField()`; +- if value is `Boolean`, it's rendered as JSON boolean: `"bool":true`; +- if value is `Iterable`, it's rendered as JSON array, `writeObjectField()` to render each element: `"list":[0,1,2]`, similar to `writeArrayField()`; +- if value is `RawJsonString`, its value is rendered as raw JSON without any escaping (see the section below), similar to `writeRawJsonField()`; +- for any other type of value, the result of `toString()` is rendered as JSON string: `"obj":"TestJsonKvData@1de5f259"`. + +#### Direct access to `JsonEventWriter` + +If you write a custom provider, `JsonEventWriter` is exposed to you directly. +You can also configure `KeyValuePairsJsonProvider` to intercept writer calls as well by setting `fieldSerializer`: + +```xml + + ... + + + + +``` + +Your custom serializer must implement interface `JsonFieldSerializer`: + +```java +import com.github.loki4j.logback.json.JsonEventWriter; +import com.github.loki4j.logback.json.JsonFieldSerializer; + +public class TestFieldSerializer implements JsonFieldSerializer { + @Override + public void writeField(JsonEventWriter writer, String fieldName, Object fieldValue) { + if (fieldValue instanceof TestJsonKvData) { + writer.writeCustomField(fieldName, w -> { + var td = (TestJsonKvData)fieldValue; + w.writeBeginObject(); + w.writeObjectField("userId", td.userId); + w.writeFieldSeparator(); + w.writeObjectField("userName", td.userName); + w.writeFieldSeparator(); + w.writeObjectField("sessionId", td.sessionId); + w.writeEndObject(); + } + ); + } else { + writer.writeObjectField(fieldName, fieldValue); + } + } +} +``` + +Instead of `"obj":"TestJsonKvData@1de5f259"`, now you will see `"obj":{"userId":1001,"userName":"admin","sessionId":"92a26b23-9f10-47d8-bb0a-df2b9b15c374"}`. + +#### `RawJsonString` + +If for some reason you can not statically define serialization algorithm for your objects, you can use reflection-based frameworks and put the resulting string into `RawJsonString`. +Writer will render it as is, without any escaping. + +```java +import com.github.loki4j.logback.json.RawJsonString; + +public class TestFieldSerializer implements JsonFieldSerializer { + @Override + public void writeField(JsonEventWriter writer, String fieldName, Object fieldValue) { + writer.writeObjectField(fieldName, new RawJsonString(MyJson.serialize(fieldValue))); + // or + writer.writeRawJsonField(fieldName, MyJson.serialize(fieldValue)); + } +} +``` + +It is your responsibility to make sure that you always put valid JSON into `RawJsonString`. + +#### Custom message layout + +You can use any implementation of `Layout` as a message layout, including third-party JSON layouts: + +```xml + + + +``` + +Please note that Loki4j provides *no support* for any third-party components. From 0dc57efdfc8fdc272c2c16b1e21c1c4bd1f31ed4 Mon Sep 17 00:00:00 2001 From: Anton Nekhaev Date: Wed, 18 Dec 2024 00:09:12 +0200 Subject: [PATCH 15/15] Minor formatting fix --- docs/docus/docs/jsonlayout.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docus/docs/jsonlayout.md b/docs/docus/docs/jsonlayout.md index 3ae4979a..f1fa415d 100644 --- a/docs/docus/docs/jsonlayout.md +++ b/docs/docus/docs/jsonlayout.md @@ -248,4 +248,4 @@ You can use any implementation of `Layout` as a message layout, i ``` -Please note that Loki4j provides *no support* for any third-party components. +Please note that Loki4j provides **no support** for any third-party components.