Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how timestamps set in Kafka messages affect retention #30025

Merged
merged 2 commits into from
Jan 26, 2022
+2 −2
Merged

Document how timestamps set in Kafka messages affect retention #30025

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3826037
Document how timestamps set in Kafka messages affect retention
rdner Jan 26, 2022
bb72882
Apply the suggestion
rdner Jan 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions libbeat/outputs/kafka/docs/kafka.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ To use this output, edit the {beatname_uc} configuration file to disable the {es
output by commenting it out, and enable the Kafka output by uncommenting the
Kafka section.

NOTE: For Kafka version 0.10.0.0+ the message creation timestamp is set by beats and equals to the initial timestamp of the event. This affects the retention policy in Kafka: for example, if a beat event was created 2 weeks ago, the retention policy is set to 7 days and the message from beats arrives to Kafka today, it's going to be immediately discarded since the timestamp value is before the last 7 days. It's possible to change this behavior by setting timestamps on message arrival instead, so the message is not discarded but kept for 7 more days. To do that, please set `log.message.timestamp.type` to `LogAppendTime` (default `CreateTime`) in the Kafka configuration.

Example configuration:

[source,yaml]
Expand Down Expand Up @@ -62,8 +64,6 @@ The cluster metadata contain the actual Kafka brokers events are published to.

Kafka version {beatname_lc} is assumed to run against. Defaults to 1.0.0.

Event timestamps will be added, if version 0.10.0.0+ is enabled.

Valid values are all kafka releases in between `0.8.2.0` and `2.0.0`.

See <<kafka-compatibility>> for information on supported versions.
Expand Down