systemd input plugin cannot read zstd compressed/hash collision hardened journal files in systemd >= 246 #2998
Description
Bug Report
Describe the bug
systemd 246 changed the default format of journal files:
* systemd-journald gained support for zstd compression of large fields
in journal files. The hash tables in journal files have been hardened
against hash collisions. This is an incompatible change and means
that journal files created with new systemd versions are not readable
with old versions. If the $SYSTEMD_JOURNAL_KEYED_HASH boolean
environment variable for systemd-journald.service is set to 0 this
new hardening functionality may be turned off, so that generated
journal files remain compatible with older journalctl
implementations.
See https://github.com/systemd/systemd/blob/v246/NEWS#L323-L331 and https://www.freedesktop.org/software/systemd/man/journald.conf.html.
The systemd input plugin is unable to read these files unless SYSTEMD_JOURNAL_KEYED_HASH=0 (to disable the hash table hardening) and Compress=false (to disable compression of large fields).
To Reproduce
- Steps to reproduce the problem:
- Configure a systemd input plugin in fluent bit
- Run fluent bit on a system using systemd >= 246 without
SYSTEMD_JOURNAL_KEYED_HASH=0andCompress=false - Observe that no records are emitted to output plugins and that metrics show no records/bytes are being ingested by the input plugin.
Expected behavior
The systemd input plugin should be able to parse any valid journal file format. Alternatively, the behavior and workaround should be documented in the input plugin's documentation.
Screenshots
Your Environment
- Version used: 1.5.4
- Configuration:
- Environment name and version (e.g. Kubernetes? What version?):
- Server type and version:
- Operating System and version: Flatcar Linux 2605.9.0+
- Filters and plugins: systemd input
Additional context
This was tricky to spot since Fluent Bit did not log any errors attempting to read the new files. The best way to detect this is to examine the input plugin metrics.
We're running an older version of Fluent Bit but the systemd input plugin hasn't changed significantly since that version.