From 6473ac7af501da5c99cab1c6881c517fd72e4083 Mon Sep 17 00:00:00 2001 From: Robert Fratto Date: Wed, 20 Nov 2019 14:02:46 -0500 Subject: [PATCH 1/2] pkg/promtail: remove journal target forced path The forced default path for the journal target (/var/log/journal) did not apply to all Linux systems: systemd actually defaults to reading from _two_ default paths: /var/log/journal and /run/log/journal. Not setting an explicit default will enable both of those paths to be searched for journal logs. The comment mentioning the local machine ID was technically accurate, but is normally dealt with by bind mounting /etc/machine-id. That's the standard approach for running fluent-bit in a container, and should be the standard approach for Promtail as well. Fixes #1286. --- docs/clients/promtail/configuration.md | 3 +-- pkg/promtail/targets/journaltarget.go | 12 +----------- 2 files changed, 2 insertions(+), 13 deletions(-) diff --git a/docs/clients/promtail/configuration.md b/docs/clients/promtail/configuration.md index fc2237893f90..b8bab6645f26 100644 --- a/docs/clients/promtail/configuration.md +++ b/docs/clients/promtail/configuration.md @@ -576,7 +576,7 @@ labels: [ : ... ] # Path to a directory to read entries from. Defaults to system -# path when empty. +# paths (/var/log/journal and /run/log/journal) when empty. [path: ] ``` @@ -964,7 +964,6 @@ scrape_configs: - job_name: journal journal: max_age: 12h - path: /var/log/journal labels: job: systemd-journal relabel_configs: diff --git a/pkg/promtail/targets/journaltarget.go b/pkg/promtail/targets/journaltarget.go index c4a36aaddb16..321668c48830 100644 --- a/pkg/promtail/targets/journaltarget.go +++ b/pkg/promtail/targets/journaltarget.go @@ -154,18 +154,8 @@ func journalTargetWithReader( return nil, errors.Wrap(err, "parsing journal reader 'max_age' config value") } - // Default to system path if not defined. Passing an empty string to - // sdjournal is valid but forces reads from the journal to be from - // the local machine id only, which contradicts the default behavior - // of when a path is specified. To standardize, we manually default the - // path here. - journalPath := targetConfig.Path - if journalPath == "" { - journalPath = "/var/log/journal" - } - cfg := t.generateJournalConfig(journalConfigBuilder{ - JournalPath: journalPath, + JournalPath: targetConfig.Path, Position: position, MaxAge: maxAge, EntryFunc: entryFunc, From 46e530570908d710b9feb89dd9074e68ae108f65 Mon Sep 17 00:00:00 2001 From: Robert Fratto Date: Mon, 25 Nov 2019 16:16:21 -0500 Subject: [PATCH 2/2] docs: document required bind mounts for reading from systemd journal --- docs/clients/promtail/scraping.md | 34 +++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/docs/clients/promtail/scraping.md b/docs/clients/promtail/scraping.md index 44bcb0d069c6..b9f3f26d1012 100644 --- a/docs/clients/promtail/scraping.md +++ b/docs/clients/promtail/scraping.md @@ -77,8 +77,8 @@ See [Relabeling](#relabeling) for more information. ## Journal Scraping (Linux Only) -On systems with `systemd`, Promtail also supports reading from the journal. Unlike -file scraping which is defined in the `static_configs` stanza, journal scraping is +On systems with `systemd`, Promtail also supports reading from the journal. Unlike +file scraping which is defined in the `static_configs` stanza, journal scraping is defined in a `journal` stanza: ```yaml @@ -94,16 +94,32 @@ scrape_configs: target_label: 'unit' ``` -All fields defined in the `journal` section are optional, and are just provided -here for reference. The `max_age` field ensures that no older entry than the +All fields defined in the `journal` section are optional, and are just provided +here for reference. The `max_age` field ensures that no older entry than the time specified will be sent to Loki; this circumvents "entry too old" errors. -The `path` field tells Promtail where to read journal entries from. The labels +The `path` field tells Promtail where to read journal entries from. The labels map defines a constant list of labels to add to every journal entry that Promtail -reads. +reads. + +By default, Promtail reads from the journal by looking in the `/var/log/journal` +and `/run/log/journal` paths. If running Promtail inside of a Docker container, +the path appropriate to your distribution should be bind mounted inside of +Promtail along with binding `/etc/machine-id`. Bind mounting `/etc/machine-id` +to the path of the same name is required for the journal reader to know which +specific journal to read from. For example: + +```bash +docker run \ + -v /var/log/journal/:/var/log/journal/ \ + -v /run/log/journal/:/run/log/journal/ \ + -v /etc/machine-id:/etc/machine-id \ + grafana/promtail:latest \ + -config.file=/path/to/config/file.yaml +``` -When Promtail reads from the journal, it brings in all fields prefixed with -`__journal_` as internal labels. Like in the example above, the `_SYSTEMD_UNIT` -field from the journal was transformed into a label called `unit` through +When Promtail reads from the journal, it brings in all fields prefixed with +`__journal_` as internal labels. Like in the example above, the `_SYSTEMD_UNIT` +field from the journal was transformed into a label called `unit` through `relabel_configs`. See [Relabeling](#relabeling) for more information. ## Relabeling