-
Notifications
You must be signed in to change notification settings - Fork 422
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[journald] Create custom journald input package (#1739)
Create a generic package for ingesting journald logs.
- Loading branch information
1 parent
ea959b4
commit d0dcc58
Showing
16 changed files
with
487 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
dependencies: | ||
ecs: | ||
reference: git@1.11 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
# Journald Input | ||
|
||
The journald input integration reads logs from the `journald` system service. | ||
The journald input reads the log data and the metadata associated with it. | ||
|
||
The journald input is available on Linux systems with `systemd` installed. | ||
|
||
{{fields "log"}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
version: '2.3' | ||
services: | ||
journald: | ||
image: alpine | ||
volumes: | ||
- ./sample_logs:/sample_logs:ro | ||
- ${SERVICE_LOGS_DIR}:/var/log | ||
command: /bin/sh -c "cp /sample_logs/* /var/log/" |
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# newer versions go on top | ||
- version: "0.0.1" | ||
changes: | ||
- description: Initial release of generic journald input package. | ||
type: enhancement | ||
link: https://github.com/elastic/integrations/pull/1739 |
7 changes: 7 additions & 0 deletions
7
packages/journald/data_stream/log/_dev/test/system/test-journald-config.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
service: journald | ||
input: journald | ||
data_stream: | ||
vars: | ||
paths: | ||
- "{{SERVICE_LOGS_DIR}}/test.journal" | ||
tags: [forwarded] |
29 changes: 29 additions & 0 deletions
29
packages/journald/data_stream/log/agent/stream/journald.yml.hbs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
condition: ${host.platform} == 'linux' | ||
|
||
{{#if paths}} | ||
paths: | ||
{{#each paths as |path i|}} | ||
- {{path}} | ||
{{/each}} | ||
{{/if}} | ||
|
||
{{#if include_matches}} | ||
include_matches: | ||
{{#each include_matches as |include_match i|}} | ||
- '{{include_match}}' | ||
{{/each}} | ||
{{/if}} | ||
|
||
tags: | ||
{{#each tags as |tag i|}} | ||
- {{tag}} | ||
{{/each}} | ||
|
||
{{#contains "forwarded" tags}} | ||
publisher_pipeline.disable_host: true | ||
{{/contains}} | ||
|
||
{{#if processors}} | ||
processors: | ||
{{processors}} | ||
{{/if}} |
43 changes: 43 additions & 0 deletions
43
packages/journald/data_stream/log/elasticsearch/ingest_pipeline/default.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
--- | ||
description: Pipeline for processing journald logs. | ||
processors: | ||
- remove: | ||
field: | ||
- container.id_truncated # CONTAINER_ID from journald driver. | ||
- syslog.facility # Duplicated as log.syslog.facility.code. | ||
- syslog.priority # Duplicated as log.syslog.priority. | ||
ignore_missing: true | ||
|
||
# Journald CODE_FILE, CODE_LINE, CODE_FUNC. | ||
- rename: | ||
field: journald.code.file | ||
target_field: log.origin.file.name | ||
ignore_missing: true | ||
- rename: | ||
field: journald.code.line | ||
target_field: log.origin.file.line | ||
ignore_missing: true | ||
- rename: | ||
field: journald.code.func | ||
target_field: log.origin.function | ||
ignore_missing: true | ||
|
||
# Journald SYSLOG_IDENTIFIER, SYSLOG_PID. | ||
- rename: | ||
field: syslog.pid | ||
target_field: log.syslog.pid | ||
ignore_missing: true | ||
- rename: | ||
field: syslog.identifier | ||
target_field: log.syslog.identifier | ||
ignore_missing: true | ||
|
||
# Cleanup an empty syslog object. | ||
- remove: | ||
if: ctx?.syslog != null && ctx.syslog instanceof Map && ctx.syslog.isEmpty() | ||
field: syslog | ||
|
||
on_failure: | ||
- set: | ||
field: error.message | ||
value: '{{ _ingest.on_failure_message }}' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
- name: input.type | ||
type: keyword |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
- name: data_stream.type | ||
type: constant_keyword | ||
description: Data stream type. | ||
- name: data_stream.dataset | ||
type: constant_keyword | ||
description: Data stream dataset. | ||
- name: data_stream.namespace | ||
type: constant_keyword | ||
description: Data stream namespace. | ||
- name: '@timestamp' | ||
type: date | ||
description: Event timestamp. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
- name: ecs.version | ||
external: ecs | ||
- name: host.hostname | ||
external: ecs | ||
- name: host.id | ||
external: ecs | ||
- name: log.syslog.facility.code | ||
external: ecs | ||
- name: log.syslog.priority | ||
external: ecs | ||
- name: message | ||
external: ecs | ||
- name: process.args | ||
external: ecs | ||
- name: process.args_count | ||
external: ecs | ||
- name: process.command_line | ||
external: ecs | ||
- name: process.pid | ||
external: ecs | ||
- name: tags | ||
external: ecs | ||
- name: user.group.id | ||
external: ecs | ||
- name: user.id | ||
external: ecs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,194 @@ | ||
- name: container.log.tag | ||
type: keyword | ||
description: > | ||
User defined tag of a container. Originates from the Docker journald logging driver. | ||
- name: container.partial | ||
type: boolean | ||
description: > | ||
A field that flags log integrity when a message is split. The docker journald logging driver splits long message into multiple events. | ||
- name: journald.audit.login_uid | ||
type: long | ||
description: > | ||
The login UID of the process the journal entry originates from, as maintained by the kernel audit subsystem. | ||
- name: journald.audit.session | ||
type: keyword | ||
description: > | ||
The session of the process the journal entry originates from, as maintained by the kernel audit subsystem. | ||
- name: journald.code.file | ||
type: keyword | ||
description: > | ||
The code location generating this message, if known. Contains the source filename. | ||
- name: journald.code.func | ||
type: keyword | ||
description: > | ||
The code location generating this message, if known. Contains the function name. | ||
- name: journald.code.line | ||
type: long | ||
description: > | ||
The code location generating this message, if known. Contains the line number. | ||
- name: journald.coredump.unit | ||
type: keyword | ||
description: > | ||
Used to annotate messages containing coredumps from system units. | ||
- name: journald.coredump.user_unit | ||
type: keyword | ||
description: > | ||
Used to annotate messages containing coredumps from user units. | ||
- name: journald.gid | ||
type: long | ||
description: > | ||
The group ID of the process the journal entry originates from formatted as a decimal string. Note that entries obtained via "stdout" or "stderr" of forked processes will contain credentials valid for a parent process. | ||
- name: journald.host.boot_id | ||
type: keyword | ||
description: > | ||
The kernel boot ID for the boot the message was generated in, formatted as a 128-bit hexadecimal string. | ||
- name: journald.kernel.device | ||
type: keyword | ||
description: > | ||
The kernel device name. If the entry is associated to a block device, contains the major and minor numbers of the device node, separated by ":" and prefixed by "b". Similarly for character devices, but prefixed by "c". For network devices, this is the interface index prefixed by "n". For all other devices, this is the subsystem name prefixed by "+", followed by ":", followed by the kernel device name. | ||
- name: journald.kernel.device_name | ||
type: keyword | ||
description: > | ||
The kernel device name as it shows up in the device tree below `/sys/`. | ||
- name: journald.kernel.device_node_path | ||
type: keyword | ||
description: > | ||
The device node path of this device in `/dev/`. | ||
- name: journald.kernel.device_symlinks | ||
type: keyword | ||
description: > | ||
Additional symlink names pointing to the device node in `/dev/`. This field is frequently set more than once per entry. | ||
- name: journald.kernel.subsystem | ||
type: keyword | ||
description: > | ||
The kernel subsystem name. | ||
- name: journald.object.pid | ||
type: long | ||
description: > | ||
Privileged programs (currently UID 0) may attach OBJECT_PID= to a message. This will instruct systemd-journald to attach additional `journald.object.*` on behalf of the caller. These additional fields added automatically by systemd-journald. These additional `journald.object.*` fields are the same as the equivalent `journald.*` field except that the process identified by PID is described, instead of the process which logged the message. | ||
- name: journald.object.audit.login_uid | ||
type: long | ||
- name: journald.object.audit.session | ||
type: long | ||
- name: journald.object.gid | ||
type: long | ||
- name: journald.object.process.command_line | ||
type: keyword | ||
- name: journald.object.process.executable | ||
type: keyword | ||
- name: journald.object.process.name | ||
type: keyword | ||
- name: journald.object.systemd.owner_uid | ||
type: long | ||
- name: journald.object.systemd.session | ||
type: keyword | ||
- name: journald.object.systemd.unit | ||
type: keyword | ||
- name: journald.object.systemd.user_unit | ||
type: keyword | ||
- name: journald.object.uid | ||
type: long | ||
- name: journald.pid | ||
type: long | ||
description: > | ||
The process ID of the process the journal entry originates from formatted as a decimal string. Note that entries obtained via "stdout" or "stderr" of forked processes will contain credentials valid for a parent process. | ||
- name: journald.process.capabilities | ||
type: keyword | ||
description: The effective capabilities(7) of the process the journal entry originates from. | ||
- name: journald.process.command_line | ||
type: keyword | ||
description: > | ||
The command line of the process the journal entry originates from. | ||
- name: journald.process.executable | ||
type: keyword | ||
description: > | ||
The executable path of the process the journal entry originates from. | ||
- name: journald.process.name | ||
type: keyword | ||
description: > | ||
The name of the process the journal entry originates from. | ||
- name: journald.uid | ||
type: long | ||
description: > | ||
The user ID of the process the journal entry originates from formatted as a decimal string. Note that entries obtained via "stdout" or "stderr" of forked processes will contain credentials valid for a parent process. | ||
- name: log.syslog.identifier | ||
type: keyword | ||
description: > | ||
Identifier (usually process) contained in the syslog header. | ||
- name: log.syslog.pid | ||
type: long | ||
description: > | ||
PID contained in the syslog header. | ||
- name: systemd.cgroup | ||
type: keyword | ||
description: > | ||
The control group path in the systemd hierarchy. | ||
- name: systemd.invocation_id | ||
type: keyword | ||
description: > | ||
The invocation ID for the runtime cycle of the unit the message was generated in, as available to processes of the unit in $INVOCATION_ID. | ||
- name: systemd.owner_uid | ||
type: long | ||
description: > | ||
The owner UID of the systemd user unit or systemd session (if any) of the process the journal entry originates from. | ||
- name: systemd.session | ||
type: keyword | ||
description: > | ||
The systemd session ID (if any). | ||
- name: systemd.slice | ||
type: keyword | ||
description: > | ||
The systemd slice unit name. | ||
- name: systemd.transport | ||
type: keyword | ||
description: > | ||
How the entry was received by the journal service. | ||
- name: systemd.unit | ||
type: keyword | ||
description: > | ||
The systemd unit name. | ||
- name: systemd.user_slice | ||
type: keyword | ||
description: > | ||
The systemd user slice name. | ||
- name: systemd.user_unit | ||
type: keyword | ||
description: > | ||
The unit name in the systemd user manager (if any). | ||
- name: journald.custom | ||
type: flattened | ||
description: >- | ||
Structured fields added to the log message by the caller. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
title: "Journald Log" | ||
type: logs | ||
streams: | ||
- input: journald | ||
title: Journald logs | ||
template_path: journald.yml.hbs | ||
description: Collect journald logs | ||
vars: | ||
- name: include_matches | ||
type: text | ||
title: Include Matches | ||
multi: true | ||
show_user: true | ||
description: > | ||
A list of filter expressions used to select the logs to read (e.g. `_SYSTEMD_UNIT=vault.service`). Defaults to all logs. See [include_matches](https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-input-journald.html#filebeat-input-journald-include-matches) for details. | ||
- name: paths | ||
type: text | ||
title: Journal paths | ||
multi: true | ||
show_user: false | ||
description: > | ||
List of journals to read from. Defaults to the system journal. | ||
- name: tags | ||
type: text | ||
title: Tags | ||
multi: true | ||
required: true | ||
show_user: false | ||
default: | ||
- journald-log | ||
- name: processors | ||
type: yaml | ||
title: Processors | ||
multi: false | ||
required: false | ||
show_user: false | ||
description: > | ||
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. | ||
Oops, something went wrong.