LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124 #2830

vparfonov · 2024-10-15T18:47:19Z

Description

This PR addressed to align Syslog Output with syslog specification RFC3164 and RFC5424

Default values for fields:

RFC3164

Format: <PRI>TIMESTAMP HOSTNAME TAG: MESSAGE
Example:<34>Oct 11 22:14:15 mymachine su[1234]: 'su root' failed for lonvick on /dev/pts/

	journal	application	infrastructure	note
AppName	SYSLOG_IDENTIFIER*	namespacePodContainer	.log_source
ProcId	_PID*	N/A	.auditID (if available)
MsgId	N/A	N/A	N/A	will have no effect in settings
Payloadkey	.message	.message	.message
Severity	N/A	.level	information (6)
Facility	N/A	user (1)	security(13)

*If ProcId available, it will aggregate with AppName in the .tag field: appname[procid]

RFC5424

Format: <PRI> VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [STRUCTURED-DATA] MESSAGE
Example: <PRI>1 2024-11-08T14:35:04.123Z ip-10-0-88-196.ec2.internal app-name 1234 ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] This is a sample message

	journal	application	infrastructure
AppName	SYSLOG_IDENTIFIER	namespace_pod_container	.log_source
ProcId	_PID	pod_id	.auditID (if available)
MsgId	.log_source	.log_source	.log_sourcel
Payloadkey	.message	.message	.message
Severity	N/A	.level	information (6)
Facility	N/A	user (1)	security(13)

Changes:

add calculation for default value if field not configured according to the table above
fix configuration for deployed syslog server in functional test to follow RFC3164 specification

/cc @cahartma @Clee2691
/assign @jcantrill

/cherry-pick

Links

Depending on PR(s):
Bugzilla:
Github issue:
JIRA: https://issues.redhat.com/browse/LOG-6236
Enhancement proposal:

openshift-ci · 2024-10-15T18:47:23Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

jcantrill · 2024-11-04T19:16:18Z

internal/generator/vector/output/syslog/rfc3164_with_app_name_setting.toml

+source = '''
+. = merge(., parse_json!(string!(.message))) ?? .
+
+.tag = to_string!(.systemd.u.SYSLOG_IDENTIFIER[.systemd.t.PID] || .systemd.u.SYSLOG_IDENTIFIER || "-")


I think we want the fallback to be empty spaces for all of these. Fluentd used "-" but I do not see where dash is anything in the spec

vparfonov · 2024-11-07T23:28:18Z

/test all

vparfonov · 2024-11-07T23:47:07Z

/test all

openshift-ci-robot · 2024-11-08T15:43:26Z

@vparfonov: This pull request references LOG-6236 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.8.0" version, but no target version was set.

In response to this:

Description

This PR addressed to align CLO Syslog Output with syslog specification RFC3164 and RFC5424

RFC3164

Format: <PRI> TIMESTAMP HOSTNAME TAG: MESSAGE
<34>.@timestamp .hostname .tag: .message
Example:
<34>Oct 11 22:14:15 mymachine su[1234]: 'su root' failed for lonvick on /dev/pts/

journal application infrastructure

AppName SYSLOG_IDENTIFIER* namespace_pod_container .log_source

ProcId _PID* N/A .auditID or "-" l

MsgId N/A .log_source .log_sourcel

Payloadkey .message .message .message

Severity N/A .level information (6)

Facility N/A user (1) security(13)

/cc
/assign

/cherry-pick

Links

Depending on PR(s):

Bugzilla:

Github issue:

JIRA:

Enhancement proposal:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

cahartma · 2024-11-08T16:44:06Z

/hold

jcantrill · 2024-11-08T16:49:27Z

internal/generator/vector/output/syslog/syslog.go

 		})
 	}

-	if s.ProcId != "" {
+	if s.ProcId == "" {
+		s.ProcId = `{._internal.syslog.proc_id || "-"}`


My previous comment regarding "default" applies and the reason I really want to hash this out in the document of what the various scenarios look like. IMO the fallback should not be -; it should be empty

yes for RFC3164 should be empty, but for RFC5164 -. I will update

kattz-kawa · 2024-11-14T14:17:07Z

internal/generator/vector/output/syslog/rfc3164_with_defaults.toml

+inputs = ["application"]
+source = '''
+. = merge(., parse_json!(string!(.message))) ?? .
+if .log_type == "infrastructure" && .log_source == "node" {


@vparfonov There might be logs with .log_type == "infrastructure" that do not have .log_source == "node". As far as I can see, I found that the logs from namespace_name=openshift-kube-apiserver and container_name=kube-apiserver do not have .log_source == "node". This line may need a conditional statement for container logs within the openshift-* namespace.

@kattz-kawa Good point, it can be .log_source == "container", but it will be not journal log something closer to application logs.

If the only input to this transform truly is "application" logs, "application" is ever only container logs; it is never sourced from journald. This means we should alter the transforms accordingly

I think this logic is correct, could simplify slightly :

if (node) - these are journald logs and we can map the syslog fields exactly from the logs.
Note: 'if (node)' is equivalent to 'if (infra && node)' because all node logs are infra logs.
The difference in format is really node vs. container, not application vs. infra.

if (container) - container logs are formatted the same regardless of type (application or infra)

if (audit) - audit logs formatted differently from either of the previous two cases.

…C5124 Signed-off-by: Vitalii Parfonov <vparfono@redhat.com>

vparfonov · 2024-11-18T11:29:06Z

/retest

alanconway

Looks good to me.

alanconway · 2024-11-19T01:05:16Z

internal/generator/vector/output/syslog/rfc3164_with_defaults.toml

+inputs = ["application"]
+source = '''
+. = merge(., parse_json!(string!(.message))) ?? .
+if .log_type == "infrastructure" && .log_source == "node" {


I think this logic is correct, could simplify slightly :

if (node) - these are journald logs and we can map the syslog fields exactly from the logs.
Note: 'if (node)' is equivalent to 'if (infra && node)' because all node logs are infra logs.
The difference in format is really node vs. container, not application vs. infra.

if (container) - container logs are formatted the same regardless of type (application or infra)

if (audit) - audit logs formatted differently from either of the previous two cases.

openshift-ci · 2024-11-19T01:17:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alanconway, vparfonov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alanconway]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

alanconway · 2024-11-19T01:17:17Z

/lgtm

cahartma · 2024-12-11T14:36:11Z

/unhold

vparfonov · 2024-12-11T15:23:25Z

/retest

openshift-ci · 2024-12-11T17:18:07Z

@vparfonov: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-cherrypick-robot · 2024-12-11T17:22:58Z

@vparfonov: cannot checkout <!--: error checking out "<!--": exit status 1 error: pathspec '<!--' did not match any file(s) known to git

In response to this:

Description

This PR addressed to align Syslog Output with syslog specification RFC3164 and RFC5424

Default values for fields:

RFC3164

Format: <PRI>TIMESTAMP HOSTNAME TAG: MESSAGE
Example:<34>Oct 11 22:14:15 mymachine su[1234]: 'su root' failed for lonvick on /dev/pts/

journal application infrastructure note

AppName SYSLOG_IDENTIFIER* namespacePodContainer .log_source

ProcId _PID* N/A .auditID (if available)

MsgId N/A N/A N/A will have no effect in settings

Payloadkey .message .message .message

Severity N/A .level information (6)

Facility N/A user (1) security(13)

*If ProcId available, it will aggregate with AppName in the .tag field: appname[procid]

RFC5424

Format: <PRI> VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [STRUCTURED-DATA] MESSAGE
Example: <PRI>1 2024-11-08T14:35:04.123Z ip-10-0-88-196.ec2.internal app-name 1234 ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] This is a sample message

journal application infrastructure

AppName SYSLOG_IDENTIFIER namespace_pod_container .log_source

ProcId _PID pod_id .auditID (if available)

MsgId .log_source .log_source .log_sourcel

Payloadkey .message .message .message

Severity N/A .level information (6)

Facility N/A user (1) security(13)

Changes:

add calculation for default value if field not configured according to the table above

fix configuration for deployed syslog server in functional test to follow RFC3164 specification

/cc @cahartma @Clee2691
/assign @jcantrill

/cherry-pick

Links

Depending on PR(s):

Bugzilla:

Github issue:

JIRA: https://issues.redhat.com/browse/LOG-6236

Enhancement proposal:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

vparfonov · 2024-12-11T17:25:12Z

/cherry-pick release-6.1

openshift-cherrypick-robot · 2024-12-11T17:25:49Z

@vparfonov: new pull request created: #2905

In response to this:

/cherry-pick release-6.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

vparfonov · 2024-12-13T12:54:35Z

/cherry-pick release-6.0

openshift-cherrypick-robot · 2024-12-13T12:55:20Z

@vparfonov: #2830 failed to apply on top of branch "release-6.0":

Applying: LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124
.git/rebase-apply/patch:37: space before tab in indent.
   	._internal.syslog.tag = replace(._internal.syslog.tag, r'[^a-zA-Z0-9]', "")
.git/rebase-apply/patch:92: space before tab in indent.
   	._internal.syslog.tag = join!([.kubernetes.namespace_name, .kubernetes.pod_name, .kubernetes.container_name], "")
.git/rebase-apply/patch:93: space before tab in indent.
   	._internal.syslog.severity = .level
.git/rebase-apply/patch:94: space before tab in indent.
   	._internal.syslog.facility = "user"
.git/rebase-apply/patch:95: space before tab in indent.
   	#Remove non-alphanumeric characters
warning: squelched 3 whitespace errors
warning: 8 lines add whitespace errors.
Using index info to reconstruct a base tree...
M	internal/generator/vector/output/syslog/syslog.go
M	internal/generator/vector/output/syslog/syslog_test.go
M	test/functional/outputs/syslog/forward_to_syslog_test.go
Falling back to patching base and 3-way merge...
Auto-merging test/functional/outputs/syslog/forward_to_syslog_test.go
CONFLICT (content): Merge conflict in test/functional/outputs/syslog/forward_to_syslog_test.go
Auto-merging internal/generator/vector/output/syslog/syslog_test.go
CONFLICT (content): Merge conflict in internal/generator/vector/output/syslog/syslog_test.go
Auto-merging internal/generator/vector/output/syslog/syslog.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124

In response to this:

/cherry-pick release-6.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 15, 2024

vparfonov changed the title ~~Add functional test infrastructure -> syslog~~ Align syslog output implementation with spec Nov 4, 2024

jcantrill reviewed Nov 4, 2024

View reviewed changes

vparfonov force-pushed the log6236 branch from 9e0fb5f to 5aa9925 Compare November 7, 2024 23:28

vparfonov force-pushed the log6236 branch from 1b9a61a to a9bf370 Compare November 7, 2024 23:36

vparfonov force-pushed the log6236 branch from 97a6543 to b87ed0d Compare November 8, 2024 15:42

vparfonov changed the title ~~Align syslog output implementation with spec~~ LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124 Nov 8, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 8, 2024

vparfonov marked this pull request as ready for review November 8, 2024 15:45

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 8, 2024

openshift-ci bot requested review from alanconway and jcantrill November 8, 2024 15:45

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 8, 2024

jcantrill reviewed Nov 8, 2024

View reviewed changes

kattz-kawa reviewed Nov 14, 2024

View reviewed changes

vparfonov force-pushed the log6236 branch from b87ed0d to 1d7a310 Compare November 15, 2024 17:30

LOG-6236: Align syslog output implementation with spec RFC3164 and RF…

2b0ac9b

…C5124 Signed-off-by: Vitalii Parfonov <vparfono@redhat.com>

vparfonov force-pushed the log6236 branch from 1d7a310 to 2b0ac9b Compare November 18, 2024 09:42

alanconway approved these changes Nov 19, 2024

View reviewed changes

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 19, 2024

openshift-ci bot assigned alanconway Nov 19, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 19, 2024

vparfonov requested review from jcantrill and alanconway November 20, 2024 13:26

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 11, 2024

openshift-merge-bot bot merged commit a3d72d9 into openshift:master Dec 11, 2024
8 checks passed

vparfonov deleted the log6236 branch December 11, 2024 17:22

openshift-cherrypick-robot mentioned this pull request Dec 11, 2024

[release-6.1] LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124 #2905

Merged

vparfonov mentioned this pull request Dec 12, 2024

LOG-6236: add docs about how to forwarding log to Syslog #2907

Merged

jcantrill added the release/6.2 label Dec 12, 2024

vparfonov mentioned this pull request Dec 13, 2024

[release-6.0]LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124 #2910

Merged

vparfonov mentioned this pull request Feb 4, 2025

LOG-5706: Align syslog output implementation with spec RFC3164 and RFC5124 for Vector #2953

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124 #2830

LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124 #2830

vparfonov commented Oct 15, 2024 •

edited

Loading

openshift-ci bot commented Oct 15, 2024

jcantrill Nov 4, 2024

vparfonov commented Nov 7, 2024

vparfonov commented Nov 7, 2024

openshift-ci-robot commented Nov 8, 2024

Description

Links

cahartma commented Nov 8, 2024

jcantrill Nov 8, 2024

vparfonov Nov 8, 2024

kattz-kawa Nov 14, 2024

vparfonov Nov 14, 2024 •

edited

Loading

jcantrill Nov 14, 2024

alanconway Nov 19, 2024

vparfonov commented Nov 18, 2024

alanconway left a comment

alanconway Nov 19, 2024

openshift-ci bot commented Nov 19, 2024

alanconway commented Nov 19, 2024

cahartma commented Dec 11, 2024

vparfonov commented Dec 11, 2024

openshift-ci bot commented Dec 11, 2024

openshift-cherrypick-robot commented Dec 11, 2024

Description

Default values for fields:

RFC3164

RFC5424

Changes:

Links

vparfonov commented Dec 11, 2024

openshift-cherrypick-robot commented Dec 11, 2024

vparfonov commented Dec 13, 2024

openshift-cherrypick-robot commented Dec 13, 2024

LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124 #2830

LOG-6236: Align syslog output implementation with spec RFC3164 and RFC5124 #2830

Conversation

vparfonov commented Oct 15, 2024 • edited Loading

Description

Default values for fields:

Changes:

Links

openshift-ci bot commented Oct 15, 2024

Choose a reason for hiding this comment

vparfonov commented Nov 7, 2024

vparfonov commented Nov 7, 2024

openshift-ci-robot commented Nov 8, 2024

Description

Links

cahartma commented Nov 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vparfonov Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vparfonov commented Nov 18, 2024

alanconway left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci bot commented Nov 19, 2024

alanconway commented Nov 19, 2024

cahartma commented Dec 11, 2024

vparfonov commented Dec 11, 2024

openshift-ci bot commented Dec 11, 2024

openshift-cherrypick-robot commented Dec 11, 2024

Description

Default values for fields:

Changes:

Links

vparfonov commented Dec 11, 2024

openshift-cherrypick-robot commented Dec 11, 2024

vparfonov commented Dec 13, 2024

openshift-cherrypick-robot commented Dec 13, 2024

vparfonov commented Oct 15, 2024 •

edited

Loading

vparfonov Nov 14, 2024 •

edited

Loading