Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: App specific transformations #4683

Draft
wants to merge 11 commits into
base: master
Choose a base branch
from
Draft

WIP: App specific transformations #4683

wants to merge 11 commits into from

Conversation

bazsi
Copy link
Collaborator

@bazsi bazsi commented Oct 24, 2023

This PR implements app-transform() parser that turns an app's messages conform to a destination's requirements.

Example config:

@version: current
@include "scl.conf"

transformation sudo[splunk] {
	translate {
		rewrite {
			set("sudo" value(".splunk.sourcetype"));
		};
	};
};


log {
	source { tcp(port(2000)); };
	parser { app-parser(topic(syslog)); };
	parser { app-transform(target(splunk)); };
};

@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi bazsi force-pushed the app-specific-transformations branch from 7f9039d to 76e9e64 Compare October 26, 2023 09:00
@kira-syslogng
Copy link
Contributor

Build FAILURE

@bazsi bazsi changed the title App specific transformations WIP: App specific transformations Oct 26, 2023
@MrAnno MrAnno added this to the syslog-ng 4.6 milestone Nov 29, 2023
@MrAnno MrAnno removed this from the syslog-ng 4.6 milestone Jan 9, 2024
@MrAnno MrAnno marked this pull request as draft February 2, 2024 07:43
bazsi added 11 commits May 15, 2024 18:40
@bazsi
appmodel: avoid using global variables during parsing
639c84c 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
appmodel-context: get rid off support for '*' in topic names
d660d96 
The entire app-parser() mechanism is undocumented as of now and I've looked
at the users (our own scl and SC4S), neither is using it.

I think this is a bad idea and just complicates the code anyway,
so let's get rid of it.

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
appmodel: extract AppModelObject base class from Application
0f6ccb2 
In preparation for adding a new root-level object type, extract the
abstract base object from Application.

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
app-parser-generator: extract AppObjectGenerator base class
6d64262 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
appmodel: extract AppObjectGenerator base class to its own module
f14dd18 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
syslog-format: don't try to turn a timestamp to unixtime if one is no…
ca0c17e 
…t present

If the timestamp is not actually there in the message, we still tried
to call convert_and_normalize_wall_clock_time_to_unix_time_with_tz_hint(),
which in turn calls mktime(), which is a heavyweight operation.

Let's not do that.

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
timeutils: cache mktime() return values for an hour
b70df98 
Without this patch, cached_mktime() produces a cached value for
every second, and would call mktime() again as the second changes.

This can be problematic when timestamps are parsed from different
seconds in succession (e.g. when the syslog timestamp is 1 second away
from a timestamp being parsed using strptime or date-parser).

In these cases the cache was invalidated and mktime() was called again.

mktime() is _slow_ as it even calls tzset() and validates that
/etc/localtime is still pointing to the same timezone.

This patch changes the caching strategy:
 - instead of specific seconds we calculate the cached value for the
   top of the hour (e.g. minutes==seconds==0)
 - timezones never change within the same hour, so as long as we are
   trying to use the cached value, we will do so as long as the
   year/month/day/hour value matches (plus isdst)
 - with this, mktime() will be called once every hour.

If there's some fluctuation between subsequent timestamps at the turn of the
hour, we can still exhibit limited caching (e.g. subsequent timestamps
from the last hour and this one), but this is a lot better than the
existing behaviour where we do that for every second.

Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
appmodel: add support for filterx expressions
462fa33 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
appmodel: add Transformation objects
8ede165 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
appmodel: add support for parsing Transformations
7605180 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi
appmodel: add app-transform() parser
1b76a4b 
Signed-off-by: Balazs Scheidler <balazs.scheidler@axoflow.com>
@bazsi bazsi force-pushed the app-specific-transformations branch from 76e9e64 to 1b76a4b Compare May 16, 2024 06:52
@kira-syslogng
Copy link
Contributor

Build FAILURE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
abandoned
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@bazsi @kira-syslogng @MrAnno @HofiOne