-
Notifications
You must be signed in to change notification settings - Fork 481
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add $(sum), $(min) and $(max) template functions #1037
Conversation
} | ||
|
||
void | ||
assert_template_format_with_context_msgs(const gchar *template, const gchar *expected, LogMessage **msgs, gint num_messages) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A bit long line there
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
This is great and very useful stuff which I'd liked merged. My notes are no biggies, but would be great to see at least some of these to be addressed. |
I'm going to fix these problems tomorrow. Thank you for the review. |
f0521e7
to
1715890
Compare
I modified multiple parts based on @bazsi's and @bkil-syslogng's review notes. Can I squash these fixups? |
The followups look great, please squash them. There's one unaddressed review note, which is not a showstopper, so can go in without addressing it, e.g. it's up to you. The function that gets the first value from the list has the same loop as the aggregation function. Presumably this is because the aggregators are expecting two values. This can be eliminated by using an explicit initialization value for the accumulator. For maximum aggregation it can by thr int64 minimum value, for sum it is zero and so on. This is how it is solved in functional languages. |
Example usage: parser p_groupingby { grouping_by( key("${KEY}") timeout(5) aggregate( value("SUM" "$(sum ${NUMBER})") inherit-mode("none") ) inject-mode("pass-through") ); }; Signed-off-by: László Várady <laszlo.varady@balabit.com>
Example usage: parser p_groupingby { grouping_by( key("${KEY}") timeout(5) aggregate( value("MAX" "$(max ${NUMBER})") inherit-mode("none") ) inject-mode("pass-through") ); }; Signed-off-by: László Várady <laszlo.varady@balabit.com>
Signed-off-by: László Várady <laszlo.varady@balabit.com>
Signed-off-by: László Várady <laszlo.varady@balabit.com>
Signed-off-by: bkil-syslogng <tamas.nagy@balabit.com> Signed-off-by: László Várady <laszlo.varady@balabit.com>
Originally, I created the What should I do when all values are "NaN" in the context? I think it would be nasty to return int64-min or int64-max in that case. |
@bazsi I've offered the same alternative in person, and we've discussed it with @MrAnno that it has drawbacks. For example, you either don't support INT max/min as a value and treat it as completely extremal, or you must keep track of whether we have already found a valid value or not. If we don't encounter a value, we return an empty string (Nothing), otherwise we return the stringified number (Maybe Int). The latter would take extra compute cycles compared to the optimized solution of @MrAnno. The former isn't that elegant. And what would you do for sum? An empty sum could be defined as On the other hand, the iteration might be abstracted again on a functional basis if it hurts that much - I'd love to see more Haskell code in C anyway. Note that we've implemented it just like |
Okay, as per the above, I'll give my 👍. I have further improvements in the pipe, but let's not block this feature with those for now. I'll submit them separately. Here's a work in progress about what I had in mind: So if I'm reading @bazsi correctly, he also gave a plus one: @ihrwein @dnsjts want to merge it? |
👍 from me, I'll push the merge button |
@fekete-robert : Could you document this new feature, please? Thanks! |
This merge adds $(average) template function & further stylistic improvements to #1037
This PR contains three new template functions, which can be used together with correlation plugins such as correlation-parser, grouping-by() parser and PatternDB.
These correlation plugins create message contexts that contain multiple messages.
The newly introduced
$(sum)
,$(min)
and$(max)
functions receive one argument, and operate on message contexts. The argument of the template function will be used to calculate the summation, the minimum and the maximum value of the context.Example usage with the
grouping-by()
parser:This parser groups messages based on their
${KEY}
value.After the group is complete (timeout is reached), the aggregation will be triggered and
${MAX}
will contain the maximum value of the${NUMBER}
fields in the message group.