Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filter_grep: Improve the performance #1940

Merged
merged 2 commits into from
Apr 13, 2018
Merged

Conversation

repeatedly
Copy link
Member

@repeatedly repeatedly commented Apr 11, 2018

  • Remove catch
  • Reduce method call
Warming up --------------------------------------
                 old     3.371k i/100ms
                 new     3.522k i/100ms
Calculating -------------------------------------
                 old     33.531k (± 4.1%) i/s -    168.550k in   5.035547s
                 new     35.888k (± 3.4%) i/s -    179.622k in   5.011044s

Signed-off-by: Masahiro Nakagawa repeatedly@gmail.com

- Remove catch
- Reduce method call

Warming up --------------------------------------
                 old     3.371k i/100ms
                 new     3.522k i/100ms
Calculating -------------------------------------
                 old     33.531k (± 4.1%) i/s -    168.550k in   5.035547s
                 new     35.888k (± 3.4%) i/s -    179.622k in   5.011044s

Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
@repeatedly
Copy link
Member Author

@okkez How about this change?

@repeatedly repeatedly self-assigned this Apr 11, 2018
@repeatedly repeatedly added v1 enhancement Feature request or improve operations labels Apr 11, 2018
@okkez
Copy link
Contributor

okkez commented Apr 12, 2018

Looks nice.
And more, how about applying following changes?
But my additional changes do not improve performance.
My benchmark script reports the unstable result.

diff --git a/lib/fluent/plugin/filter_grep.rb b/lib/fluent/plugin/filter_grep.rb
index 5c62f012..f590550b 100644
--- a/lib/fluent/plugin/filter_grep.rb
+++ b/lib/fluent/plugin/filter_grep.rb
@@ -161,10 +161,8 @@ module Fluent::Plugin
 
     def filter(tag, time, record)
       begin
-        if @_regexp_and_conditions
-          @_regexp_and_conditions.each do |expression|
-            return nil unless expression.match?(record)
-          end
+        if @_regexp_and_conditions && @_regexp_and_conditions.any? { |expression| !expression.match?(record) }
+          return nil
         end
         if @_regexp_or_conditions && @_regexp_or_conditions.none? { |expression| expression.match?(record) }
           return nil
@@ -172,10 +170,8 @@ module Fluent::Plugin
         if @_exclude_and_conditions && @_exclude_and_conditions.all? { |expression| expression.match?(record) }
           return nil
         end
-        if @_exclude_or_conditions
-          @_exclude_or_conditions.each do |expression|
-            return nil if expression.match?(record)
-          end
+        if @_exclude_or_conditions && @_exclude_or_conditions.any? { |expression| expression.match?(record) }
+          return nil
         end
       rescue => e
         log.warn "failed to grep events", error: e

@repeatedly
Copy link
Member Author

okay. Will apply it.

Signed-off-by: Masahiro Nakagawa <repeatedly@gmail.com>
@repeatedly repeatedly merged commit 6184928 into master Apr 13, 2018
@repeatedly repeatedly deleted the grep-optimize-performance branch April 13, 2018 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Feature request or improve operations v1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants