Skip to content

Commit

Permalink
in_sample: support to renew record functionality
Browse files Browse the repository at this point in the history
When in_sample plugin is used with filter parser which uses
remove_key_name_field, it raises the following error repeatedly.

  #0 dump an error event: error_class=ArgumentError error="message
  does not exist"

This kind of error occurs when key_name and remove_key_name_field
removes key from record with destructive change in filter parser.
It affects generated sample data.

To fix this issue, it is simple to just dup every record even though
it has a significant performance penalty.

Considering keeping compatibility and providing way to a workaround,
added option to control whether it always re-generate a new record or
re-use existing record - renew_record, enabled by default.

ref. #4575

Here is the small benchmark:

<source>
    @type sample
    tag test
    size xxx
    rate 100
    #renew_record false
    renew_record true
</source>
<match test>
  @type null
</match>

size: 10000

current master
 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 201025 kenhys    20   0  340940  54800  13648 S  24.6   0.1   0:04.62 ruby

renew_record: false
 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 202342 kenhys    20   0  340948  54732  13576 S  25.7   0.1   0:06.83 ruby

renew_record: true
 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 203402 kenhys    20   0  343288  63556  13500 S  30.7   0.1   0:06.86 ruby

size: 100000

current master
 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 204681 kenhys    20   0  364324  80880  13448 S 100.0   0.1   0:14.46 ruby

renew_record: false
 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 205560 kenhys    20   0  364332  80868  13448 S  99.7   0.1   0:31.34 ruby

renew_record: true
 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 207133 kenhys    20   0  365804 113320  13576 S  99.7   0.2 0:13.87 ruby

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
  • Loading branch information
kenhys committed Aug 16, 2024
1 parent a6978e5 commit 433160c
Showing 1 changed file with 13 additions and 3 deletions.
16 changes: 13 additions & 3 deletions lib/fluent/plugin/in_sample.rb
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ class SampleInput < Input
config_param :auto_increment_key, :string, default: nil
desc "The boolean to suspend-and-resume incremental value after restart"
config_param :suspend, :bool, default: false,deprecated: 'This parameters is ignored'
desc "It force to re-generate a new record by default. You can disable it if filter does not do destructive change"
config_param :renew_record, :bool, default: true
desc "The sample data to be generated. An array of JSON hashes or a single JSON hash."
config_param :sample, alias: :dummy, default: [{"message" => "sample"}] do |val|
begin
Expand Down Expand Up @@ -118,14 +120,22 @@ def emit(num)
end

def generate
d = @sample[@sample_index]
if @renew_record
d = @sample[@sample_index].dup
else
d = @sample[@sample_index]
end
unless d
@sample_index = 0
d = @sample[@sample_index]
if @renew_record
d = @sample[@sample_index].dup
else
d = @sample[@sample_index]
end
end
@sample_index += 1
if @auto_increment_key
d = d.dup
d = d.dup unless @renew_record
d[@auto_increment_key] = @storage.update(:auto_increment_value){|v| v + 1 }
end
d
Expand Down

0 comments on commit 433160c

Please sign in to comment.