logstash stuck in read mode on big number of files

[SLepUbIn]

• Version: 6.4.2
• Operating System:ubuntu 18.04

I'm trying to do a one time processing of a big number of files organized as such : folder/subfolder1/subfolder2/files
There are 298 427 files, 138 "subfolder1" levels, each "subfolder1" level containing between 10 and 20 "subfolder2" levels and each "subfolder2" level contains between 20 and 240 files.

I use the file input plugin with the new read mode,there is a filter and the output is stdout for test (for production the target is elasticSearch).

Here is my config :

input {
file {
sincedb_path => "/dev/null"
path => "/home/me/WORK/logstash/20180917///*.json"
close_older => 5
codec => "json"
mode => "read"
file_completed_action => "log"
file_completed_log_path => "/dev/null"
}
}

I tested my configuration on a fewer number of files and everything goes as intended.
When I launch logstash on the big folder, I see the initialization, and then nothing happens.

The following sequence keeps repeating in the logs :

[2018-10-30T15:57:16,332][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x1e0466b2 sle$
[2018-10-30T15:57:16,437][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-10-30T15:57:16,437][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}

And nothing happens.
I waited an hour with no luck.

So I decided to investigate a bit.
I did kill -3 logstashPID to get a thread dump.

In this thread dump, with the exception of threads busy with java.nio, the only RUNNABLE thread is busy in ruby code, all others are WAITING.

here are two samples below, from two different thread dumps taken at 5 minutes interval (lot of output, explanations go on below) :

Oct 30 15:56:32   logstash[10719]:    java.lang.Thread.State: RUNNABLE
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1209358542.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.BlockBody.yield(BlockBody.java:122)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.Block.yieldArray(Block.java:177)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyArray$6.compare(RubyArray.java:3519)
Oct 30 15:56:32   logstash[10719]: #011at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
Oct 30 15:56:32   logstash[10719]: #011at java.util.TimSort.sort(TimSort.java:234)
Oct 30 15:56:32   logstash[10719]: #011at java.util.Arrays.sort(Arrays.java:1512)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyArray.sortInternal(RubyArray.java:3513)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyArray.sort_bang(RubyArray.java:3458)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyArray$INVOKER$i$0$0$sort_bang.call(RubyArray$INVOKER$i$0$0$sort_bang.gen)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.SynchronizedDynamicMethod.call(SynchronizedDynamicMethod.java:83)
Oct 30 15:56:32   logstash[10719]: #011- locked <0x00000000dde264a0> (a org.jruby.RubyArray)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1692036285.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/769429195.delegate(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1659969211.guard(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/769429195.delegate(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1659969211.guard(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watched_files_collection.RUBY$method$last_modified_asc$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watched_files_collection.rb:51)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/775236830.invokeStatic_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1176254217.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:90)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyMethod.call(RubyMethod.java:111)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/2123492724.invokeVirtual_L3_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/237630710.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/534906248.delegate(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1806640599.guard(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/534906248.delegate(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1806640599.guard(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/966567431.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watched_files_collection.RUBY$method$add$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watched_files_collection.rb:15)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/669132924.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/769429195.delegate(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1659969211.guard(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/769429195.delegate(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1659969211.guard(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$block$discover_any_files$2(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb:99)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/1394463314.invokeStatic_L8_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1209358542.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.Block.yield(Block.java:165)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyArray.each(RubyArray.java:1734)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/524536503.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$discover_any_files$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb:61)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$discover_any_files$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/123721308.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/222927891.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$discover_files_new_path$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb:51)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$discover_files_new_path$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$add_path$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb:24)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$add_path$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watch.RUBY$method$watch$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watch.rb:27)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watch.RUBY$method$watch$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watch.rb)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.observing_base.RUBY$method$watch_this$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:70)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.observing_base.RUBY$method$watch_this$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$block$start_processing$1(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:333)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/634540230.invokeStatic_L8_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1209358542.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.Block.yield(Block.java:165)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyArray.each(RubyArray.java:1734)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/524536503.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$method$start_processing$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:333)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$method$start_processing$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1131673199.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/966567431.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:337)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:409)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:403)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/634540230.invokeStatic_L8_L(LambdaForm$DMH)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1209358542.invokeExact_MT(LambdaForm$MH)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.runtime.Block.call(Block.java:124)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyProc.call(RubyProc.java:289)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.RubyProc.call(RubyProc.java:246)
Oct 30 15:56:32   logstash[10719]: #011at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
Oct 30 15:56:32   logstash[10719]: #011at java.lang.Thread.run(Thread.java:748)

Other one :

Oct 30 15:50:48   logstash[10719]:    java.lang.Thread.State: RUNNABLE
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyString.eql19(RubyString.java:343)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyString.eql(RubyString.java:337)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyHash.internalKeyExist(RubyHash.java:565)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyHash.internalGetEntry(RubyHash.java:556)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyHash.op_asetForString(RubyHash.java:989)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyHash.fastASetCheckString(RubyHash.java:946)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyHash.op_aset(RubyHash.java:983)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyHash$INVOKER$i$2$0$op_aset.call(RubyHash$INVOKER$i$2$0$op_aset.gen)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.SynchronizedDynamicMethod.call(SynchronizedDynamicMethod.java:62)
Oct 30 15:50:48   logstash[10719]: #011- locked <0x00000000dda3a818> (a org.jruby.RubyHash)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/1801756410.invokeVirtual_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1032382137.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/867148091.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1194587893.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/867148091.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1194587893.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/222927891.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watched_files_collection.RUBY$block$refresh_pointers$1(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watched_files_collection.rb:80)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/802571466.invokeStatic_L8_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1209358542.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:55)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.Block.call(Block.java:146)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyEnumerable$EachWithIndex.call(RubyEnumerable.java:1003)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.CallBlock.doYield(CallBlock.java:91)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.BlockBody.yield(BlockBody.java:116)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.Block.yield(Block.java:165)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyArray.each(RubyArray.java:1734)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.SynchronizedDynamicMethod.call(SynchronizedDynamicMethod.java:111)
Oct 30 15:50:48   logstash[10719]: #011- locked <0x00000000dde264a0> (a org.jruby.RubyArray)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyClass.finvoke(RubyClass.java:522)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.Helpers.invoke(Helpers.java:395)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyEnumerable.callEach(RubyEnumerable.java:140)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyEnumerable.each_with_indexCommon(RubyEnumerable.java:1037)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyEnumerable.each_with_index(RubyEnumerable.java:1067)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyEnumerable$INVOKER$s$0$0$each_with_index.call(RubyEnumerable$INVOKER$s$0$0$each_with_index.gen)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:196)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.SynchronizedDynamicMethod.call(SynchronizedDynamicMethod.java:83)
Oct 30 15:50:48   logstash[10719]: #011- locked <0x00000000dde264a0> (a org.jruby.RubyArray)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1692036285.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/769429195.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1659969211.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/769429195.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1659969211.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watched_files_collection.RUBY$method$refresh_pointers$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watched_files_collection.rb:79)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/2051853139.invokeStatic_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/561783758.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/534906248.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1806640599.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/534906248.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1806640599.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/966567431.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watched_files_collection.RUBY$method$last_modified_asc$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watched_files_collection.rb:54)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/775236830.invokeStatic_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1176254217.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:90)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:128)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyMethod.call(RubyMethod.java:111)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/2123492724.invokeVirtual_L3_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/237630710.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/534906248.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1806640599.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/534906248.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1806640599.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/966567431.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watched_files_collection.RUBY$method$add$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watched_files_collection.rb:15)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/669132924.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/769429195.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1659969211.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/769429195.delegate(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1659969211.guard(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$block$discover_any_files$2(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb:99)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/1394463314.invokeStatic_L8_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1209358542.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.Block.yield(Block.java:165)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyArray.each(RubyArray.java:1734)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/524536503.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$discover_any_files$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb:61)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$discover_any_files$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/123721308.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/222927891.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$discover_files_new_path$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb:51)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$discover_files_new_path$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$add_path$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb:24)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.discoverer.RUBY$method$add_path$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/discoverer.rb)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watch.RUBY$method$watch$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watch.rb:27)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.watch.RUBY$method$watch$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watch.rb)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.observing_base.RUBY$method$watch_this$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb:70)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.filewatch.observing_base.RUBY$method$watch_this$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/observing_base.rb)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$block$start_processing$1(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:333)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/634540230.invokeStatic_L8_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1209358542.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.Block.yield(Block.java:165)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyArray.each(RubyArray.java:1734)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/524536503.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$method$start_processing$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:333)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$method$start_processing$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1131673199.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/966567431.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb:337)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_1_dot_6.lib.logstash.inputs.file.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/logstash/inputs/file.rb)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:409)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/416153648.invokeStatic_L7_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/722764585.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/153048669.invokeVirtual_L6_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$BMH/1097684722.reinvoke(LambdaForm$BMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1848760713.linkToCallSite(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:403)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$DMH/634540230.invokeStatic_L8_L(LambdaForm$DMH)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.invoke.LambdaForm$MH/1209358542.invokeExact_MT(LambdaForm$MH)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.runtime.Block.call(Block.java:124)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyProc.call(RubyProc.java:289)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.RubyProc.call(RubyProc.java:246)
Oct 30 15:50:48   logstash[10719]: #011at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
Oct 30 15:50:48   logstash[10719]: #011at java.lang.Thread.run(Thread.java:748)

the code seems stuck in watched_files_collection.rb.
Looking at the ruby code in /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watched_files_collection.rb it seems that the sort method is called each time a file is discovered and added to the list of files to process.
So, in my case, actually the growing array of files to process is sorted more than 290 000 times.
The read mode is quite new for this plugin, so maybe the sorting that was appropriate in tail mode could now lead to bad performance (in this case no performance at all since the processing doesn't even start).

As a test (a very very very dirty test) I commented out the line @sort_method.call in

def add(watched_file)
      @files << watched_file
      @sort_method.call
    end

in file /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-file-4.1.6/lib/filewatch/watched_files_collection.rb

And launched logstash on my big folder. After a few seconds I saw processed files being output to stdout...
Which makes me think that sorting is one cause of my problem.

So maybe the file_sort_by parameter of the plugin should provide a third possibility which is, no sorting at all ?

[guyboertje]

In your case (an extreme one) sorting is the problem. I spiked a test of a discovered files "feeder" system that, when more than 4095 files are discovered, feeds a smaller number of files into the files-to-process collection. I would keep a keen eye on the sorting performance. This is on the unofficial road map for this plugin

[Tsukiand]

@guyboertje I am using logstash-input-file 4.1.7. We have many files(60000 files) and I use tail mode, I found that our logs have delay issue. It seems that logstash-input-file did not do discover and stat operations on the input files. I suspected that my delay issue also cause by the "sorting" operation.

[guyboertje]

I suspected that my delay issue also cause by the "sorting" operation

You suspect correctly.

[gmonk]

Version: 6.8.0
Operating System:ubuntu 16.04

Any updates on this ? Having the same issues. If fold has too many files logstash will hang. The only way to get this working is with the work around listed above and comment out @sort_method.call

[travismgibson]

Any update on this issue? What is the impact of commenting out the @sort_method.call ?

[kares]

sorting watched files performance has been addressed in 4.2.0 #219