NullPointerException (NPE) happening with version 7.3.0

[andsel]

Logstash information:

Please include the following information:

1. Logstash version (e.g. bin/logstash --version)
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
4. How was the Logstash Plugin installed

JVM (e.g. java -version):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

1. JVM version (java -version)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. modify a sample test DB hosted in this project, updating a record with a custom information is_in_european_union

• use the customized example from https://github.com/andsel/mmdb-from-go-blogpost
• edit the path to the DB file https://github.com/andsel/mmdb-from-go-blogpost/blob/fix/example_for_NPE/main.go#L15 to load and to write https://github.com/andsel/mmdb-from-go-blogpost/blob/118d7f6f5b0c9be7a627051b0151dcb01ecc8720/main.go#L42 with your own path.
• run

go build
./mmdb-from-go-blogpost

2. run the following pipeline, setting the path to the DB file

input {
 generator {
   message => '{"ip": "216.160.83.58"}'
   codec => json
   count => 1
 }
}


filter {
 geoip {
   ecs_compatibility => disabled
   database => "/path/to/logstash_plugins/logstash-filter-geoip/src/test/resources/maxmind-test-data/GeoIP2-Country-Test.mmdb"
   source => "ip"
   target => "[client][ip]"
 }
}

output {
 stdout{}
}

3. With Logstash 8.15.0 which bundles logstash-filter-geoip 7.3.0 throws the following exception:

 com.maxmind.db.Decoder.decodeMapIntoObject(com/maxmind/db/Decoder.java:446
com.maxmind.db.Decoder.decodeMap(com/maxmind/db/Decoder.java:342
com.maxmind.db.Decoder.decodeByType(com/maxmind/db/Decoder.java:162
com.maxmind.db.Decoder.decode(com/maxmind/db/Decoder.java:151
com.maxmind.db.Decoder.decodeMapIntoObject(com/maxmind/db/Decoder.java:429
com.maxmind.db.Decoder.decodeMap(com/maxmind/db/Decoder.java:342
com.maxmind.db.Decoder.decodeByType(com/maxmind/db/Decoder.java:162
com.maxmind.db.Decoder.decode(com/maxmind/db/Decoder.java:151
com.maxmind.db.Decoder.decode(com/maxmind/db/Decoder.java:76
com.maxmind.db.Reader.resolveDataPointer(com/maxmind/db/Reader.java:275
com.maxmind.db.Reader.getRecord(com/maxmind/db/Reader.java:185
com.maxmind.geoip2.DatabaseReader.get(com/maxmind/geoip2/DatabaseReader.java:263
com.maxmind.geoip2.DatabaseReader.getCountry(com/maxmind/geoip2/DatabaseReader.java:309
com.maxmind.geoip2.DatabaseReader.country(com/maxmind/geoip2/DatabaseReader.java:292
org.logstash.filters.geoip.GeoIPFilter.retrieveCountryGeoData(org/logstash/filters/geoip/GeoIPFilter.java:340
org.logstash.filters.geoip.GeoIPFilter.handleEvent(org/logstash/filters/geoip/GeoIPFilter.java:168
jdk.internal.reflect.DirectMethodHandleAccessor.invoke(jdk/internal/reflect/DirectMethodHandleAccessor.java:103
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:580
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:315
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:176
RUBY.filter(/Users/andrea/workspace/elastic_products/logstash-8.15.0/vendor/bundle/jruby/3.1.0/gems/logstash-filter-geoip-7.3.0-java/lib/logstash/filters/geoip.rb:117
RUBY.do_filter(/Users/andrea/workspace/elastic_products/logstash-8.15.0/logstash-core/lib/logstash/filters/base.rb:158
RUBY.multi_filter(/Users/andrea/workspace/elastic_products/logstash-8.15.0/logstash-core/lib/logstash/filters/base.rb:176
org.jruby.RubyArray.each(org/jruby/RubyArray.java:1981
org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen
RUBY.multi_filter(/Users/andrea/workspace/elastic_products/logstash-8.15.0/logstash-core/lib/logstash/filters/base.rb:173
org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(org/logstash/config/ir/compiler/FilterDelegatorExt.java:128
org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.lambda$multiFilter$0(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133
org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(org/logstash/instrument/metrics/timer/ConcurrentLiveTimerMetric.java:47
org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133
org.logstash.generated.CompiledDataset1.compute(org/logstash/generated/CompiledDataset1
org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:347
org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:341
org.logstash.execution.ObservedExecution.lambda$compute$0(org/logstash/execution/ObservedExecution.java:17
org.logstash.execution.WorkerObserver.lambda$observeExecutionComputation$0(org/logstash/execution/WorkerObserver.java:39
org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(org/logstash/instrument/metrics/timer/ConcurrentLiveTimerMetric.java:47
org.logstash.execution.WorkerObserver.lambda$executeWithTimers$1(org/logstash/execution/WorkerObserver.java:50
org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(org/logstash/instrument/metrics/timer/ConcurrentLiveTimerMetric.java:47
org.logstash.execution.WorkerObserver.executeWithTimers(org/logstash/execution/WorkerObserver.java:50
org.logstash.execution.WorkerObserver.observeExecutionComputation(org/logstash/execution/WorkerObserver.java:38
org.logstash.execution.ObservedExecution.compute(org/logstash/execution/ObservedExecution.java:17
org.logstash.execution.WorkerLoop.abortableCompute(org/logstash/execution/WorkerLoop.java:113
org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:86
jdk.internal.reflect.DirectMethodHandleAccessor.invoke(jdk/internal/reflect/DirectMethodHandleAccessor.java:103
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:580
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:300
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:164
RUBY.start_workers(/Users/andrea/workspace/elastic_products/logstash-8.15.0/logstash-core/lib/logstash/java_pipeline.rb:304
org.jruby.RubyProc.call(org/jruby/RubyProc.java:354
java.lang.Thread.run(java/lang/Thread.java:1583)

Handle the exception tagging the event on failure. Could be verified also on #225 test PR

Provide logs (if relevant):

[2024-09-16T12:34:40,066][ERROR][logstash.javapipeline    ][main] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"main", :error=>"Cannot invoke \"Object.getClass()\" because \"parameters[index]\" is null", :exception=>Java::JavaLang::NullPointerException, :backtrace=>["com.maxmind.db.Decoder.decodeMapIntoObject(com/maxmind/db/Decoder.java:446)", "com.maxmind.db.Decoder.decodeMap(com/maxmind/db/Decoder.java:342)", "com.maxmind.db.Decoder.decodeByType(com/maxmind/db/Decoder.java:162)", "com.maxmind.db.Decoder.decode(com/maxmind/db/Decoder.java:151)", "com.maxmind.db.Decoder.decodeMapIntoObject(com/maxmind/db/Decoder.java:429)", "com.maxmind.db.Decoder.decodeMap(com/maxmind/db/Decoder.java:342)", "com.maxmind.db.Decoder.decodeByType(com/maxmind/db/Decoder.java:162)", "com.maxmind.db.Decoder.decode(com/maxmind/db/Decoder.java:151)", "com.maxmind.db.Decoder.decode(com/maxmind/db/Decoder.java:76)", "com.maxmind.db.Reader.resolveDataPointer(com/maxmind/db/Reader.java:275)", "com.maxmind.db.Reader.getRecord(com/maxmind/db/Reader.java:185)", "com.maxmind.geoip2.DatabaseReader.get(com/maxmind/geoip2/DatabaseReader.java:263)", "com.maxmind.geoip2.DatabaseReader.getCountry(com/maxmind/geoip2/DatabaseReader.java:309)", "com.maxmind.geoip2.DatabaseReader.country(com/maxmind/geoip2/DatabaseReader.java:292)", "org.logstash.filters.geoip.GeoIPFilter.retrieveCountryGeoData(org/logstash/filters/geoip/GeoIPFilter.java:340)", "org.logstash.filters.geoip.GeoIPFilter.handleEvent(org/logstash/filters/geoip/GeoIPFilter.java:168)", "jdk.internal.reflect.DirectMethodHandleAccessor.invoke(jdk/internal/reflect/DirectMethodHandleAccessor.java:103)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:580)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:315)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:176)", "RUBY.filter(/Users/andrea/workspace/elastic_products/logstash-8.15.0/vendor/bundle/jruby/3.1.0/gems/logstash-filter-geoip-7.3.0-java/lib/logstash/filters/geoip.rb:117)", "RUBY.do_filter(/Users/andrea/workspace/elastic_products/logstash-8.15.0/logstash-core/lib/logstash/filters/base.rb:158)", "RUBY.multi_filter(/Users/andrea/workspace/elastic_products/logstash-8.15.0/logstash-core/lib/logstash/filters/base.rb:176)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1981)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "RUBY.multi_filter(/Users/andrea/workspace/elastic_products/logstash-8.15.0/logstash-core/lib/logstash/filters/base.rb:173)", "org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(org/logstash/config/ir/compiler/FilterDelegatorExt.java:128)", "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.lambda$multiFilter$0(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133)", "org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(org/logstash/instrument/metrics/timer/ConcurrentLiveTimerMetric.java:47)", "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133)", "org.logstash.generated.CompiledDataset1.compute(org/logstash/generated/CompiledDataset1)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:347)", "org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:341)", "org.logstash.execution.ObservedExecution.lambda$compute$0(org/logstash/execution/ObservedExecution.java:17)", "org.logstash.execution.WorkerObserver.lambda$observeExecutionComputation$0(org/logstash/execution/WorkerObserver.java:39)", "org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(org/logstash/instrument/metrics/timer/ConcurrentLiveTimerMetric.java:47)", "org.logstash.execution.WorkerObserver.lambda$executeWithTimers$1(org/logstash/execution/WorkerObserver.java:50)", "org.logstash.instrument.metrics.timer.ConcurrentLiveTimerMetric.time(org/logstash/instrument/metrics/timer/ConcurrentLiveTimerMetric.java:47)", "org.logstash.execution.WorkerObserver.executeWithTimers(org/logstash/execution/WorkerObserver.java:50)", "org.logstash.execution.WorkerObserver.observeExecutionComputation(org/logstash/execution/WorkerObserver.java:38)", "org.logstash.execution.ObservedExecution.compute(org/logstash/execution/ObservedExecution.java:17)", "org.logstash.execution.WorkerLoop.abortableCompute(org/logstash/execution/WorkerLoop.java:113)", "org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:86)", "jdk.internal.reflect.DirectMethodHandleAccessor.invoke(jdk/internal/reflect/DirectMethodHandleAccessor.java:103)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:580)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:300)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:164)", "RUBY.start_workers(/Users/andrea/workspace/elastic_products/logstash-8.15.0/logstash-core/lib/logstash/java_pipeline.rb:304)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:354)", "java.lang.Thread.run(java/lang/Thread.java:1583)"], :thread=>"#<Thread:0x20f11ee9 /Users/andrea/workspace/elastic_products/logstash-8.15.0/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}

---

This issue originates from maxmind/MaxMind-DB-Reader-java#164 once that PR is published in a new version of the library, this plugin will be updated and manage the DeserializationException and tagging the event.

[oschwald]

maxmind-db shouldn't be throwing a NullPointerException. That seems to be a bug there. I would have expected a DeserializationException with an error about is_in_european_union being a uint16 when a boolean was expected.

[oschwald]

Oh, I think this is maxmind/MaxMind-DB-Reader-java#164. It appears the fix for that has not been released, but it is merged into main. I'll look at getting it released.

[oschwald]

A new version of maxmind-db and geoip2 were released.