org.apache.lucene.util.MergedIterator.pullTop() Crash

[WUMUXIAN]

Elasticsearch version (bin/elasticsearch --version):
6.3.0

Plugins installed: [ingest-geoip, ingest-user-agent]

JVM version (java -version):
openjdk version "10.0.1" 2018-04-17
OpenJDK Runtime Environment (build 10.0.1+10)
OpenJDK 64-Bit Server VM (build 10.0.1+10, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux es-data-us-east-1c-0 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I am running a elasticsearch data node using this docker image:

docker.elastic.co/elasticsearch/elasticsearch:6.3.0

Somehow it always crashes after running for ~5 mins with error like this:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fc7ebc050eb, pid=1, tid=151
#
# JRE version: OpenJDK Runtime Environment (10.0.1+10) (build 10.0.1+10)
# Java VM: OpenJDK 64-Bit Server VM (10.0.1+10, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# J 14088 c2 org.apache.lucene.util.MergedIterator.pullTop()V (135 bytes) @ 0x00007fc7ebc050eb [0x00007fc7ebc041e0+0x0000000000000f0b]
#
# Core dump will be written. Default location: /usr/share/elasticsearch/core
#
# An error report file with more information is saved as:
# logs/hs_err_pid1.log
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]


[error occurred during error reporting (), id 0xb]

[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]
[Too many errors, abort]

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. Deploy 3 master nodes
2. Deploy 2 data nodes
3. One of the data nodes always crashes with the above error.

[vladimirdolzhenko]

@elastic/es-core-infra could you pls have a look ?

[ywelsch]

@WUMUXIAN can you share more information on the machine type that you're using, in particular the output of cat /proc/cpuinfo?

[WUMUXIAN]

@yevhen Here you go:

cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping	: 4
microcode	: 0x2000043
cpu MHz		: 2500.000
cache size	: 33792 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
bugs		: cpu_meltdown spectre_v1 spectre_v2
bogomips	: 5000.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

A few more information:

1. I ran two docker containers with the same images on the same OS (m5.large AWS EC2 instances, Debian 1.9). But on one of the machine, it crashes always, on another one it does not.
2. I updated the -Xms and -Xmx to 2g for the one that always crashes, it still crashed once, but somehow it stopped crashing and is up and running since 12 hours ago.

I am not sure why this is the behaviour, very strange to me, hope it can help you.

Let me know if you need more information.

[jpountz]

Thank you @WUMUXIAN. Can you also share the content of logs/hs_err_pid1.log?

[WUMUXIAN]

@jpountz Sure, see it below:

cat elasticsearch/logs/hs_err_pid1.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8c43de5371, pid=1, tid=140
#
# JRE version: OpenJDK Runtime Environment (10.0.1+10) (build 10.0.1+10)
# Java VM: OpenJDK 64-Bit Server VM (10.0.1+10, mixed mode, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# J 13039 c2 org.apache.lucene.util.MergedIterator.pushTop()V (92 bytes) @ 0x00007f8c43de5371 [0x00007f8c43de4c40+0x0000000000000731]
#
# Core dump will be written. Default location: /usr/share/elasticsearch/core
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.9Ts2lgAF -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Des.cgroups.hierarchy.override=/ -Xms2g -Xmx2g -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=tar org.elasticsearch.bootstrap.Elasticsearch -Ecluster.name=tds-es-db -Expack.watcher.enabled=false -Expack.ml.enabled=false -Expack.logstash.enabled=false -Ediscovery.zen.ping.unicast.hosts=elasticsearch-discovery -Ehttp.enabled=false -Epath.data=/data -Enode.data=true -Enode.master=false -Enode.name=es-data-us-east-1c-0 -Enetwork.host=_eth0:ipv4_ -Expack.graph.enabled=false -Ediscovery.zen.minimum_master_nodes=2 -Expack.security.enabled=false -Enode.ingest=false

Host: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz, 2 cores, 7G, CentOS Linux release 7.5.1804 (Core)
Time: Tue Jun 19 10:15:27 2018 UTC elapsed time: 247 seconds (0d 0h 4m 7s)

---------------  T H R E A D  ---------------

Current thread (0x00007f8c18125000):  JavaThread "elasticsearch[es-data-us-east-1c-0][[logstash-2018.06.19][2]: Lucene Merge Thread #8]" daemon [_thread_in_Java, id=140, stack(0x00007f8bf12c1000,0x00007f8bf13c2000)]

Stack: [0x00007f8bf12c1000,0x00007f8bf13c2000],  sp=0x00007f8bf13c01e8,  free space=1020k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 13039 c2 org.apache.lucene.util.MergedIterator.pushTop()V (92 bytes) @ 0x00007f8c43de5371 [0x00007f8c43de4c40+0x0000000000000731]

[error occurred during error reporting (printing native stack), id 0xb]


siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000007fffffc3

Register to memory mapping:

RAX=0xffffffffffffffe0 is an unknown value
RBX=0x000000008015d6f8 is an oop
org.apache.lucene.util.MergedIterator
{0x000000008015d6f8} - klass: 'org/apache/lucene/util/MergedIterator'
RCX=0x0000000000000005 is an unknown value
RDX=0x0000000000000020 is an unknown value
RSP=0x00007f8bf13c01e8 is pointing into the stack for thread: 0x00007f8c18125000
RBP=0x0000000000000023 is an unknown value
RSI=0x000000007fffffe3 is an unknown value
RDI=0x000000008f2bee53 is pointing into object: 0x000000008f2bee40
org.apache.lucene.analysis.tokenattributes.BytesTermAttributeImpl
{0x000000008f2bee40} - klass: 'org/apache/lucene/analysis/tokenattributes/BytesTermAttributeImpl'
R8 =0x0000000093598e60 is an oop
[B
{0x0000000093598e60} - klass: {type array byte}
 - length: 35
R9 =0x0000000000000000 is an unknown value
R10=0x00000000842d9ff0 is an oop
[B
{0x00000000842d9ff0} - klass: {type array byte}
 - length: 42
R11=0x00000000842d9ff0 is an oop
[B
{0x00000000842d9ff0} - klass: {type array byte}
 - length: 42
R12=0x0000000000000000 is an unknown value
R13=0x000000000000000b is an unknown value
R14=0x000000008015d7c0 is an oop
org.apache.lucene.util.MergedIterator$SubIterator
{0x000000008015d7c0} - klass: 'org/apache/lucene/util/MergedIterator$SubIterator'
R15=0x00007f8c18125000 is a thread


Registers:
RAX=0xffffffffffffffe0, RBX=0x000000008015d6f8, RCX=0x0000000000000005, RDX=0x0000000000000020
RSP=0x00007f8bf13c01e8, RBP=0x0000000000000023, RSI=0x000000007fffffe3, RDI=0x000000008f2bee53
R8 =0x0000000093598e60, R9 =0x0000000000000000, R10=0x00000000842d9ff0, R11=0x00000000842d9ff0
R12=0x0000000000000000, R13=0x000000000000000b, R14=0x000000008015d7c0, R15=0x00007f8c18125000
RIP=0x00007f8c43de5371, EFLAGS=0x0000000000010283, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f8bf13c01e8)
0x00007f8bf13c01e8:   00000000fffffff9 8015d7a800000000
0x00007f8bf13c01f8:   0000000000000002 000000008015d6f8
0x00007f8bf13c0208:   000000008015d6f8 00007f8bf13c0250
0x00007f8bf13c0218:   00007f8bf13c0240 00007f8c18125000

Instructions: (pc=0x00007f8c43de5371)
0x00007f8c43de5351:   98 ff 0f 83 14 01 00 00 48 83 c0 40 83 ea 40 75
0x00007f8c43de5361:   de 62 a1 7d a1 ef c0 eb 64 62 e1 7e a9 6f 04 07
0x00007f8c43de5371:   62 e1 7d a1 ef 04 06 48 83 ec 40 62 f1 7e 49 7f
0x00007f8c43de5381:   04 24 48 83 ec 40 62 f1 7e 49 7f 0c 24 62 b1 7e



---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x00007f8c20004670, length=38, elements={
0x00007f8c540c7000, 0x00007f8c540c9000, 0x00007f8c540e0800, 0x00007f8c540e2800,
0x00007f8c540e4800, 0x00007f8c540e6800, 0x00007f8c54157800, 0x00007f8c54169800,
0x00007f8c54729800, 0x00007f8c55090000, 0x00007f8c55687800, 0x00007f8c55cf6000,
0x00007f8c55d17800, 0x00007f8c55d1f800, 0x00007f8c08008000, 0x00007f8c10013800,
0x00007f8c14928000, 0x00007f8c1c005800, 0x00007f8c143eb800, 0x00007f8c20003800,
0x00007f8c0800c800, 0x00007f8bfc014800, 0x00007f8c55d32800, 0x00007f8bfc02f800,
0x00007f8c04002000, 0x00007f8c04003800, 0x00007f8c04005800, 0x00007f8c04083000,
0x00007f8c54a81000, 0x00007f8c54012000, 0x00007f8c0000e800, 0x00007f8c55974800,
0x00007f8bfc00e000, 0x00007f8c549bf000, 0x00007f8c554ae800, 0x00007f8c549c0000,
0x00007f8c24244800, 0x00007f8c18125000
}

Java Threads: ( => current thread )
  0x00007f8c540c7000 JavaThread "Reference Handler" daemon [_thread_blocked, id=52, stack(0x00007f8c58dab000,0x00007f8c58eac000)]
  0x00007f8c540c9000 JavaThread "Finalizer" daemon [_thread_blocked, id=53, stack(0x00007f8c58caa000,0x00007f8c58dab000)]
  0x00007f8c540e0800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=54, stack(0x00007f8c58ba9000,0x00007f8c58caa000)]
  0x00007f8c540e2800 JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=55, stack(0x00007f8c58aa8000,0x00007f8c58ba9000)]
  0x00007f8c540e4800 JavaThread "C1 CompilerThread1" daemon [_thread_blocked, id=56, stack(0x00007f8c589a7000,0x00007f8c58aa8000)]
  0x00007f8c540e6800 JavaThread "Sweeper thread" daemon [_thread_blocked, id=57, stack(0x00007f8c588a6000,0x00007f8c589a7000)]
  0x00007f8c54157800 JavaThread "Service Thread" daemon [_thread_blocked, id=58, stack(0x00007f8c587a5000,0x00007f8c588a6000)]
  0x00007f8c54169800 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=60, stack(0x00007f8c585a2000,0x00007f8c586a3000)]
  0x00007f8c54729800 JavaThread "process reaper" daemon [_thread_in_native, id=62, stack(0x00007f8c58158000,0x00007f8c5817a000)]
  0x00007f8c55090000 JavaThread "elasticsearch[es-data-us-east-1c-0][[timer]]" daemon [_thread_blocked, id=66, stack(0x00007f8c58035000,0x00007f8c58136000)]
  0x00007f8c55687800 JavaThread "elasticsearch[es-data-us-east-1c-0][scheduler][T#1]" daemon [_thread_blocked, id=67, stack(0x00007f8c39319000,0x00007f8c3941a000)]
  0x00007f8c55cf6000 JavaThread "elasticsearch[es-data-us-east-1c-0][[transport_server_worker.default]][T#1]" daemon [_thread_in_native, id=68, stack(0x00007f8c300ff000,0x00007f8c30200000)]
  0x00007f8c55d17800 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#1]" daemon [_thread_blocked, id=69, stack(0x00007f8bfbeff000,0x00007f8bfc000000)]
  0x00007f8c55d1f800 JavaThread "elasticsearch[es-data-us-east-1c-0][clusterApplierService#updateTask][T#1]" daemon [_thread_blocked, id=70, stack(0x00007f8bfbdfe000,0x00007f8bfbeff000)]
  0x00007f8c08008000 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#2]" daemon [_thread_blocked, id=72, stack(0x00007f8bfb7dc000,0x00007f8bfb8dd000)]
  0x00007f8c10013800 JavaThread "elasticsearch[es-data-us-east-1c-0][transport_client_boss][T#1]" daemon [_thread_in_native, id=73, stack(0x00007f8bfb6db000,0x00007f8bfb7dc000)]
  0x00007f8c14928000 JavaThread "elasticsearch[es-data-us-east-1c-0][transport_client_boss][T#2]" daemon [_thread_in_native, id=76, stack(0x00007f8bfb3d8000,0x00007f8bfb4d9000)]
  0x00007f8c1c005800 JavaThread "threadDeathWatcher-2-1" daemon [_thread_blocked, id=77, stack(0x00007f8bfb2d7000,0x00007f8bfb3d8000)]
  0x00007f8c143eb800 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#3]" daemon [_thread_blocked, id=78, stack(0x00007f8bf91d4000,0x00007f8bf92d5000)]
  0x00007f8c20003800 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#4]" daemon [_thread_blocked, id=79, stack(0x00007f8bf90d3000,0x00007f8bf91d4000)]
  0x00007f8c0800c800 JavaThread "elasticsearch[es-data-us-east-1c-0][masterService#updateTask][T#1]" daemon [_thread_blocked, id=80, stack(0x00007f8bf8fd2000,0x00007f8bf90d3000)]
  0x00007f8bfc014800 JavaThread "elasticsearch[es-data-us-east-1c-0][[transport_server_worker.default]][T#2]" daemon [_thread_in_native, id=81, stack(0x00007f8bf8cd1000,0x00007f8bf8dd2000)]
  0x00007f8c55d32800 JavaThread "elasticsearch[es-data-us-east-1c-0][fetch_shard_store][T#1]" daemon [_thread_blocked, id=82, stack(0x00007f8bf8bd0000,0x00007f8bf8cd1000)]
  0x00007f8bfc02f800 JavaThread "elasticsearch[es-data-us-east-1c-0][fetch_shard_store][T#2]" daemon [_thread_blocked, id=83, stack(0x00007f8bf8acf000,0x00007f8bf8bd0000)]
  0x00007f8c04002000 JavaThread "elasticsearch[es-data-us-east-1c-0][management][T#1]" daemon [_thread_blocked, id=84, stack(0x00007f8bf89ce000,0x00007f8bf8acf000)]
  0x00007f8c04003800 JavaThread "elasticsearch[es-data-us-east-1c-0][management][T#2]" daemon [_thread_blocked, id=85, stack(0x00007f8bf88cd000,0x00007f8bf89ce000)]
  0x00007f8c04005800 JavaThread "elasticsearch[es-data-us-east-1c-0][management][T#3]" daemon [_thread_blocked, id=86, stack(0x00007f8bf87cc000,0x00007f8bf88cd000)]
  0x00007f8c04083000 JavaThread "trigger_engine_scheduler[T#1]" daemon [_thread_blocked, id=87, stack(0x00007f8bf84cb000,0x00007f8bf85cc000)]
  0x00007f8c54a81000 JavaThread "elasticsearch[keepAlive/6.3.0]" [_thread_blocked, id=88, stack(0x00007f8bf83ca000,0x00007f8bf84cb000)]
  0x00007f8c54012000 JavaThread "DestroyJavaVM" [_thread_blocked, id=48, stack(0x00007f8c5d669000,0x00007f8c5d76a000)]
  0x00007f8c0000e800 JavaThread "elasticsearch[es-data-us-east-1c-0][refresh][T#1]" daemon [_thread_blocked, id=89, stack(0x00007f8bf80c9000,0x00007f8bf81ca000)]
  0x00007f8c55974800 JavaThread "elasticsearch[es-data-us-east-1c-0][generic][T#5]" daemon [_thread_blocked, id=90, stack(0x00007f8bf7fc8000,0x00007f8bf80c9000)]
  0x00007f8bfc00e000 JavaThread "elasticsearch[es-data-us-east-1c-0][write][T#1]" daemon [_thread_blocked, id=91, stack(0x00007f8bf3b04000,0x00007f8bf3c05000)]
  0x00007f8c549bf000 JavaThread "elasticsearch[es-data-us-east-1c-0][search][T#1]" daemon [_thread_blocked, id=94, stack(0x00007f8bf366c000,0x00007f8bf376d000)]
  0x00007f8c554ae800 JavaThread "elasticsearch[es-data-us-east-1c-0][get][T#1]" daemon [_thread_blocked, id=95, stack(0x00007f8bf13c2000,0x00007f8bf14c3000)]
  0x00007f8c549c0000 JavaThread "elasticsearch[es-data-us-east-1c-0][search][T#2]" daemon [_thread_blocked, id=97, stack(0x00007f8bf356b000,0x00007f8bf366c000)]
  0x00007f8c24244800 JavaThread "elasticsearch[es-data-us-east-1c-0][flush][T#1]" daemon [_thread_blocked, id=101, stack(0x00007f8bfb4d9000,0x00007f8bfb5da000)]
=>0x00007f8c18125000 JavaThread "elasticsearch[es-data-us-east-1c-0][[logstash-2018.06.19][2]: Lucene Merge Thread #8]" daemon [_thread_in_Java, id=140, stack(0x00007f8bf12c1000,0x00007f8bf13c2000)]

Other Threads:
  0x00007f8c540bd000 VMThread "VM Thread" [stack: 0x00007f8c58eae000,0x00007f8c58fae000] [id=51]
  0x00007f8c5415a800 WatcherThread [stack: 0x00007f8c586a5000,0x00007f8c587a5000] [id=59]
  0x00007f8c54023800 GCTaskThread "GC Thread#0" [stack: 0x00007f8c5a35d000,0x00007f8c5a45d000] [id=49]
  0x00007f8c54046000 ConcurrentGCThread "CMS Main Thread" [stack: 0x00007f8c59881000,0x00007f8c59981000] [id=50]

Threads with active compile tasks:
C2 CompilerThread013047 %     4       org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexWriter::writeBlock @ 326 (496 bytes)

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap address: 0x0000000080000000, size: 2048 MB, Compressed Oops mode: 32-bit
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x0000000100000000

Heap:
 par new generation   total 76672K, used 7596K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   9% used [0x0000000080000000, 0x000000008060de20, 0x0000000084290000)
  from space 8512K,  16% used [0x0000000084290000, 0x00000000843ed2e8, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 177343K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73262K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K

Card table byte_map: [0x00007f8c59f5a000,0x00007f8c5a35b000] byte_map_base: 0x00007f8c59b5a000

Marking Bits: (CMSBitMap*) 0x00007f8c54044ee8
 Bits: [0x00007f8c39c73000, 0x00007f8c3bb26400)

Mod Union Table: (CMSBitMap*) 0x00007f8c54044f88
 Bits: [0x00007f8c5d5da000, 0x00007f8c5d654cd0)

Polling page: 0x00007f8c5d770000

CodeHeap 'non-profiled nmethods': size=120036Kb used=8163Kb max_used=8163Kb free=111872Kb
 bounds [0x00007f8c435ee000, 0x00007f8c43dee000, 0x00007f8c4ab27000]
CodeHeap 'profiled nmethods': size=120032Kb used=23258Kb max_used=23258Kb free=96773Kb
 bounds [0x00007f8c3c0b6000, 0x00007f8c3d786000, 0x00007f8c435ee000]
CodeHeap 'non-nmethods': size=5692Kb used=1352Kb max_used=1433Kb free=4339Kb
 bounds [0x00007f8c3bb27000, 0x00007f8c3bd97000, 0x00007f8c3c0b6000]
 total_blobs=16241 nmethods=11803 adapters=716
 compilation: enabled

Compilation events (10 events):
Event: 246.993 Thread 0x00007f8c540e4800 13045 %     3       org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter::merge @ 913 (1010 bytes)
Event: 247.019 Thread 0x00007f8c540e4800 nmethod 13045% 0x00007f8c3d76d390 code [0x00007f8c3d76dc80, 0x00007f8c3d772d58]
Event: 247.020 Thread 0x00007f8c540e2800 nmethod 13039 0x00007f8c43de4a10 code [0x00007f8c43de4c40, 0x00007f8c43de5c18]
Event: 247.020 Thread 0x00007f8c540e2800 13042       4       org.apache.lucene.util.bkd.BKDWriter$1::visit (10 bytes)
Event: 247.023 Thread 0x00007f8c540e2800 nmethod 13042 0x00007f8c43de6490 code [0x00007f8c43de6640, 0x00007f8c43de68d8]
Event: 247.023 Thread 0x00007f8c540e2800 13046       4       org.apache.lucene.index.IndexWriter::ensureOpen (33 bytes)
Event: 247.093 Thread 0x00007f8c540e2800 nmethod 13046 0x00007f8c43de6b90 code [0x00007f8c43de6d20, 0x00007f8c43de6db8]
Event: 247.101 Thread 0x00007f8c540e4800 13048       3       java.util.Spliterators::checkFromToBounds (74 bytes)
Event: 247.103 Thread 0x00007f8c540e4800 nmethod 13048 0x00007f8c3d774f10 code [0x00007f8c3d7751e0, 0x00007f8c3d775e48]
Event: 247.103 Thread 0x00007f8c540e2800 13047 %     4       org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexWriter::writeBlock @ 326 (496 bytes)

GC Heap History (10 events):
Event: 245.405 GC heap before
{Heap before GC invocations=219 (full 3):
 par new generation   total 76672K, used 75988K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,  99% used [0x0000000080000000, 0x000000008428b0f0, 0x0000000084290000)
  from space 8512K,  92% used [0x0000000084ae0000, 0x000000008528a288, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 171228K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73245K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 245.421 GC heap after
{Heap after GC invocations=220 (full 3):
 par new generation   total 76672K, used 5001K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  58% used [0x0000000084290000, 0x0000000084772578, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73245K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 245.593 GC heap before
{Heap before GC invocations=220 (full 3):
 par new generation   total 76672K, used 72835K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,  99% used [0x0000000080000000, 0x000000008423e860, 0x0000000084290000)
  from space 8512K,  58% used [0x0000000084290000, 0x0000000084772578, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73251K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 245.606 GC heap after
{Heap after GC invocations=221 (full 3):
 par new generation   total 76672K, used 1307K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  15% used [0x0000000084ae0000, 0x0000000084c26c18, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73251K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 246.598 GC heap before
{Heap before GC invocations=221 (full 3):
 par new generation   total 76672K, used 69266K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,  99% used [0x0000000080000000, 0x000000008425df90, 0x0000000084290000)
  from space 8512K,  15% used [0x0000000084ae0000, 0x0000000084c26c18, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73253K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 246.612 GC heap after
{Heap after GC invocations=222 (full 3):
 par new generation   total 76672K, used 7361K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  86% used [0x0000000084290000, 0x00000000849c0740, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73253K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 246.706 GC heap before
{Heap before GC invocations=222 (full 3):
 par new generation   total 76672K, used 75521K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K, 100% used [0x0000000080000000, 0x0000000084290000, 0x0000000084290000)
  from space 8512K,  86% used [0x0000000084290000, 0x00000000849c0740, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 174623K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73253K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 246.722 GC heap after
{Heap after GC invocations=223 (full 3):
 par new generation   total 76672K, used 5070K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  59% used [0x0000000084ae0000, 0x0000000084fd3a38, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 177343K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73253K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 247.002 GC heap before
{Heap before GC invocations=223 (full 3):
 par new generation   total 76672K, used 73183K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,  99% used [0x0000000080000000, 0x0000000084284450, 0x0000000084290000)
  from space 8512K,  59% used [0x0000000084ae0000, 0x0000000084fd3a38, 0x0000000085330000)
  to   space 8512K,   0% used [0x0000000084290000, 0x0000000084290000, 0x0000000084ae0000)
 concurrent mark-sweep generation total 2011968K, used 177343K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73262K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}
Event: 247.016 GC heap after
{Heap after GC invocations=224 (full 3):
 par new generation   total 76672K, used 1396K [0x0000000080000000, 0x0000000085330000, 0x0000000085330000)
  eden space 68160K,   0% used [0x0000000080000000, 0x0000000080000000, 0x0000000084290000)
  from space 8512K,  16% used [0x0000000084290000, 0x00000000843ed2e8, 0x0000000084ae0000)
  to   space 8512K,   0% used [0x0000000084ae0000, 0x0000000084ae0000, 0x0000000085330000)
 concurrent mark-sweep generation total 2011968K, used 177343K [0x0000000085330000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 73262K, capacity 76804K, committed 77352K, reserved 1116160K
  class space    used 8718K, capacity 10325K, committed 10336K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 199.208 Thread 0x00007f8c1860c000 Uncommon trap: reason=bimorphic_or_optimized_type_check action=maybe_recompile pc=0x00007f8c43d0d028 method=org.apache.lucene.util.packed.MonotonicLongValues.get(II)J @ 23 c2
Event: 199.208 Thread 0x00007f8c1860c000 Uncommon trap: reason=bimorphic_or_optimized_type_check action=maybe_recompile pc=0x00007f8c43d0dda8 method=org.apache.lucene.util.packed.MonotonicLongValues.get(II)J @ 23 c2
Event: 201.194 Thread 0x00007f8c549bf000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f8c43caceb4 method=java.util.Comparators$NaturalOrderComparator.compare(Ljava/lang/Object;Ljava/lang/Object;)I @ 2 c2
Event: 201.194 Thread 0x00007f8c549bf000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f8c43caceb4 method=java.util.Comparators$NaturalOrderComparator.compare(Ljava/lang/Object;Ljava/lang/Object;)I @ 2 c2
Event: 201.194 Thread 0x00007f8c549bf000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f8c43caceb4 method=java.util.Comparators$NaturalOrderComparator.compare(Ljava/lang/Object;Ljava/lang/Object;)I @ 2 c2
Event: 201.194 Thread 0x00007f8c549bf000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f8c43caceb4 method=java.util.Comparators$NaturalOrderComparator.compare(Ljava/lang/Object;Ljava/lang/Object;)I @ 2 c2
Event: 214.831 Thread 0x00007f8c186fe000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c43d90c14 method=org.apache.lucene.store.GrowableByteArrayDataOutput.writeBytes([BII)V @ 15 c2
Event: 215.501 Thread 0x00007f8c186ff000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c43c552e0 method=org.apache.lucene.store.RateLimitedIndexOutput.checkRate()V @ 9 c2
Event: 220.893 Thread 0x00007f8bfc014800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c43d52988 method=io.netty.util.internal.shaded.org.jctools.queues.atomic.BaseMpscLinkedAtomicArrayQueue.isEmpty()Z @ 9 c2
Event: 245.900 Thread 0x00007f8c55cf6000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f8c43ddc2c0 method=java.util.HashSet.remove(Ljava/lang/Object;)Z @ 11 c2

Classes redefined (0 events):
No events

Internal exceptions (10 events):
Event: 122.204 Thread 0x00007f8c55974800 Implicit null exception at 0x00007f8c43a97d28 to 0x00007f8c43a98302
Event: 124.917 Thread 0x00007f8c55d1f800 Exception <a 'sun/nio/fs/UnixException'{0x0000000080912250}> (0x0000000080912250) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 125.000 Thread 0x00007f8c55d1f800 Exception <a 'sun/nio/fs/UnixException'{0x0000000080919e20}> (0x0000000080919e20) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 129.817 Thread 0x00007f8c55d1f800 Exception <a 'sun/nio/fs/UnixException'{0x0000000083fb12e0}> (0x0000000083fb12e0) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 129.824 Thread 0x00007f8c55d1f800 Exception <a 'sun/nio/fs/UnixException'{0x0000000083fb8eb0}> (0x0000000083fb8eb0) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 131.820 Thread 0x00007f8c18063800 Implicit null exception at 0x00007f8c43abc2bc to 0x00007f8c43ac8e5a
Event: 144.216 Thread 0x00007f8c2c00b000 Implicit null exception at 0x00007f8c43b023b0 to 0x00007f8c43b0d8ae
Event: 163.709 Thread 0x00007f8c18357000 Implicit null exception at 0x00007f8c43c0f8d4 to 0x00007f8c43c0fb1e
Event: 184.106 Thread 0x00007f8c04005800 Exception <a 'sun/nio/fs/UnixException'{0x0000000081fba9d0}> (0x0000000081fba9d0) thrown at [/scratch/opt/mach5/mesos/work_dir/slaves/e283bbd4-d3d7-4cc0-a152-4405cadfe7cb-S457/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2a1032ca-d005-43
Event: 245.899 Thread 0x00007f8c55cf6000 Implicit null exception at 0x00007f8c43ddc25c to 0x00007f8c43ddc2ae

Events (10 events):
Event: 246.622 Thread 0x00007f8c18123000 DEOPT PACKING pc=0x00007f8c3d0e4dfb sp=0x00007f8bfbdfc3f0
Event: 246.622 Thread 0x00007f8c18123000 DEOPT UNPACKING pc=0x00007f8c3bb4fe4a sp=0x00007f8bfbdfb918 mode 0
Event: 246.702 Thread 0x00007f8c18125000 Thread added: 0x00007f8c18125000
Event: 246.706 Executing VM operation: GenCollectForAllocation
Event: 246.722 Executing VM operation: GenCollectForAllocation done
Event: 246.793 Executing VM operation: RevokeBias
Event: 246.793 Executing VM operation: RevokeBias done
Event: 247.002 Executing VM operation: GenCollectForAllocation
Event: 247.016 Executing VM operation: GenCollectForAllocation done
Event: 247.097 Thread 0x00007f8c18123000 Thread exited: 0x00007f8c18123000


Dynamic libraries:
00400000-00401000 r-xp 00000000 103:02 2625764                           /opt/jdk-10.0.1/bin/java
00600000-00601000 rw-p 00000000 103:02 2625764                           /opt/jdk-10.0.1/bin/java
006f8000-0073e000 rw-p 00000000 00:00 0                                  [heap]
80000000-100a18000 rw-p 00000000 00:00 0
100a18000-140000000 ---p 00000000 00:00 0
7f8be8fda000-7f8be97ff000 r--s 00000000 103:03 2752559                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc_Lucene70_0.dvd
7f8be97ff000-7f8bea36d000 r--s 00000000 103:03 2752831                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc.fdt
7f8bea36d000-7f8bead09000 r--s 00000000 103:03 2752840                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc_Lucene50_0.tim
7f8bead09000-7f8beb112000 r--s 00000000 103:03 2752563                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc_Lucene50_0.doc
7f8beb311000-7f8beb7b9000 r--s 00000000 103:03 2752698                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r_Lucene70_0.dvd
7f8beb7b9000-7f8bebc77000 r--s 00000000 103:03 2752851                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r.fdt
7f8bebc77000-7f8bec210000 r--s 00000000 103:03 2752878                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r_Lucene50_0.tim
7f8bec23b000-7f8bec403000 r--s 00000000 103:03 2752832                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc_Lucene50_0.pos
7f8bee107000-7f8bee10b000 ---p 00000000 00:00 0
7f8bee10b000-7f8bee208000 rw-p 00000000 00:00 0
7f8bee685000-7f8bee7c1000 r--s 00000000 103:03 2752828                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n_Lucene70_0.dvd
7f8bee7c1000-7f8bee936000 r--s 00000000 103:03 2753020                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n_Lucene50_0.tim
7f8beea6c000-7f8bef277000 r--s 00000000 103:03 2752727                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt_Lucene70_0.dvd
7f8bef277000-7f8befd4f000 r--s 00000000 103:03 2752908                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt.fdt
7f8befd4f000-7f8bf0213000 r--s 00000000 103:03 2752774                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz_Lucene70_0.dvd
7f8bf0213000-7f8bf0271000 r--s 00000000 103:03 2752803                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz.dim
7f8bf0271000-7f8bf08e7000 r--s 00000000 103:03 2752781                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz.fdt
7f8bf08e7000-7f8bf0956000 r--s 00000000 103:03 2752871                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz.nvd
7f8bf0956000-7f8bf0f0d000 r--s 00000000 103:03 2752859                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz_Lucene50_0.tim
7f8bf0f0d000-7f8bf1025000 r--s 00000000 103:03 2752855                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz_Lucene50_0.pos
7f8bf1025000-7f8bf128e000 r--s 00000000 103:03 2752854                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_nz_Lucene50_0.doc
7f8bf12c1000-7f8bf12c5000 ---p 00000000 00:00 0
7f8bf12c5000-7f8bf13c2000 rw-p 00000000 00:00 0
7f8bf13c2000-7f8bf13c6000 ---p 00000000 00:00 0
7f8bf13c6000-7f8bf14c3000 rw-p 00000000 00:00 0
7f8bf14c3000-7f8bf1640000 rw-p 00000000 00:00 0
7f8bf1640000-7f8bf16c3000 ---p 00000000 00:00 0
7f8bf16c3000-7f8bf2671000 r--s 00000000 103:03 2752860                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_1.cfs
7f8bf2671000-7f8bf356b000 r--s 00000000 103:03 2752863                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_0.cfs
7f8bf356b000-7f8bf356f000 ---p 00000000 00:00 0
7f8bf356f000-7f8bf366c000 rw-p 00000000 00:00 0
7f8bf366c000-7f8bf3670000 ---p 00000000 00:00 0
7f8bf3670000-7f8bf376d000 rw-p 00000000 00:00 0
7f8bf3773000-7f8bf386f000 r--s 00000000 103:03 2752989                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n.fdt
7f8bf386f000-7f8bf3904000 r--s 00000000 103:03 2752947                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc.dim
7f8bf3904000-7f8bf3b04000 rw-p 00000000 00:00 0
7f8bf3b04000-7f8bf3b08000 ---p 00000000 00:00 0
7f8bf3b08000-7f8bf3c05000 rw-p 00000000 00:00 0
7f8bf3c05000-7f8bf3e05000 rw-p 00000000 00:00 0
7f8bf3e2b000-7f8bf3ee7000 r--s 00000000 103:03 2752735                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_fc.nvd
7f8bf40ec000-7f8bf4a6b000 r--s 00000000 103:03 2753085                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt_Lucene50_0.tim
7f8bf4a6b000-7f8bf5040000 r--s 00000000 103:03 2752644                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_hk.cfs
7f8bf521e000-7f8bf529a000 r--s 00000000 103:03 2752970                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_d5.cfs
7f8bf53f1000-7f8bf54ee000 r--s 00000000 103:03 2753045                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_qp.cfs
7f8bf54ee000-7f8bf555f000 r--s 00000000 103:03 2753018                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n_Lucene50_0.doc
7f8bf555f000-7f8bf55f9000 r--s 00000000 103:03 2753091                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5p.cfs
7f8bf560b000-7f8bf568c000 r--s 00000000 103:03 2752967                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_df.cfs
7f8bf56d6000-7f8bf5767000 r--s 00000000 103:03 2752745                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt.dim
7f8bf5767000-7f8bf57a6000 r--s 00000000 103:03 2753005                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r.dim
7f8bf57a6000-7f8bf57f9000 r--s 00000000 103:03 2752973                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r.nvd
7f8bf57f9000-7f8bf58a9000 r--s 00000000 103:03 2752877                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r_Lucene50_0.pos
7f8bf58a9000-7f8bf5a60000 r--s 00000000 103:03 2752847                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_3r_Lucene50_0.doc
7f8bf5a6c000-7f8bf6b0d000 r--s 00000000 103:03 2752714                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_0.cfs
7f8bf6b0d000-7f8bf7785000 r--s 00000000 103:03 2752639                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_hb.cfs
7f8bf77b5000-7f8bf77b9000 ---p 00000000 00:00 0
7f8bf77b9000-7f8bf78b6000 rw-p 00000000 00:00 0
7f8bf78f8000-7f8bf79aa000 r--s 00000000 103:03 2753010                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt.nvd
7f8bf79aa000-7f8bf7b5c000 r--s 00000000 103:03 2753084                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt_Lucene50_0.pos
7f8bf7b5c000-7f8bf7f38000 r--s 00000000 103:03 2753083                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_bt_Lucene50_0.doc
7f8bf7f6d000-7f8bf7fc8000 r--s 00000000 103:03 2753015                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_h9.cfs
7f8bf7fc8000-7f8bf7fcc000 ---p 00000000 00:00 0
7f8bf7fcc000-7f8bf80c9000 rw-p 00000000 00:00 0
7f8bf80c9000-7f8bf80cd000 ---p 00000000 00:00 0
7f8bf80cd000-7f8bf81ca000 rw-p 00000000 00:00 0
7f8bf81ca000-7f8bf83ca000 rw-p 00000000 00:00 0
7f8bf83ca000-7f8bf83ce000 ---p 00000000 00:00 0
7f8bf83ce000-7f8bf84cb000 rw-p 00000000 00:00 0
7f8bf84cb000-7f8bf84cf000 ---p 00000000 00:00 0
7f8bf84cf000-7f8bf85cc000 rw-p 00000000 00:00 0
7f8bf85cc000-7f8bf87cc000 rw-p 00000000 00:00 0
7f8bf87cc000-7f8bf87d0000 ---p 00000000 00:00 0
7f8bf87d0000-7f8bf88cd000 rw-p 00000000 00:00 0
7f8bf88cd000-7f8bf88d1000 ---p 00000000 00:00 0
7f8bf88d1000-7f8bf89ce000 rw-p 00000000 00:00 0
7f8bf89ce000-7f8bf89d2000 ---p 00000000 00:00 0
7f8bf89d2000-7f8bf8acf000 rw-p 00000000 00:00 0
7f8bf8acf000-7f8bf8ad3000 ---p 00000000 00:00 0
7f8bf8ad3000-7f8bf8bd0000 rw-p 00000000 00:00 0
7f8bf8bd0000-7f8bf8bd4000 ---p 00000000 00:00 0
7f8bf8bd4000-7f8bf8cd1000 rw-p 00000000 00:00 0
7f8bf8cd1000-7f8bf8cd5000 ---p 00000000 00:00 0
7f8bf8cd5000-7f8bf8dd2000 rw-p 00000000 00:00 0
7f8bf8dd2000-7f8bf8fd2000 rw-p 00000000 00:00 0
7f8bf8fd2000-7f8bf8fd6000 ---p 00000000 00:00 0
7f8bf8fd6000-7f8bf90d3000 rw-p 00000000 00:00 0
7f8bf90d3000-7f8bf90d7000 ---p 00000000 00:00 0
7f8bf90d7000-7f8bf91d4000 rw-p 00000000 00:00 0
7f8bf91d4000-7f8bf91d8000 ---p 00000000 00:00 0
7f8bf91d8000-7f8bfb2d7000 rw-p 00000000 00:00 0
7f8bfb2d7000-7f8bfb2db000 ---p 00000000 00:00 0
7f8bfb2db000-7f8bfb3d8000 rw-p 00000000 00:00 0
7f8bfb3d8000-7f8bfb3dc000 ---p 00000000 00:00 0
7f8bfb3dc000-7f8bfb4d9000 rw-p 00000000 00:00 0
7f8bfb4d9000-7f8bfb4dd000 ---p 00000000 00:00 0
7f8bfb4dd000-7f8bfb5da000 rw-p 00000000 00:00 0
7f8bfb5da000-7f8bfb5de000 ---p 00000000 00:00 0
7f8bfb5de000-7f8bfb6db000 rw-p 00000000 00:00 0
7f8bfb6db000-7f8bfb6df000 ---p 00000000 00:00 0
7f8bfb6df000-7f8bfb7dc000 rw-p 00000000 00:00 0
7f8bfb7dc000-7f8bfb7e0000 ---p 00000000 00:00 0
7f8bfb7e0000-7f8bfb8dd000 rw-p 00000000 00:00 0
7f8bfb8dd000-7f8bfb8f3000 r-xp 00000000 103:02 2107384                   /usr/lib64/libresolv-2.17.so
7f8bfb8f3000-7f8bfbaf2000 ---p 00016000 103:02 2107384                   /usr/lib64/libresolv-2.17.so
7f8bfbaf2000-7f8bfbaf3000 r--p 00015000 103:02 2107384                   /usr/lib64/libresolv-2.17.so
7f8bfbaf3000-7f8bfbaf4000 rw-p 00016000 103:02 2107384                   /usr/lib64/libresolv-2.17.so
7f8bfbaf4000-7f8bfbaf6000 rw-p 00000000 00:00 0
7f8bfbaf6000-7f8bfbafb000 r-xp 00000000 103:02 2107325                   /usr/lib64/libnss_dns-2.17.so
7f8bfbafb000-7f8bfbcfb000 ---p 00005000 103:02 2107325                   /usr/lib64/libnss_dns-2.17.so
7f8bfbcfb000-7f8bfbcfc000 r--p 00005000 103:02 2107325                   /usr/lib64/libnss_dns-2.17.so
7f8bfbcfc000-7f8bfbcfd000 rw-p 00006000 103:02 2107325                   /usr/lib64/libnss_dns-2.17.so
7f8bfbcfd000-7f8bfbd01000 ---p 00000000 00:00 0
7f8bfbd01000-7f8bfbdfe000 rw-p 00000000 00:00 0
7f8bfbdfe000-7f8bfbe02000 ---p 00000000 00:00 0
7f8bfbe02000-7f8bfbeff000 rw-p 00000000 00:00 0
7f8bfbeff000-7f8bfbf03000 ---p 00000000 00:00 0
7f8bfbf03000-7f8bfc000000 rw-p 00000000 00:00 0
7f8bfc000000-7f8bfc052000 rw-p 00000000 00:00 0
7f8bfc052000-7f8c00000000 ---p 00000000 00:00 0
7f8c00000000-7f8c00026000 rw-p 00000000 00:00 0
7f8c00026000-7f8c04000000 ---p 00000000 00:00 0
7f8c04000000-7f8c0412f000 rw-p 00000000 00:00 0
7f8c0412f000-7f8c08000000 ---p 00000000 00:00 0
7f8c08000000-7f8c08058000 rw-p 00000000 00:00 0
7f8c08058000-7f8c0c000000 ---p 00000000 00:00 0
7f8c0c000000-7f8c0c094000 rw-p 00000000 00:00 0
7f8c0c094000-7f8c10000000 ---p 00000000 00:00 0
7f8c10000000-7f8c10026000 rw-p 00000000 00:00 0
7f8c10026000-7f8c14000000 ---p 00000000 00:00 0
7f8c14000000-7f8c172f3000 rw-p 00000000 00:00 0
7f8c172f3000-7f8c18000000 ---p 00000000 00:00 0
7f8c18000000-7f8c18a77000 rw-p 00000000 00:00 0
7f8c18a77000-7f8c1c000000 ---p 00000000 00:00 0
7f8c1c000000-7f8c1c042000 rw-p 00000000 00:00 0
7f8c1c042000-7f8c20000000 ---p 00000000 00:00 0
7f8c20000000-7f8c20028000 rw-p 00000000 00:00 0
7f8c20028000-7f8c24000000 ---p 00000000 00:00 0
7f8c24000000-7f8c2456a000 rw-p 00000000 00:00 0
7f8c2456a000-7f8c28000000 ---p 00000000 00:00 0
7f8c28000000-7f8c28070000 rw-p 00000000 00:00 0
7f8c28070000-7f8c2c000000 ---p 00000000 00:00 0
7f8c2c000000-7f8c2c021000 rw-p 00000000 00:00 0
7f8c2c021000-7f8c30000000 ---p 00000000 00:00 0
7f8c3002a000-7f8c30076000 r--s 00000000 103:03 2752898                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_gz.cfs
7f8c30098000-7f8c300a8000 r--s 00000000 103:03 2752834                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n.dim
7f8c300a8000-7f8c300cc000 r--s 00000000 103:03 2752880                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n.nvd
7f8c300d4000-7f8c300e2000 r--s 00000000 103:03 2752893                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dm.cfs
7f8c300e2000-7f8c300f0000 r--s 00000000 103:03 2753078                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/3/index/_ha.cfs
7f8c300f0000-7f8c300ff000 r--s 00000000 103:03 2753054                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5w.cfs
7f8c300ff000-7f8c30103000 ---p 00000000 00:00 0
7f8c30103000-7f8c30200000 rw-p 00000000 00:00 0
7f8c30200000-7f8c30400000 rw-p 00000000 00:00 0
7f8c30400000-7f8c30600000 rw-p 00000000 00:00 0
7f8c30600000-7f8c30800000 rw-p 00000000 00:00 0
7f8c30800000-7f8c30a00000 rw-p 00000000 00:00 0
7f8c30a00000-7f8c30c00000 rw-p 00000000 00:00 0
7f8c30c00000-7f8c30e00000 rw-p 00000000 00:00 0
7f8c30e00000-7f8c31000000 rw-p 00000000 00:00 0
7f8c31000000-7f8c31200000 rw-p 00000000 00:00 0
7f8c31200000-7f8c313f5000 rw-p 00000000 00:00 0
7f8c313f5000-7f8c31400000 ---p 00000000 00:00 0
7f8c31400000-7f8c31600000 rw-p 00000000 00:00 0
7f8c31600000-7f8c31800000 rw-p 00000000 00:00 0
7f8c31800000-7f8c31a00000 rw-p 00000000 00:00 0
7f8c31a00000-7f8c31c00000 rw-p 00000000 00:00 0
7f8c31c00000-7f8c31e00000 rw-p 00000000 00:00 0
7f8c31e00000-7f8c32000000 rw-p 00000000 00:00 0
7f8c32000000-7f8c34000000 rw-p 00000000 00:00 0
7f8c34000000-7f8c34021000 rw-p 00000000 00:00 0
7f8c34021000-7f8c38000000 ---p 00000000 00:00 0
7f8c38003000-7f8c3802c000 r--s 00000000 103:03 2753019                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5n_Lucene50_0.pos
7f8c3802f000-7f8c3803c000 r--s 00000000 103:03 2753048                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5y.cfs
7f8c3803c000-7f8c38049000 r--s 00000000 103:03 2753050                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_qr.cfs
7f8c38049000-7f8c3804e000 r--s 00000000 103:03 2753068                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/0/index/_qq.cfs
7f8c3804e000-7f8c38054000 r--s 00000000 103:03 2753074                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dk.cfs
7f8c38059000-7f8c3805e000 r--s 00000000 103:03 2753060                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5w.cfs
7f8c38064000-7f8c3806c000 r--s 00000000 103:03 2753026                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dj.cfs
7f8c38073000-7f8c3807a000 r--s 00000000 103:03 2753031                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5t.cfs
7f8c3807a000-7f8c38080000 r--s 00000000 103:03 2753033                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5v.cfs
7f8c3808c000-7f8c38093000 r--s 00000000 103:03 2752969                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_di.cfs
7f8c38099000-7f8c3809f000 r--s 00000000 103:03 2752995                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5s.cfs
7f8c3809f000-7f8c380a5000 r--s 00000000 103:03 2752997                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5u.cfs
7f8c380a5000-7f8c380ac000 r--s 00000000 103:03 2752975                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dg.cfs
7f8c380b9000-7f8c380c3000 r--s 00000000 103:03 2752983                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5s.cfs
7f8c380c3000-7f8c382c3000 rw-p 00000000 00:00 0
7f8c382c3000-7f8c382c4000 r-xp 00000000 103:02 2626260                   /opt/jdk-10.0.1/lib/libextnet.so
7f8c382c4000-7f8c384c3000 ---p 00001000 103:02 2626260                   /opt/jdk-10.0.1/lib/libextnet.so
7f8c384c3000-7f8c384c4000 rw-p 00000000 103:02 2626260                   /opt/jdk-10.0.1/lib/libextnet.so
7f8c384c4000-7f8c386c4000 rw-p 00000000 00:00 0
7f8c386c4000-7f8c388c4000 rw-p 00000000 00:00 0
7f8c388c4000-7f8c38ac4000 rw-p 00000000 00:00 0
7f8c38ac4000-7f8c38adc000 r-xp 00000000 103:02 6292735                   /tmp/elasticsearch.9Ts2lgAF/jna--1985354563/jna2246371085384706757.tmp (deleted)
7f8c38adc000-7f8c38cdc000 ---p 00018000 103:02 6292735                   /tmp/elasticsearch.9Ts2lgAF/jna--1985354563/jna2246371085384706757.tmp (deleted)
7f8c38cdc000-7f8c38cdd000 rw-p 00018000 103:02 6292735                   /tmp/elasticsearch.9Ts2lgAF/jna--1985354563/jna2246371085384706757.tmp (deleted)
7f8c38cdd000-7f8c38edd000 rw-p 00000000 00:00 0
7f8c38edd000-7f8c390dd000 rw-p 00000000 00:00 0
7f8c390dd000-7f8c390e2000 r-xp 00000000 103:02 2626278                   /opt/jdk-10.0.1/lib/libmanagement_ext.so
7f8c390e2000-7f8c392e1000 ---p 00005000 103:02 2626278                   /opt/jdk-10.0.1/lib/libmanagement_ext.so
7f8c392e1000-7f8c392e2000 rw-p 00004000 103:02 2626278                   /opt/jdk-10.0.1/lib/libmanagement_ext.so
7f8c392e2000-7f8c392eb000 r--s 00000000 103:03 2752980                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5q.cfs
7f8c392f4000-7f8c392f7000 r--s 00000000 103:03 2752875                   /data/nodes/0/indices/6UUqrS5_RUSS5Ozy2tSYTg/0/index/_4.cfs
7f8c39304000-7f8c39308000 r--s 00000000 103:03 2753055                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5u.cfs
7f8c39308000-7f8c3930d000 r--s 00000000 103:03 2752895                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dh.cfs
7f8c3930d000-7f8c39313000 r--s 00000000 103:03 2752949                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5p.cfs
7f8c39313000-7f8c39319000 r--s 00000000 103:03 2752951                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5r.cfs
7f8c39319000-7f8c3931d000 ---p 00000000 00:00 0
7f8c3931d000-7f8c3941a000 rw-p 00000000 00:00 0
7f8c3941a000-7f8c3941e000 r-xp 00000000 103:02 2626276                   /opt/jdk-10.0.1/lib/libmanagement.so
7f8c3941e000-7f8c3961d000 ---p 00004000 103:02 2626276                   /opt/jdk-10.0.1/lib/libmanagement.so
7f8c3961d000-7f8c3961e000 rw-p 00003000 103:02 2626276                   /opt/jdk-10.0.1/lib/libmanagement.so
7f8c3961e000-7f8c3965b000 r-xp 00000000 103:02 2626287                   /opt/jdk-10.0.1/lib/libsunec.so
7f8c3965b000-7f8c3985b000 ---p 0003d000 103:02 2626287                   /opt/jdk-10.0.1/lib/libsunec.so
7f8c3985b000-7f8c39862000 rw-p 0003d000 103:02 2626287                   /opt/jdk-10.0.1/lib/libsunec.so
7f8c39862000-7f8c39a73000 rw-p 00000000 00:00 0
7f8c39a73000-7f8c39c73000 rw-p 00000000 00:00 0
7f8c39c73000-7f8c3bb27000 rw-p 00000000 00:00 0
7f8c3bb27000-7f8c3bd97000 rwxp 00000000 00:00 0
7f8c3bd97000-7f8c3c0b6000 ---p 00000000 00:00 0
7f8c3c0b6000-7f8c3d786000 rwxp 00000000 00:00 0
7f8c3d786000-7f8c435ee000 ---p 00000000 00:00 0
7f8c435ee000-7f8c43dee000 rwxp 00000000 00:00 0
7f8c43dee000-7f8c4ab27000 ---p 00000000 00:00 0
7f8c4ab27000-7f8c54000000 r--s 00000000 103:02 2626291                   /opt/jdk-10.0.1/lib/modules
7f8c54000000-7f8c55d4e000 rw-p 00000000 00:00 0
7f8c55d4e000-7f8c58000000 ---p 00000000 00:00 0
7f8c58009000-7f8c5800e000 r--s 00000000 103:03 2752916                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5r.cfs
7f8c58015000-7f8c5801a000 r--s 00000000 103:03 2752915                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5o.cfs
7f8c5801b000-7f8c58022000 r--s 00000000 103:03 2752926                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/2/index/_5v.cfs
7f8c58022000-7f8c58027000 r--s 00000000 103:03 2752920                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5t.cfs
7f8c58027000-7f8c5802c000 r--s 00000000 103:03 2752918                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5q.cfs
7f8c58031000-7f8c58035000 r--s 00000000 103:03 2752922                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/4/index/_dl.cfs
7f8c58035000-7f8c58039000 ---p 00000000 00:00 0
7f8c58039000-7f8c58136000 rw-p 00000000 00:00 0
7f8c58136000-7f8c5813a000 ---p 00000000 00:00 0
7f8c5813a000-7f8c58158000 rw-p 00000000 00:00 0
7f8c58158000-7f8c5815c000 ---p 00000000 00:00 0
7f8c5815c000-7f8c5817a000 rw-p 00000000 00:00 0
7f8c5817a000-7f8c58190000 r-xp 00000000 103:02 2626280                   /opt/jdk-10.0.1/lib/libnet.so
7f8c58190000-7f8c58390000 ---p 00016000 103:02 2626280                   /opt/jdk-10.0.1/lib/libnet.so
7f8c58390000-7f8c58391000 rw-p 00016000 103:02 2626280                   /opt/jdk-10.0.1/lib/libnet.so
7f8c58391000-7f8c583a1000 r-xp 00000000 103:02 2626281                   /opt/jdk-10.0.1/lib/libnio.so
7f8c583a1000-7f8c585a1000 ---p 00010000 103:02 2626281                   /opt/jdk-10.0.1/lib/libnio.so
7f8c585a1000-7f8c585a2000 rw-p 00010000 103:02 2626281                   /opt/jdk-10.0.1/lib/libnio.so
7f8c585a2000-7f8c585a6000 ---p 00000000 00:00 0
7f8c585a6000-7f8c586a3000 rw-p 00000000 00:00 0
7f8c586a3000-7f8c586a4000 ---p 00000000 00:00 0
7f8c586a4000-7f8c587a5000 rw-p 00000000 00:00 0
7f8c587a5000-7f8c587a9000 ---p 00000000 00:00 0
7f8c587a9000-7f8c588a6000 rw-p 00000000 00:00 0
7f8c588a6000-7f8c588aa000 ---p 00000000 00:00 0
7f8c588aa000-7f8c589a7000 rw-p 00000000 00:00 0
7f8c589a7000-7f8c589ab000 ---p 00000000 00:00 0
7f8c589ab000-7f8c58aa8000 rw-p 00000000 00:00 0
7f8c58aa8000-7f8c58aac000 ---p 00000000 00:00 0
7f8c58aac000-7f8c58ba9000 rw-p 00000000 00:00 0
7f8c58ba9000-7f8c58bad000 ---p 00000000 00:00 0
7f8c58bad000-7f8c58caa000 rw-p 00000000 00:00 0
7f8c58caa000-7f8c58cae000 ---p 00000000 00:00 0
7f8c58cae000-7f8c58dab000 rw-p 00000000 00:00 0
7f8c58dab000-7f8c58daf000 ---p 00000000 00:00 0
7f8c58daf000-7f8c58eac000 rw-p 00000000 00:00 0
7f8c58eac000-7f8c58ead000 ---p 00000000 00:00 0
7f8c58ead000-7f8c5907f000 rw-p 00000000 00:00 0
7f8c5907f000-7f8c5987f000 rw-p 00000000 00:00 0
7f8c5987f000-7f8c59880000 ---p 00000000 00:00 0
7f8c59880000-7f8c59a82000 rw-p 00000000 00:00 0
7f8c59a82000-7f8c59e59000 rw-p 00000000 00:00 0
7f8c59e59000-7f8c59f5a000 rw-p 00000000 00:00 0
7f8c59f5a000-7f8c5a35a000 rw-p 00000000 00:00 0
7f8c5a35a000-7f8c5a35b000 rw-p 00000000 00:00 0
7f8c5a35b000-7f8c5a35c000 ---p 00000000 00:00 0
7f8c5a35c000-7f8c5a45d000 rw-p 00000000 00:00 0
7f8c5a45d000-7f8c5a46d000 rw-p 00000000 00:00 0
7f8c5a46d000-7f8c5a548000 ---p 00000000 00:00 0
7f8c5a548000-7f8c5a576000 rw-p 00000000 00:00 0
7f8c5a576000-7f8c5a633000 ---p 00000000 00:00 0
7f8c5a633000-7f8c5a64d000 r-xp 00000000 103:02 2626271                   /opt/jdk-10.0.1/lib/libjimage.so
7f8c5a64d000-7f8c5a84c000 ---p 0001a000 103:02 2626271                   /opt/jdk-10.0.1/lib/libjimage.so
7f8c5a84c000-7f8c5a84e000 rw-p 00019000 103:02 2626271                   /opt/jdk-10.0.1/lib/libjimage.so
7f8c5a84e000-7f8c5a860000 rw-p 00000000 00:00 0
7f8c5a860000-7f8c5a867000 r-xp 00000000 103:02 2626290                   /opt/jdk-10.0.1/lib/libzip.so
7f8c5a867000-7f8c5aa66000 ---p 00007000 103:02 2626290                   /opt/jdk-10.0.1/lib/libzip.so
7f8c5aa66000-7f8c5aa67000 rw-p 00006000 103:02 2626290                   /opt/jdk-10.0.1/lib/libzip.so
7f8c5aa67000-7f8c5aa73000 r-xp 00000000 103:02 2107327                   /usr/lib64/libnss_files-2.17.so
7f8c5aa73000-7f8c5ac72000 ---p 0000c000 103:02 2107327                   /usr/lib64/libnss_files-2.17.so
7f8c5ac72000-7f8c5ac73000 r--p 0000b000 103:02 2107327                   /usr/lib64/libnss_files-2.17.so
7f8c5ac73000-7f8c5ac74000 rw-p 0000c000 103:02 2107327                   /usr/lib64/libnss_files-2.17.so
7f8c5ac74000-7f8c5ac7a000 rw-p 00000000 00:00 0
7f8c5ac7a000-7f8c5aca4000 r-xp 00000000 103:02 2626267                   /opt/jdk-10.0.1/lib/libjava.so
7f8c5aca4000-7f8c5aea4000 ---p 0002a000 103:02 2626267                   /opt/jdk-10.0.1/lib/libjava.so
7f8c5aea4000-7f8c5aea6000 rw-p 0002a000 103:02 2626267                   /opt/jdk-10.0.1/lib/libjava.so
7f8c5aea6000-7f8c5aeb3000 r-xp 00000000 103:02 2626289                   /opt/jdk-10.0.1/lib/libverify.so
7f8c5aeb3000-7f8c5b0b2000 ---p 0000d000 103:02 2626289                   /opt/jdk-10.0.1/lib/libverify.so
7f8c5b0b2000-7f8c5b0b4000 rw-p 0000c000 103:02 2626289                   /opt/jdk-10.0.1/lib/libverify.so
7f8c5b0b4000-7f8c5b0bb000 r-xp 00000000 103:02 2107394                   /usr/lib64/librt-2.17.so
7f8c5b0bb000-7f8c5b2ba000 ---p 00007000 103:02 2107394                   /usr/lib64/librt-2.17.so
7f8c5b2ba000-7f8c5b2bb000 r--p 00006000 103:02 2107394                   /usr/lib64/librt-2.17.so
7f8c5b2bb000-7f8c5b2bc000 rw-p 00007000 103:02 2107394                   /usr/lib64/librt-2.17.so
7f8c5b2bc000-7f8c5b3bd000 r-xp 00000000 103:02 2107298                   /usr/lib64/libm-2.17.so
7f8c5b3bd000-7f8c5b5bc000 ---p 00101000 103:02 2107298                   /usr/lib64/libm-2.17.so
7f8c5b5bc000-7f8c5b5bd000 r--p 00100000 103:02 2107298                   /usr/lib64/libm-2.17.so
7f8c5b5bd000-7f8c5b5be000 rw-p 00101000 103:02 2107298                   /usr/lib64/libm-2.17.so
7f8c5b5be000-7f8c5c5ce000 r-xp 00000000 103:02 2626301                   /opt/jdk-10.0.1/lib/server/libjvm.so
7f8c5c5ce000-7f8c5c7cd000 ---p 01010000 103:02 2626301                   /opt/jdk-10.0.1/lib/server/libjvm.so
7f8c5c7cd000-7f8c5c8aa000 r--p 0100f000 103:02 2626301                   /opt/jdk-10.0.1/lib/server/libjvm.so
7f8c5c8aa000-7f8c5c8e3000 rw-p 010ec000 103:02 2626301                   /opt/jdk-10.0.1/lib/server/libjvm.so
7f8c5c8e3000-7f8c5c940000 rw-p 00000000 00:00 0
7f8c5c940000-7f8c5cb03000 r-xp 00000000 103:02 2107190                   /usr/lib64/libc-2.17.so
7f8c5cb03000-7f8c5cd02000 ---p 001c3000 103:02 2107190                   /usr/lib64/libc-2.17.so
7f8c5cd02000-7f8c5cd06000 r--p 001c2000 103:02 2107190                   /usr/lib64/libc-2.17.so
7f8c5cd06000-7f8c5cd08000 rw-p 001c6000 103:02 2107190                   /usr/lib64/libc-2.17.so
7f8c5cd08000-7f8c5cd0d000 rw-p 00000000 00:00 0
7f8c5cd0d000-7f8c5cd0f000 r-xp 00000000 103:02 2107217                   /usr/lib64/libdl-2.17.so
7f8c5cd0f000-7f8c5cf0f000 ---p 00002000 103:02 2107217                   /usr/lib64/libdl-2.17.so
7f8c5cf0f000-7f8c5cf10000 r--p 00002000 103:02 2107217                   /usr/lib64/libdl-2.17.so
7f8c5cf10000-7f8c5cf11000 rw-p 00003000 103:02 2107217                   /usr/lib64/libdl-2.17.so
7f8c5cf11000-7f8c5cf1f000 r-xp 00000000 103:02 2626252                   /opt/jdk-10.0.1/lib/jli/libjli.so
7f8c5cf1f000-7f8c5d11f000 ---p 0000e000 103:02 2626252                   /opt/jdk-10.0.1/lib/jli/libjli.so
7f8c5d11f000-7f8c5d120000 rw-p 0000e000 103:02 2626252                   /opt/jdk-10.0.1/lib/jli/libjli.so
7f8c5d120000-7f8c5d137000 r-xp 00000000 103:02 2107375                   /usr/lib64/libpthread-2.17.so
7f8c5d137000-7f8c5d336000 ---p 00017000 103:02 2107375                   /usr/lib64/libpthread-2.17.so
7f8c5d336000-7f8c5d337000 r--p 00016000 103:02 2107375                   /usr/lib64/libpthread-2.17.so
7f8c5d337000-7f8c5d338000 rw-p 00017000 103:02 2107375                   /usr/lib64/libpthread-2.17.so
7f8c5d338000-7f8c5d33c000 rw-p 00000000 00:00 0
7f8c5d33c000-7f8c5d351000 r-xp 00000000 103:02 2107454                   /usr/lib64/libz.so.1.2.7
7f8c5d351000-7f8c5d550000 ---p 00015000 103:02 2107454                   /usr/lib64/libz.so.1.2.7
7f8c5d550000-7f8c5d551000 r--p 00014000 103:02 2107454                   /usr/lib64/libz.so.1.2.7
7f8c5d551000-7f8c5d552000 rw-p 00015000 103:02 2107454                   /usr/lib64/libz.so.1.2.7
7f8c5d552000-7f8c5d574000 r-xp 00000000 103:02 2107166                   /usr/lib64/ld-2.17.so
7f8c5d575000-7f8c5d577000 r--s 00000000 103:03 2752869                   /data/nodes/0/indices/6UUqrS5_RUSS5Ozy2tSYTg/0/index/_3.cfs
7f8c5d57a000-7f8c5d57f000 r--s 00000000 103:03 2752934                   /data/nodes/0/indices/iQV-I-DjST-grD77Jo6Rrg/1/index/_5x.cfs
7f8c5d57f000-7f8c5d580000 rwxp 00000000 00:00 0
7f8c5d580000-7f8c5d581000 r--p 00000000 00:00 0
7f8c5d581000-7f8c5d5da000 rw-p 00000000 00:00 0
7f8c5d5da000-7f8c5d65a000 rw-p 00000000 00:00 0
7f8c5d65a000-7f8c5d661000 ---p 00000000 00:00 0
7f8c5d661000-7f8c5d669000 rw-s 00000000 103:02 6292730                   /tmp/hsperfdata_elasticsearch/1
7f8c5d669000-7f8c5d66d000 ---p 00000000 00:00 0
7f8c5d66d000-7f8c5d770000 rw-p 00000000 00:00 0
7f8c5d770000-7f8c5d771000 ---p 00000000 00:00 0
7f8c5d771000-7f8c5d772000 r--p 00000000 00:00 0
7f8c5d772000-7f8c5d773000 rw-p 00000000 00:00 0
7f8c5d773000-7f8c5d774000 r--p 00021000 103:02 2107166                   /usr/lib64/ld-2.17.so
7f8c5d774000-7f8c5d775000 rw-p 00022000 103:02 2107166                   /usr/lib64/ld-2.17.so
7f8c5d775000-7f8c5d776000 rw-p 00000000 00:00 0
7ffc0a637000-7ffc0a659000 rw-p 00000000 00:00 0                          [stack]
7ffc0a7b1000-7ffc0a7b3000 r--p 00000000 00:00 0                          [vvar]
7ffc0a7b3000-7ffc0a7b5000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


VM Arguments:
jvm_args: -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.9Ts2lgAF -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Des.cgroups.hierarchy.override=/ -Xms2g -Xmx2g -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=tar
java_command: org.elasticsearch.bootstrap.Elasticsearch -Ecluster.name=tds-es-db -Expack.watcher.enabled=false -Expack.ml.enabled=false -Expack.logstash.enabled=false -Ediscovery.zen.ping.unicast.hosts=elasticsearch-discovery -Ehttp.enabled=false -Epath.data=/data -Enode.data=true -Enode.master=false -Enode.name=es-data-us-east-1c-0 -Enetwork.host=_eth0:ipv4_ -Expack.graph.enabled=false -Ediscovery.zen.minimum_master_nodes=2 -Expack.security.enabled=false -Enode.ingest=false
java_class_path (initial): /usr/share/elasticsearch/lib/spatial4j-0.7.jar:/usr/share/elasticsearch/lib/lucene-backward-codecs-7.3.1.jar:/usr/share/elasticsearch/lib/elasticsearch-core-6.3.0.jar:/usr/share/elasticsearch/lib/jackson-core-2.8.10.jar:/usr/share/elasticsearch/lib/lucene-analyzers-common-7.3.1.jar:/usr/share/elasticsearch/lib/log4j-1.2-api-2.9.1.jar:/usr/share/elasticsearch/lib/HdrHistogram-2.1.9.jar:/usr/share/elasticsearch/lib/elasticsearch-x-content-6.3.0.jar:/usr/share/elasticsearch/lib/lucene-highlighter-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-memory-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-queryparser-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-misc-7.3.1.jar:/usr/share/elasticsearch/lib/snakeyaml-1.17.jar:/usr/share/elasticsearch/lib/lucene-spatial3d-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-spatial-extras-7.3.1.jar:/usr/share/elasticsearch/lib/plugin-classloader-6.3.0.jar:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.8.10.jar:/usr/share/elasticsearch/lib/jts-core-1.15.0.jar:/usr/share/elasticsearch/lib/lucene-grouping-7.3.1.jar:/usr/share/elasticsearch/lib/log4j-core-2.9.1.jar:/usr/share/elasticsearch/lib/lucene-sandbox-7.3.1.jar:/usr/share/elasticsearch/lib/jna-4.5.1.jar:/usr/share/elasticsearch/lib/elasticsearch-6.3.0.jar:/usr/share/elasticsearch/lib/lucene-queries-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-core-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-spatial-7.3.1.jar:/usr/share/elasticsearch/lib/lucene-join-7.3.1.jar:/usr/share/elasticsearch/lib/plugin-cli-6.3.0.jar:/usr/share/elasticsearch/lib/log4j-api-2.9.1.jar:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.8.10.jar:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.8.10.jar:/usr/share/elasticsearch/lib/hppc-0.7.1.jar:/usr/share/elasticsearch/lib/t-digest-3.2.jar:/usr/share/elasticsearch/lib/joda-time-2.9.9.jar:/usr/share/elasticsearch/lib/elasticsearch-cli-6.3.0.jar:/usr/share/elasticsearch/lib/jopt-simple-5.0.2.jar:/usr/share/elasticsearch
Launcher Type: SUN_STANDARD

Logging:
Log output configuration:
#0: stdout all=warning uptime,level,tags
#1: stderr all=off uptime,level,tags
#2: file=logs/gc.log gc+ref=info,gc+ref+start=info,gc+compaction=info,safepoint=info,gc+metaspace+alloc=info,gc+metaspace=info,gc+metaspace+freelist=info,gc+metaspace+freelist+blocks=info,gc+heap+exit=info,gc+cpu=info,gc+task+time=info,gc+task+thread=info,gc+jni=info,gc+stringdedup=info,gc+marking+start=info,gc+remset+exit=info,gc+remset=info,gc+mmu=info,gc+ihop=info,gc+phases+task=info,gc+phases+ref=info,gc+verify+start=info,gc+ergo+ihop=info,gc+refine=info,gc+ergo+refine=info,gc+stats=info,gc+ergo+cset=info,gc+humongous=info,gc+stringtable=info,gc+task+stats=info,gc+heap+region=info,gc+ergo+heap=info,gc+marking=info,gc+phases+verify=info,gc+phases+verify+start=info,gc+phases=info,gc+phases+start=info,gc+survivor=info,gc+state=info,gc+promotion=info,gc+plab=info,gc+verify=info,gc+alloc=info,gc+liveness=info,gc+classhisto+start=info,gc+start=info,gc+classhisto=info,gc+barrier=info,gc+bot=info,gc+freelist+census=info,gc+freelist+stats=info,gc+region=info,gc+heap+coops=info,gc+heap=info,gc+age=trace,gc+ergo=info,gc=info,gc+tlab=info,gc+task=info,gc+freelist=info utctime,pid,tags filecount=32,filesize=64M

Environment Variables:
JAVA_HOME=/opt/jdk-10.0.1
PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Signal Handlers:
SIGSEGV: [libjvm.so+0xd3d000], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xd3d000], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xd3d000], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xb511c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xb511c0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xd3d000], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0xb51550], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0xb514e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.so+0xb514e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.so+0xb514e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0xb514e0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:CentOS Linux release 7.5.1804 (Core)
uname:Linux 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE infinity, NPROC 1048576, NOFILE 1048576, AS infinity, DATA infinity, FSIZE infinity
load average:0.48 1.16 1.00

/proc/meminfo:
MemTotal:        7876552 kB
MemFree:         1727588 kB
MemAvailable:    3184348 kB
Buffers:           73012 kB
Cached:          1497432 kB
SwapCached:            0 kB
Active:          4694876 kB
Inactive:        1150576 kB
Active(anon):    4275856 kB
Inactive(anon):     1952 kB
Active(file):     419020 kB
Inactive(file):  1148624 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             11684 kB
Writeback:             0 kB
AnonPages:       4273520 kB
Mapped:           320080 kB
Shmem:              2720 kB
Slab:             235824 kB
SReclaimable:     193912 kB
SUnreclaim:        41912 kB
KernelStack:        8592 kB
PageTables:        15824 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3938276 kB
Committed_AS:    6391232 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      141288 kB
DirectMap2M:     6891520 kB
DirectMap1G:     1048576 kB

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-1
cpu_memory_nodes: 0
active_processor_count: 1
cpu_quota: 50000
cpu_period: 100000
cpu_shares: 512
memory_limit_in_bytes: -1
memory_and_swap_limit_in_bytes: -2
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 2799579136
memory_max_usage_in_bytes: 2803335168


CPU:total 2 (initial active 1) (2 cores per cpu, 2 threads per core) family 6 model 85 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx, evex, fma
CPU Model and flags from /proc/cpuinfo:
model name	: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke

Memory: 4k page, physical 7876552k(1727588k free), swap 0k(0k free)

vm_info: OpenJDK 64-Bit Server VM (10.0.1+10) for linux-amd64 JRE (10.0.1+10), built on Mar 26 2018 18:10:26 by "mach5one" with gcc 4.9.2

END.

[jpountz]

Thanks. For the record, I asked on the Lucene lists as well in case someone already saw this segfault: https://markmail.org/message/n7rjurxzinyrhrwb.

[WUMUXIAN]

@jpountz Thanks for the info.

[lachlancooper]

I'm also encountering this error in a similar environment: official docker image 6.3.0, m5.large AWS EC2 instance, Ubuntu 16.04. The previous version I was running, 6.2.4, was not affected. I'm happy to provide more info if required.

[xrl]

I am also seeing this error on a m5.4xlarge in AWS. Linux 4.9. Debian stretch on the host. Using the docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.0 image.

[SMAPPER]

I am also experiencing this issue with the elasticsearch:6.3.0 docker image.

[xose]

I started experiencing this crash today:

I've been running the 6.3.0 Docker image on AWS i3 instances for weeks without any crashes. Today I launched some c5d instances to help with some hot indexes, and every single one of those has been crashing every couple minutes when I started indexing on them. I have since moved all indexes back to the i3s but I'll try to fetch some logs, although they were nearly identical to the ones already provided here.

Based on the comments above, this might be an issue with 6.3.0 and the M5/C5 instance family on AWS. Please let me know if I can help debug this in some way.

OS: Amazon Linux ECS 2018.03.a

[xose]

I've been doing some tests, and I believe I found the issue.

The 6.3.0 Docker image ships with Java 10.0.1, more specifically the generic binary targz from Oracle. I modified the Docker image to use the OpenJDK 8 binary provided by the base CentOS image package manager (1.8.0.171). It's now been running for ~1h without a single crash.

So the issue seems to be with Java 10 on AWS C5/M5 instances.