Periodic node crash. JVM crashes with SIGSEGV every few days.

[najik]

Elasticsearch version: 2.2.0

JVM version: java version "1.8.0_74" Java(TM) SE Runtime Environment (build 1.8.0_74-b02) Java HotSpot(TM) 64-Bit Server VM (build 25.74-b02, mixed mode)

OS version: Ubuntu 14.04.4 LTS, Linux 4.2.0-30-generic #36~14.04.1-Ubuntu SMP Fri Feb 26 18:49:23 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior: During normal operation, one of my nodes will randomly crash every few days with a JVM segfault

Steps to reproduce: None, problem is intermittent during normal operation. Workload is constant indexing (log data), very infrequent searches.

Provide logs (if relevant): See attached files and below
kaiju-v4.txt
hs_err_pid21172.txt

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  [thread 140246293530368 also had an error]
[thread 140241095403264 also had an error]
[thread 140236838332160 also had an error]
SIGSEGV (0xb) at pc=0x00007f8e650b3cd7, pid=21172, tid=140246295635712
#
# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x436cd7]  ciObjectFactory::create_new_metadata(Metadata*)+0x327
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /mnt/elastic/elasticsearch-2.2.0/hs_err_pid21172.log

[error occurred during error reporting , id 0xb]

#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)

[jasontedor]

That screams JVM bug. From your logs:

Current thread (0x00007f8e6021f000):  JavaThread "C2 CompilerThread5" daemon [_thread_in_vm, id=21253, stack(0x00007f8da291e000,0x00007f8da2a1f000)]

It's crashing in the server compiler. Maybe try upgrading your JVM to 8u77? If the issue persists, you'll have to open an OpenJDK issue. Sorry.