A fatal error has been detected by the Java Runtime Environment

[weishirongzhen]

System information

Ubuntu 20.04

OS : Linux
JVM : Oracle Corporation 1.8.0_341 amd64
Git : 349116e
Version : 4.5.2
Code : 17071

Run:
java -Xmx50g -XX:+UseConcMarkSweepGC -jar FullNode.jar -c main_net_config.conf

after somtime here is about 8min, crash with log:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f7072de5d45, pid=35351, tid=0x00007f6f639f0700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_341-b10) (build 1.8.0_341-b10)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.341-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 14006 C2 org.tron.core.db2.core.SnapshotImpl.get(Lorg/tron/core/db2/core/Snapshot;[B)[B (106 bytes) @ 0x00007f7072de5d45 [0x00007f7072de5b40+0x205]
#
# Core dump written. Default location: /mnt/tron_node/core or core.35351
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f6e5811e000):  JavaThread "pool-48-thread-8" [_thread_in_Java, id=35600, stack(0x00007f6f638f0000,0x00007f6f639f1000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 4 (unknown), si_addr: 0x00000002bb280444

Registers:
RAX=0x000000006b7c8d5c, RBX=0x0000000000000023, RCX=0x0000000057650087, RDX=0x00000000a05edb42
RSP=0x00007f6f639ef6d0, RBP=0x000000026d914320, RSI=0x00000001c2794800, RDI=0x0000000000000015
R8 =0x00000002b029a3b0, R9 =0x0000000056053476, R10=0x000000006b7ce620, R11=0x0000000000000220
R12=0x0000000000000000, R13=0x0000000000e13ca9, R14=0x0000000000000076, R15=0x00007f6e5811e000
RIP=0x00007f7072de5d45, EFLAGS=0x0000000000010206, CSGSFS=0x002b000000000033, ERR=0x0000000000000025
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f6f639ef6d0)
0x00007f6f639ef6d0:   00000001c2794800 00007f70384f2900
0x00007f6f639ef6e0:   00000007c033e7c8 0000000000000000
0x00007f6f639ef6f0:   00000007c044d3a8 00000002b93c5148
0x00007f6f639ef700:   00000001c27952c8 00000002170c4900
0x00007f6f639ef710:   00000002170c4900 000000016dad490b
0x00007f6f639ef720:   00000002132193a0 00007f7072cf3558
0x00007f6f639ef730:   000000028a715488 00007f70718a20d4
0x00007f6f639ef740:   00000001c2794690 00007f7000000000
0x00007f6f639ef750:   00000001c2794650 00000002131729f8
0x00007f6f639ef760:   0000000282f67718 00007f7072e10718
0x00007f6f639ef770:   00000001c2794690 000000021943b840
0x00007f6f639ef780:   0000000282f67718 00007f7073459178
0x00007f6f639ef790:   00000002170c4900 00007f70732b061c
0x00007f6f639ef7a0:   00000001c2794270 000000002fa8d9de
0x00007f6f639ef7b0:   00000000384f28e5 00007f70724c13e9
0x00007f6f639ef7c0:   00000001c2794800 00000002510de25c
0x00007f6f639ef7d0:   00000002886f12e0 00007f7000000000
0x00007f6f639ef7e0:   00000002170c4900 00000001c2794478
0x00007f6f639ef7f0:   00000001c2794458 00000001c2794478
0x00007f6f639ef800:   00000001c2794270 00007f7000000013
0x00007f6f639ef810:   0000000282f67718 00007f70723a2aec
0x00007f6f639ef820:   0000000100000005 0000000212996d20
0x00007f6f639ef830:   00000001c2794478 0000000211e69758
0x00007f6f639ef840:   00000002891e93d8 00000002891e9438
0x00007f6f639ef850:   0000000100000000 00000001c2794800
0x00007f6f639ef860:   000000004233ca7e 00000001c2794458
0x00007f6f639ef870:   00000001d6aa8a30 00000002190eff00
0x00007f6f639ef880:   00000001c0775ef0 00007f7071903dbc
0x00007f6f639ef890:   0000000282f67718 00007f7073483914
0x00007f6f639ef8a0:   0000000282f67718 00000002847b66f8
0x00007f6f639ef8b0:   0000000212996d20 0000000211e69758
0x00007f6f639ef8c0:   508f6cdf00000001 00007f6f000000d3 

Instructions: (pc=0x00007f7072de5d45)
0x00007f7072de5d25:   0f 8e 52 1b 00 00 4f 8d 04 cc 41 ff cb 45 23 da
0x00007f7072de5d35:   4d 63 db 43 8b 4c 98 10 85 c9 0f 84 6a fe ff ff
0x00007f7072de5d45:   45 8b 5c cc 0c 49 8d 3c cc 45 3b da 0f 84 d9 00
0x00007f7072de5d55:   00 00 45 85 db 0f 8c 62 18 00 00 eb 12 44 8b d5 

Register to memory mapping:

RAX=0x000000006b7c8d5c is an unknown value
RBX=0x0000000000000023 is an unknown value
RCX=0x0000000057650087 is an unknown value
RDX=0x00000000a05edb42 is an unknown value
RSP=0x00007f6f639ef6d0 is pointing into the stack for thread: 0x00007f6e5811e000
RBP=0x000000026d914320 is an oop
java.util.concurrent.ConcurrentHashMap 
 - klass: 'java/util/concurrent/ConcurrentHashMap'
RSI=
[error occurred during error reporting (printing register info), id 0xb]

Stack: [0x00007f6f638f0000,0x00007f6f639f1000],  sp=0x00007f6f639ef6d0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 14006 C2 org.tron.core.db2.core.SnapshotImpl.get(Lorg/tron/core/db2/core/Snapshot;[B)[B (106 bytes) @ 0x00007f7072de5d45 [0x00007f7072de5b40+0x205]
J 13882 C2 org.tron.core.db2.core.SnapshotImpl.get([B)[B (7 bytes) @ 0x00007f7072e10718 [0x00007f7072e106e0+0x38]
J 13622 C2 org.tron.core.db2.core.Chainbase.getUnchecked([B)[B (11 bytes) @ 0x00007f7073459178 [0x00007f7073459120+0x58]
J 12784 C2 org.tron.core.capsule.TransactionCapsule.validateSignature(Lorg/tron/protos/Protocol$Transaction;[BLorg/tron/core/store/AccountStore;Lorg/tron/core/store/DynamicPropertiesStore;)Z (144 bytes) @ 0x00007f70723a2aec [0x00007f70723a2a20+0xcc]
J 12789 C2 org.tron.core.capsule.TransactionCapsule.validateSignature(Lorg/tron/core/store/AccountStore;Lorg/tron/core/store/DynamicPropertiesStore;)Z (90 bytes) @ 0x00007f7073483914 [0x00007f7073483520+0x3f4]
J 14549% C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f70737305a0 [0x00007f707372ff20+0x680]
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68bc3a]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0xe1a
V  [libjvm.so+0x6892f3]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x263
V  [libjvm.so+0x6898c7]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x57
V  [libjvm.so+0x6f72bc]  thread_entry(JavaThread*, Thread*)+0x6c
V  [libjvm.so+0xa88f8b]  JavaThread::thread_main_inner()+0xdb
V  [libjvm.so+0xa89291]  JavaThread::run()+0x2d1
V  [libjvm.so+0x9173e2]  java_start(Thread*)+0x102
C  [libpthread.so.0+0x8609]  start_thread+0xd9

[tomatoishealthy]

Could you provide the core dump file? As the log mentioned: Core dump written. Default location: /mnt/tron_node/core or core.35351

[weishirongzhen]

core.35351.log
uploaded

[tomatoishealthy]

We are diving into this problem, but no conclusion for now.
If this scene happened often? crashed every 8 or 10 minutes?
If it is, you can try to use another version of jdk8 to start the node, it is just a temporary suggestion.
We will inform you as soon as the crash reason founded.

BTW, according to the log, it seems that the FullNode met a NPE when querying the SnapshotImpl, you can also add some debug logs to find if there are bad entries in the SnapshotImpl or SnapshotRoot.

[weishirongzhen]

If this scene happened often? crashed every 8 or 10 minutes?
for now, yes.

[tomatoishealthy]

If this scene happened often? crashed every 8 or 10 minutes? for now, yes.

Do you try to use a lower version of JDK, like jdk8-202 (https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html)

[weishirongzhen]

ok, I will try jdk8-202 and see if it work

[weishirongzhen]

Using jdk8-202, running 3+ hours, looks like it's stable now

[tomatoishealthy]

Using jdk8-202, running 3+ hours, looks like it's stable now

Nice, please flush 666 for me.

[weishirongzhen]

🤙🤙🤙🤙🤙🤙🤙🤙🤙