Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[JDK19] JVMTI RedefineRunningMethods Segfaults #16212

Closed
babsingh opened this issue Oct 26, 2022 · 9 comments · Fixed by #16290
Closed

[JDK19] JVMTI RedefineRunningMethods Segfaults #16212

babsingh opened this issue Oct 26, 2022 · 9 comments · Fixed by #16290
Assignees
@LinHu2016
Labels
comp:gc comp:jvmti jdk19 project:loom Used to track Project Loom related work
Milestone
Java 19 0.37

Comments

@babsingh
Copy link
Contributor

babsingh commented Oct 26, 2022 •
edited
Loading

This failure is related to Project Loom and will only be seen in JDK19+.

Related: #16187.

Issue

As per the native stack, the segfault occurs while walking the continuation stack frames during concurrent GC.

Test CMD

make test TEST="jtreg:test/hotspot/jtreg/serviceability/jvmti/vthread/RedefineClasses/RedefineRunningMethods.java" JTREG="JAVA_OPTIONS=--enable-preview -Dvm.continuations=true;VERBOSE=all"

[GDB] Native Stack Trace

(gdb) where
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:57
#1  0x00007ff44e90e31d in omrdump_create (portLibrary=0x7ff44fb74380 <j9portLibrary>,
    filename=0x7ff429acb670 "/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/", dumpType=<optimized out>, userData=<optimized out>) at /root/openj9-openjdk-jdk/omr/port/unix/omrosdump.c:188
#2  0x00007ff44dc5e282 in doSystemDump (agent=0x7ff448033030,
    label=0x7ff429acb670 "/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/", context=0x7ff429acbb40) at /root/openj9-openjdk-jdk/openj9/runtime/rasdump/dmpagent.c:756
#3  0x00007ff44dc5a0e5 in protectedDumpFunction (portLibrary=portLibrary@entry=0x7ff44fb74380 <j9portLibrary>, userData=userData@entry=0x7ff429acb5d0)
    at /root/openj9-openjdk-jdk/openj9/runtime/rasdump/dmpagent.c:2852
#4  0x00007ff44e90fde3 in omrsig_protect (portLibrary=0x7ff44fb74380 <j9portLibrary>, fn=0x7ff44dc5a0d0 <protectedDumpFunction>, fn_arg=0x7ff429acb5d0, handler=0x7ff44dc5a0f0 <signalHandler>,
    handler_arg=0x0, flags=505, result=0x7ff429acb5c8) at /root/openj9-openjdk-jdk/omr/port/unix/omrsignal.c:425
#5  0x00007ff44dc5d6eb in runDumpFunction (agent=<optimized out>,
    label=0x7ff429acb670 "/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/", context=<optimized out>) at /root/openj9-openjdk-jdk/openj9/runtime/rasdump/dmpagent.c:2830
#6  0x00007ff44dc5d87f in runDumpAgent (vm=vm@entry=0x7ff448019b60, agent=agent@entry=0x7ff448033030, context=context@entry=0x7ff429acbb40, state=state@entry=0x7ff429acbb30,
    detail=detail@entry=0x7ff429acbcc0 "", timeNow=timeNow@entry=1666824198404) at /root/openj9-openjdk-jdk/openj9/runtime/rasdump/dmpagent.c:2760
#7  0x00007ff44dc76386 in triggerDumpAgents (vm=0x7ff448019b60, self=0x1cfd00, eventFlags=8192, eventData=<optimized out>) at /root/openj9-openjdk-jdk/openj9/runtime/rasdump/trigger.c:1036
#8  0x00007ff44f35aab2 in generateDiagnosticFiles (portLibrary=portLibrary@entry=0x7ff44fb74380 <j9portLibrary>, userData=userData@entry=0x7ff429acc1a0)
    at /root/openj9-openjdk-jdk/openj9/runtime/vm/gphandle.c:1158
#9  0x00007ff44e90fde3 in omrsig_protect (portLibrary=0x7ff44fb74380 <j9portLibrary>, fn=0x7ff44f35a9d0 <generateDiagnosticFiles>, fn_arg=0x7ff429acc1a0, handler=0x7ff44f35a090 <recursiveCrashHandler>,
    handler_arg=0x7ff429acc170, flags=505, result=0x7ff429acc168) at /root/openj9-openjdk-jdk/omr/port/unix/omrsignal.c:425
#10 0x00007ff44f35acc5 in vmSignalHandler (portLibrary=0x7ff44fb74380 <j9portLibrary>, gpType=24, gpInfo=<optimized out>, userData=<optimized out>)
    at /root/openj9-openjdk-jdk/openj9/runtime/vm/gphandle.c:833
#11 0x00007ff44e90f2aa in mainSynchSignalHandler (signal=11, sigInfo=0x7ff429acd430, contextInfo=0x7ff429acd300) at /root/openj9-openjdk-jdk/omr/port/unix/omrsignal.c:1066
#12 <signal handler called>
#13 0x0000000000000000 in ?? ()
#14 0x00007ff44f39d6d3 in walkStackFrames (currentThread=<optimized out>, walkState=0x7ff429aceb50) at /root/openj9-openjdk-jdk/openj9/runtime/vm/swalk.c:345
#15 0x00007ff44f3b3027 in walkContinuationStackFrames (currentThread=0x1cfd00, continuation=0x7ff3e4002d80, walkState=0x7ff429aceb50) at /root/openj9-openjdk-jdk/openj9/runtime/vm/ContinuationHelpers.cpp:239
#16 0x00007ff44d37c1c6 in VM_VMHelpersCompressed::walkContinuationStackFramesWrapper (walkState=0x7ff429aceb50, continuationObject=0xc193b910, vmThread=0x1cfd00)
    at /root/openj9-openjdk-jdk/openj9/runtime/oti/VMHelpers.hpp:2056
#17 GC_VMThreadStackSlotIterator::scanSlots (vmThread=0x1cfd00, continuationObjectPtr=continuationObjectPtr@entry=0xc193b910, userData=userData@entry=0x7ff429acedb0,
    oSlotIterator=oSlotIterator@entry=0x7ff44d4dbca0 <stackSlotIteratorForMarkingDelegate(J9JavaVM*, J9Object**, void*, J9StackWalkState*, void const*)>, includeStackFrameClassReferences=<optimized out>,
    trackVisibleFrameDepth=trackVisibleFrameDepth@entry=false) at /root/openj9-openjdk-jdk/openj9/runtime/gc_structs/VMThreadStackSlotIterator.cpp:145
#18 0x00007ff44d4dbd38 in MM_MarkingDelegate::scanContinuationObject (this=0x7ff448042fb0, env=env@entry=0x7ff4484507c8, objectPtr=objectPtr@entry=0xc193b910)
    at /root/openj9-openjdk-jdk/openj9/runtime/gc_glue_java/MarkingDelegate.cpp:275
#19 0x00007ff44d5091f2 in MM_MarkingDelegate::getObjectScanner (sizeToDo=0x7ff429acee30, reason=SCAN_REASON_DIRTY_CARD, scannerSpace=0x7ff429acee80, objectPtr=0xc193b910, env=0x7ff4484507c8,
    this=0x7ff448042fb0) at /root/openj9-openjdk-jdk/openj9/runtime/gc_glue_java/MarkingDelegate.hpp:143
#20 MM_MarkingScheme::scanObject (sizeToDo=<optimized out>, reason=SCAN_REASON_DIRTY_CARD, objectPtr=0xc193b910, env=0x7ff4484507c8, this=0x7ff448042f90)
    at /root/openj9-openjdk-jdk/omr/gc/base/MarkingScheme.hpp:237
#21 MM_ConcurrentCardTable::cleanSingleCard (this=this@entry=0x7ff4480445c0, env=env@entry=0x7ff4484507c8, card=card@entry=0x7ff44c506d5c "\001", bytesToClean=<optimized out>,
    totalBytesCleaned=totalBytesCleaned@entry=0x7ff429aceff8) at /root/openj9-openjdk-jdk/omr/gc/base/standard/ConcurrentCardTable.cpp:987
#22 0x00007ff44d50ac85 in MM_ConcurrentCardTable::cleanCards (this=0x7ff4480445c0, env=env@entry=0x7ff4484507c8, isMutator=isMutator@entry=true, sizeToDo=sizeToDo@entry=1638400,
    sizeDone=sizeDone@entry=0x7ff429acf068, threadAtSafePoint=threadAtSafePoint@entry=true) at /root/openj9-openjdk-jdk/omr/gc/base/standard/ConcurrentCardTable.cpp:884
#23 0x00007ff44d4969fb in MM_ConcurrentGCIncrementalUpdate::cleanCards (threadAtSafePoint=true, sizeDone=0x7ff429acf068, sizeToDo=1638400, isMutator=true, env=0x7ff4484507c8, this=0x7ff448042a10)
    at /root/openj9-openjdk-jdk/omr/gc/base/standard/ConcurrentGCIncrementalUpdate.cpp:1173
#24 MM_ConcurrentGCIncrementalUpdate::doConcurrentTrace (this=0x7ff448042a10, env=0x7ff4484507c8, allocDescription=<optimized out>, sizeToTrace=1638400, subspace=0x7ff44808f630, threadAtSafePoint=true)
    at /root/openj9-openjdk-jdk/omr/gc/base/standard/ConcurrentGCIncrementalUpdate.cpp:985
#25 0x00007ff44d510769 in MM_ConcurrentGC::concurrentMark (this=this@entry=0x7ff448042a10, env=env@entry=0x7ff4484507c8, subspace=subspace@entry=0x7ff44808f630,
    allocDescription=allocDescription@entry=0x7ff429acf268) at /root/openj9-openjdk-jdk/omr/gc/base/standard/ConcurrentGC.cpp:1469
#26 0x00007ff44d510fd8 in MM_ConcurrentGC::payAllocationTax (this=0x7ff448042a10, env=0x7ff4484507c8, subspace=0x7ff44808f630, baseSubSpace=0x7ff448084310, allocDescription=0x7ff429acf268)
    at /root/openj9-openjdk-jdk/omr/gc/base/standard/ConcurrentGC.cpp:1408
#27 0x00007ff44d478734 in MM_AllocateDescription::payAllocationTax (env=0x7ff4484507c8, this=0x7ff429acf268) at /root/openj9-openjdk-jdk/omr/gc/base/AllocateDescription.hpp:175
--Type <RET> for more, q to quit, c to continue without paging--
#28 MM_AllocateInitialization::allocateAndInitializeObject (omrVMThread=<optimized out>, this=0x7ff429acf250) at /root/openj9-openjdk-jdk/omr/gc/base/AllocateInitialization.hpp:230
#29 OMR_GC_AllocateObject (omrVMThread=<optimized out>, allocator=allocator@entry=0x7ff429acf250) at /root/openj9-openjdk-jdk/omr/gc/startup/omrgcalloc.cpp:39
#30 0x00007ff44d3833c7 in J9AllocateIndexableObject (vmThread=0x1cfd00, clazz=0x25600, numberOfIndexedFields=32, allocateFlags=1) at /root/openj9-openjdk-jdk/openj9/runtime/gc_modron_startup/mgcalloc.cpp:578
#31 0x00007ff44f3c3db9 in VM_BytecodeInterpreterCompressed::newarray (_pc=<optimized out>, _sp=<optimized out>, this=<optimized out>)
    at /root/openj9-openjdk-jdk/openj9/runtime/vm/BytecodeInterpreter.hpp:8121
#32 VM_BytecodeInterpreterCompressed::run (this=0x7ff429acf8f0, vmThread=0x7ff448019b60) at /root/openj9-openjdk-jdk/openj9/runtime/vm/BytecodeInterpreter.hpp:11454
#33 0x00007ff44f3b315d in bytecodeLoopCompressed (currentThread=<optimized out>) at /root/openj9-openjdk-jdk/openj9/runtime/vm/BytecodeInterpreter.inc:112
#34 0x00007ff44f45efc2 in c_cInterpreter () at /root/openj9-openjdk-jdk/build/linux-x86_64-server-release/vm/runtime/vm/xcinterp.s:158
#35 0x00007ff44f33847a in runJavaThread (currentThread=0x7ff429acfa20, currentThread@entry=0x1cfd00) at /root/openj9-openjdk-jdk/openj9/runtime/vm/callin.cpp:682
#36 0x00007ff44f3b1e2d in javaProtectedThreadProc (portLibrary=portLibrary@entry=0x7ff44fb74380 <j9portLibrary>, entryarg=entryarg@entry=0x1cfd00)
    at /root/openj9-openjdk-jdk/openj9/runtime/vm/vmthread.cpp:2093
#37 0x00007ff44e90fde3 in omrsig_protect (portLibrary=0x7ff44fb74380 <j9portLibrary>, fn=0x7ff44f3b1d70 <javaProtectedThreadProc(J9PortLibrary*, void*)>, fn_arg=0x1cfd00,
    handler=0x7ff44f35af10 <structuredSignalHandler>, handler_arg=0x1cfd00, flags=506, result=0x7ff429acfdc8) at /root/openj9-openjdk-jdk/omr/port/unix/omrsignal.c:425
#38 0x00007ff44f3adeaa in javaThreadProc (entryarg=0x7ff448019b60) at /root/openj9-openjdk-jdk/openj9/runtime/vm/vmthread.cpp:372
#39 0x00007ff44f10f302 in thread_wrapper (arg=0x7ff4483fd6c8) at /root/openj9-openjdk-jdk/omr/thread/common/omrthread.c:1724
#40 0x00007ff44fd916db in start_thread (arg=0x7ff429ad2700) at pthread_create.c:463
#41 0x00007ff4504eb61f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Test Output

STDOUT:
test_redef
test_redef
infinite_emcp called
infinite called
test_redef
infinite_emcp called
STDERR:
JVMJ9VM007W Command-line option unrecognised: -Xlog:redefine+class+iklass+add=trace,redefine+class+iklass+purge=trace,class+loader+data=debug,safepoint+cleanup,gc+phases=debug:rt.log
Unhandled exception
Type=Segmentation error vmState=0x00020016
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
Handler1=00007FA0337E2F10 Handler2=00007FA032D97080 InaccessibleAddress=00007F9FDDD399A9
RDI=00007FA012148730 RSI=0000000000000000 RAX=00007FA012148730 RBX=00007FA03440ED74
RCX=00007FA03440ED74 RDX=0000000000000015 R8=0000000000000055 R9=FFFFFFFFFFFEA209
R10=00007FA012148B80 R11=00007FA012148730 R12=FFFFFFFFCBBF1279 R13=0000000000000015
R14=0000000000000040 R15=00007FA0121486F0
RIP=00007FA03390130C GS=0000 FS=0000 RSP=00007FA012148660
EFlags=0000000000010206 CS=0033 RBP=0000000000000004 ERR=0000000000000006
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=00007F9FDDD399A9
xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm1 396a6e65706f2f6b (f: 1886334848.000000, d: 4.072377e-32)
xmm2 000000003f19999a (f: 1058642304.000000, d: 5.230388e-315)
xmm3 0000000047d80000 (f: 1205338112.000000, d: 5.955162e-315)
xmm4 bfcb31d8a68224e9 (f: 2793547008.000000, d: -2.124587e-01)
xmm5 0065007200680054 (f: 6815828.000000, d: 9.346084e-307)
xmm6 002f006100760061 (f: 7733345.000000, d: 8.622573e-308)
xmm7 00730065006c0075 (f: 7078005.000000, d: 1.691193e-306)
xmm8 6a2e6e75732e6d6f (f: 1932422528.000000, d: 2.981605e+203)
xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10 0000002000000020 (f: 32.000000, d: 6.790387e-313)
xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/images/jdk/lib/default/libj9vm29.so
Module_base_address=00007FA0337A4000
Target=2_90_20221017_000000 (Linux 4.15.0-193-generic)
CPU=amd64 (8 logical CPUs) (0x3ea756000 RAM)
----------- Stack Backtrace -----------
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007FA0337E2F50 Handler2=00007FA032D97080 InaccessibleAddress=0000000000000001
RDI=00000000004F6150 RSI=00000000C18C30A0 RAX=0000000024400006 RBX=0000000000000000
RCX=0000000000400000 RDX=0000000000000001 R8=00000000004F6150 R9=0000000000000000
R10=00007FA031ED1008 R11=0000000000000000 R12=00007FA012593650 R13=0000000004400006
R14=0000000000000000 R15=0000000000000000
RIP=00007FA033825E0C GS=0000 FS=0000 RSP=00007FA012592B00
EFlags=0000000000010202 CS=0033 RBP=0000000000191B00 ERR=0000000000000004
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000001
xmm0 0000003000000020 (f: 32.000000, d: 1.018558e-312)
xmm1 00000000000000ef (f: 239.000000, d: 1.180817e-321)
xmm2 ffffffffffffffff (f: 4294967296.000000, d: -nan)
xmm3 bfdffa3e3a9bcbb5 (f: 983288768.000000, d: -4.996486e-01)
xmm4 bfc6574f0ac07758 (f: 180385632.000000, d: -1.745395e-01)
xmm5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm6 ffffffffffffffff (f: 4294967296.000000, d: -nan)
xmm7 0000000400000004 (f: 4.000000, d: 8.487983e-314)
xmm8 00007fa008000800 (f: 134219776.000000, d: 6.932991e-310)
xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10 0000002000000020 (f: 32.000000, d: 6.790387e-313)
xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/images/jdk/lib/default/libj9vm29.so
Module_base_address=00007FA0337A4000
Target=2_90_20221017_000000 (Linux 4.15.0-193-generic)
CPU=amd64 (8 logical CPUs) (0x3ea756000 RAM)
----------- Stack Backtrace -----------
j9stackmap_StackBitsForPC+0x12c (0x00007FA03390130C [libj9vm29.so+0x15d30c])
walkBytecodeFrameSlots+0x2bf (0x00007FA03382533F [libj9vm29.so+0x8133f])
walkStackFrames+0xc02 (0x00007FA033826222 [libj9vm29.so+0x82222])
walkContinuationStackFrames+0x77 (0x00007FA03383B027 [libj9vm29.so+0x97027])
_ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadP8J9ObjectPvPFvP8J9JavaVMPS3_S4_P16J9StackWalkStatePKvEbb+0x46 (0x00007FA0318041C6 [libj9gc29.so+0x441c6])
_ZN18MM_MarkingDelegate22scanContinuationObjectEP18MM_EnvironmentBaseP8J9Object+0x78 (0x00007FA031963D38 [libj9gc29.so+0x1a3d38])
_ZN22MM_ConcurrentCardTable15cleanSingleCardEP18MM_EnvironmentBasePhmPm+0x1d2 (0x00007FA0319911F2 [libj9gc29.so+0x1d11f2])
_ZN22MM_ConcurrentCardTable10cleanCardsEP18MM_EnvironmentBasebmPmb+0x1c5 (0x00007FA031992C85 [libj9gc29.so+0x1d2c85])
_ZN32MM_ConcurrentGCIncrementalUpdate17doConcurrentTraceEP18MM_EnvironmentBaseP22MM_AllocateDescriptionmP17MM_MemorySubSpaceb+0x26b (0x00007FA03191E9FB [libj9gc29.so+0x15e9fb])
_ZN15MM_ConcurrentGC14concurrentMarkEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription+0x119 (0x00007FA031998769 [libj9gc29.so+0x1d8769])
_ZN15MM_ConcurrentGC16payAllocationTaxEP18MM_EnvironmentBaseP17MM_MemorySubSpaceS3_P22MM_AllocateDescription+0x68 (0x00007FA031998FD8 [libj9gc29.so+0x1d8fd8])
_Z21OMR_GC_AllocateObjectP12OMR_VMThreadP25MM_AllocateInitialization+0x194 (0x00007FA031900734 [libj9gc29.so+0x140734])
J9AllocateIndexableObject+0x7f7 (0x00007FA03180B3C7 [libj9gc29.so+0x4b3c7])
_ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x10c49 (0x00007FA03384BDB9 [libj9vm29.so+0xa7db9])
bytecodeLoopCompressed+0xad (0x00007FA03383B15D [libj9vm29.so+0x9715d])
 (0x00007FA0338E6FC2 [libj9vm29.so+0x142fc2])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/10/26 15:48:00 - please wait.
JVMDUMP032I JVM requested System dump using '/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/core.20221026.154800.28773.0001.dmp' in response to an event
walkStackFrames+0x7ec (0x00007FA033825E0C [libj9vm29.so+0x81e0c])
walkContinuationStackFrames+0x77 (0x00007FA03383B027 [libj9vm29.so+0x97027])
_ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadP8J9ObjectPvPFvP8J9JavaVMPS3_S4_P16J9StackWalkStatePKvEbb+0x46 (0x00007FA0318041C6 [libj9gc29.so+0x441c6])
_ZN18MM_MarkingDelegate22scanContinuationObjectEP18MM_EnvironmentBaseP8J9Object+0x78 (0x00007FA031963D38 [libj9gc29.so+0x1a3d38])
_ZN22MM_ConcurrentCardTable15cleanSingleCardEP18MM_EnvironmentBasePhmPm+0x1d2 (0x00007FA0319911F2 [libj9gc29.so+0x1d11f2])
_ZN22MM_ConcurrentCardTable10cleanCardsEP18MM_EnvironmentBasebmPmb+0x1c5 (0x00007FA031992C85 [libj9gc29.so+0x1d2c85])
_ZN32MM_ConcurrentGCIncrementalUpdate23conHelperDoWorkInternalEP18MM_EnvironmentBasePN15MM_ConcurrentGC16ConHelperRequestEP14MM_SpinLimiterPm+0x1ab (0x00007FA03191D21B [libj9gc29.so+0x15d21b])
_ZN15MM_ConcurrentGC19conHelperEntryPointEP12OMR_VMThreadm+0x345 (0x00007FA031996975 [libj9gc29.so+0x1d6975])
con_helper_thread_proc2+0x6b (0x00007FA031996D6B [libj9gc29.so+0x1d6d6b])
omrsig_protect+0x1e3 (0x00007FA032D97DE3 [libj9prt29.so+0x29de3])
con_helper_thread_proc+0x42 (0x00007FA031994A52 [libj9gc29.so+0x1d4a52])
thread_wrapper+0x162 (0x00007FA033597302 [libj9thr29.so+0xe302])
start_thread+0xdb (0x00007FA0383F16DB [libpthread.so.0+0x76db])
clone+0x3f (0x00007FA038B4B61F [libc.so.6+0x12161f])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/10/26 15:48:00 - please wait.
JVMDUMP010I System dump written to /root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/core.20221026.154800.28773.0001.dmp
JVMDUMP032I JVM requested Java dump using '/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/javacore.20221026.154800.28773.0003.txt' in response to an event
Unhandled exception
Type=Segmentation error vmState=0x00020016
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000080
Handler1=00007FA0337E2F10 Handler2=00007FA032D97080 InaccessibleAddress=0000000000000000
RDI=0400079A1D003EB6 RSI=00007FA013C693F1 RAX=0000000000000000 RBX=0000000000000000
RCX=1C0000EF030004A0 RDX=FBFFF865E32257BA R8=0000000000229638 R9=0000000000000000
R10=0000000000000000 R11=0000000000000000 R12=00007FA011EC4B90 R13=0000000004400006
R14=0000000000000000 R15=00007FA013C693F1
RIP=00007FA03382612A GS=0000 FS=0000 RSP=00007FA011EC4040
EFlags=0000000000010293 CS=0033 RBP=00000000001E8A00 ERR=0000000000000000
TRAPNO=000000000000000D OLDMASK=0000000000000000 CR2=0000000000000000
xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm1 0000000000000011 (f: 17.000000, d: 8.399116e-323)
xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm3 bfe03f249c69afcf (f: 2624172032.000000, d: -5.077079e-01)
xmm4 3fbc5e53aa362eb4 (f: 2855677696.000000, d: 1.108143e-01)
xmm5 7465677261745f68 (f: 1635016576.000000, d: 4.903917e+252)
xmm6 6f6f745f6c616e72 (f: 1818324608.000000, d: 5.961158e+228)
xmm7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm8 00007f9fe0000800 (f: 3758098432.000000, d: 6.932958e-310)
xmm9 ffff000000ffffff (f: 16777215.000000, d: -nan)
xmm10 2020000000202020 (f: 2105376.000000, d: 5.966673e-154)
xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/images/jdk/lib/default/libj9vm29.so
Module_base_address=00007FA0337A4000
Target=2_90_20221017_000000 (Linux 4.15.0-193-generic)
CPU=amd64 (8 logical CPUs) (0x3ea756000 RAM)
----------- Stack Backtrace -----------
JVMDUMP010I Java dump written to /root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/javacore.20221026.154800.28773.0003.txt
JVMDUMP032I JVM requested System dump using '/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/core.20221026.154800.28773.0002.dmp' in response to an event
walkStackFrames+0xb0a (0x00007FA03382612A [libj9vm29.so+0x8212a])
 (0x00007FA03383B027 [libj9vm29.so+0x97027])
_ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadP8J9ObjectPvPFvP8J9JavaVMPS3_S4_P16J9StackWalkStatePKvEbb+0x46 (0x00007FA0318041C6 [libj9gc29.so+0x441c6])
_ZN18MM_MarkingDelegate22scanContinuationObjectEP18MM_EnvironmentBaseP8J9Object+0x78 (0x00007FA031963D38 [libj9gc29.so+0x1a3d38])
_ZN22MM_ConcurrentCardTable15cleanSingleCardEP18MM_EnvironmentBasePhmPm+0x1d2 (0x00007FA0319911F2 [libj9gc29.so+0x1d11f2])
_ZN22MM_ConcurrentCardTable10cleanCardsEP18MM_EnvironmentBasebmPmb+0x1c5 (0x00007FA031992C85 [libj9gc29.so+0x1d2c85])
_ZN32MM_ConcurrentGCIncrementalUpdate17doConcurrentTraceEP18MM_EnvironmentBaseP22MM_AllocateDescriptionmP17MM_MemorySubSpaceb+0x26b (0x00007FA03191E9FB [libj9gc29.so+0x15e9fb])
_ZN15MM_ConcurrentGC14concurrentMarkEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription+0x119 (0x00007FA031998769 [libj9gc29.so+0x1d8769])
_ZN15MM_ConcurrentGC16payAllocationTaxEP18MM_EnvironmentBaseP17MM_MemorySubSpaceS3_P22MM_AllocateDescription+0x68 (0x00007FA031998FD8 [libj9gc29.so+0x1d8fd8])
_Z21OMR_GC_AllocateObjectP12OMR_VMThreadP25MM_AllocateInitialization+0x194 (0x00007FA031900734 [libj9gc29.so+0x140734])
J9AllocateObject+0x226 (0x00007FA031809EE6 [libj9gc29.so+0x49ee6])
_ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0x164c6 (0x00007FA033851636 [libj9vm29.so+0xad636])
bytecodeLoopCompressed+0xad (0x00007FA03383B15D [libj9vm29.so+0x9715d])
 (0x00007FA0338E6FC2 [libj9vm29.so+0x142fc2])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/10/26 15:48:01 - please wait.
JVMDUMP010I System dump written to /root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/core.20221026.154800.28773.0002.dmp
JVMDUMP032I JVM requested Java dump using '/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/javacore.20221026.154800.28773.0006.txt' in response to an event
JVMDUMP032I JVM requested Snap dump using '/root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/Snap.20221026.154800.28773.0004.trc' in response to an event
JVMDUMP010I Snap dump written to /root/openj9-openjdk-jdk/build/linux-x86_64-server-release/test-support/jtreg_test_hotspot_jtreg_serviceability_jvmti_vthread_RedefineClasses_RedefineRunningMethods_java/scratch/0/Snap.20221026.154800.28773.0004.trc
JVMDUMP013I Processed dump event "gpf", detail "".
@babsingh babsingh added this to the Java 19 milestone Oct 26, 2022
@babsingh
Copy link
Contributor Author

fyi @LinHu2016 @amicic @dmitripivkine

@dmitripivkine
Copy link
Contributor

@babsingh Do you have preserved system core I can look at?

@babsingh
Copy link
Contributor Author

@dmitripivkine @LinHu2016 Core files: https://ibm.box.com/s/qxmhx0jap9x1of6zryq1a4usl59n6t7b.

@LinHu2016
Copy link
Contributor

LinHu2016 commented Oct 27, 2022 •
edited
Loading

@babsingh
the below is the obeservations for the issue,
1, I can reproduce 100% same crash locally
2, related continuation Object and J9VMContinuation structure looks normal
3, related J9StackWalkState looks normal
4, crash at #14 0x00007fd2268ee6a3 in walkStackFrames (currentThread=, walkState=0x7fd20103db50) at /home/linhu_j9build/openj9-openjdk-jdk19/openj9/runtime/vm/swalk.c:345
if (jitWalkStackFrames(walkState) != J9_STACKWALK_KEEP_ITERATING) {
5, J9JavaVM->jitWalkStackFrames is NULL, which cause the crash
6, running test with -Xint mode
7, the continuation is unmounted.
not sure how -Xint mode could reach swalk.c:line345

@LinHu2016
Copy link
Contributor 

!J9StackWalkState 0x7f5f648f3b50
J9StackWalkState at 0x7f5f648f3b50 {
  Fields for J9StackWalkState:
	0x0: struct J9StackWalkState* previous = !j9stackwalkstate 0x00007F5F88628700
	0x8: struct J9VMThread* walkThread = !j9vmthread 0x00007F5F648F30E0
	0x10: struct J9JavaVM* javaVM = !j9javavm 0x00007F5F8401AD40
	0x18: UDATA flags = 0x0000000024400006 (608174086)
	0x20: UDATA* bp = !j9x 0x0000000000216370
	0x28: UDATA* unwindSP = !j9x 0x0000000000216350
	0x30: U8* pc = !j9x 0x0000000000000003
	0x38: U8* nextPC = !j9x 0x00007F5F6643F2DD // "�."
	0x40: UDATA* sp = !j9x 0x0000000000216350
	0x48: UDATA* arg0EA = !j9x 0x0000000000216378
	0x50: struct J9Method* literals = !j9method 0x0000000000000000
	0x58: UDATA* walkSP = !j9x 0x0000000000216350
	0x60: UDATA argCount = 0x0000000000000001 (1)
	0x68: struct J9ConstantPool* constantPool = !j9constantpool 0x0000000000076E20 (flags = 0x0)
	0x70: struct J9Method* method = !j9method 0x0000000000077540 // jdk/internal/vm/Continuation.run()V
	0x78: struct J9JITExceptionTable* jitInfo = !j9jitexceptiontable 0x0000000000000000
	0x80: UDATA frameFlags = 0x00007F5F6643F2EF (140047714349807)
	0x88: UDATA resolveFrameFlags = 0x0000000000000000 (0)
	0x90: UDATA skipCount = 0x0000000000000000 (0)
	0x98: UDATA maxFrames = 0x000000100005EC0A (68719864842)
	0xa0: void* userData1 = !j9x 0x00007F5F882F7CA0
	0xa8: void* userData2 = !j9x 0x00007F5F8401AD40
	0xb0: void* userData3 = !j9x 0x00007F5F648F3DB0
	0xb8: void* userData4 = !j9x 0x0000000000000000
	0xc0: UDATA framesWalked = 0x0000000000000001 (1)
	0xc8: void* frameWalkFunction = !j9x 0x00007F5F648F3D30
	0xd0: void* objectSlotWalkFunction = !j9x 0x00007F5F88198050
	0xd8: void* returnAddressWalkFunction = !j9x 0x7C2D6CB9CBE73600
	0xe0: UDATA* cache = !j9x 0x0000000000000000
	0xe8: void* restartPoint = !j9x 0x00007F5F88628700
	0xf0: void* restartException = !j9x 0x00007F5F648F3DF8
	0xf8: void* inlinerMap = !j9x 0x0000000000000000
	0x100: UDATA inlineDepth = 0x0000000000000000 (0)
	0x108: UDATA* cacheCursor = !j9x 0x00007F5F88628700
	0x110: struct J9JITDecompilationInfo* decompilationRecord = !j9jitdecompilationinfo 0x0000000000000000
	0x118: struct J9WalkStackFramesAndSlotsStorage registerEAs = !j9walkstackframesandslotsstorage 0x00007F5F648F3C68
	0x198: struct J9VMEntryLocalStorage* walkedEntryLocalStorage = !j9vmentrylocalstorage 0x00007F5F648F3080
	0x1a0: struct J9I2JState* i2jState = !j9i2jstate 0x00007F5F648F3090
	0x1a8: struct J9JITDecompilationInfo* decompilationStack = !j9jitdecompilationinfo 0x0000000000000000
	0x1b0: void** pcAddress = !j9x 0x00007F5F648F3108
	0x1b8: UDATA outgoingArgCount = 0x0000000000000000 (0)
	0x1c0: U8* objectSlotBitVector = !j9x 0x0000000000000000
	0x1c8: UDATA elsBitVector = 0x0100000000000004 (72057594037927940)
	0x1d0: void* savedObjectSlotWalkFunction = !j9x 0x0000000000009002
	0x1d8: IDATA bytecodePCOffset = 0xFFFFFFFFFFFFFFFF (-1)
	0x1e0: void* dropToCurrentFrame = !j9x 0x00007F5F8A1BBB80
	0x1e8: UDATA* j2iFrame = !j9x 0x0000000000000000
	0x1f0: UDATA previousFrameFlags = 0x00007F5F6643F2EF (140047714349807)
	0x1f8: IDATA slotIndex = 0x0000000000000001 (1)
	0x200: UDATA slotType = 0x0000000000000001 (1)
	0x208: struct J9VMThread* currentThread = !j9vmthread 0x00000000001CFD00
	0x210: void* linearSlotWalker = !j9x 0x0000000000000000
	0x218: void* inlinedCallSite = !j9x 0x0000000000000000
	0x220: void* stackMap = !j9x 0x0000000000000000
	0x228: void* inlineMap = !j9x 0x0000000000000000
}

@dmitripivkine
Copy link
Contributor

dmitripivkine commented Oct 28, 2022 •
edited
Loading

frameFlags looks corrupted, frameFlags = 0x00007F5F6643F2EF is address close to nextPC = !j9x 0x00007F5F6643F2DD. Also previousFrameFlags = 0x00007F5F6643F2EF

@LinHu2016
Copy link
Contributor

@dmitripivkine mentioned
1, if walkState->frameFlags has any of J9_STACK_FLAGS_JIT_TRANSITION_TO_INTERPRETER_MASK bits (0x70080000), the code could reach line if (jitWalkStackFrames(walkState) != J9_STACKWALK_KEEP_ITERATING) {
0x80: UDATA frameFlags = 0x00007F5F6643F2EF (140047714349807)
2, walkState->frameFlags looks like memory address, not flags( close to nextPC = !j9x 0x00007F5F6643F2DD)

@dmitripivkine
Copy link
Contributor

All these crashes occur when GC is scanning Continuation object and corresponded virtual thread java stack concurrently (at any random time). It might be some reason (race?) why stack walk code does not work

@babsingh babsingh mentioned this issue Oct 31, 2022
Closed
@babsingh babsingh changed the title [JDK19] JVMTI Serviceability RedefineRunningMethods Segfaults [JDK19] JVMTI RedefineRunningMethods Segfaults Oct 31, 2022
@pshipton pshipton added the jdk19 label Nov 1, 2022
@tajila tajila added the project:loom Used to track Project Loom related work label Nov 1, 2022
@LinHu2016
Copy link
Contributor

confirmed this is a race case between scanning java stack of continuation Object and mounting/unmounting the same continuation Object. working on a fix.

@LinHu2016 LinHu2016 mentioned this issue Nov 3, 2022
Closed
@tajila tajila removed the comp:vm label Nov 3, 2022
@LinHu2016 LinHu2016 mentioned this issue Nov 8, 2022
Merged
@babsingh babsingh mentioned this issue Nov 10, 2022
Closed
babsingh added a commit to babsingh/aqa-tests that referenced this issue Nov 21, 2022
@babsingh
Re-enable tests
d6da8bb 
eclipse-openj9/openj9#16212 was fixed by
1. eclipse-openj9/openj9#16290
2. eclipse-openj9/openj9#16293

eclipse-openj9/openj9#16275 is a duplicate of
eclipse-openj9/openj9#16212.

eclipse-openj9/openj9#16229 was fixed by eclipse-openj9/openj9#16323.

FramePop/framepop02 fails with another issue, which is reported in
eclipse-openj9/openj9#16346.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
babsingh added a commit to babsingh/aqa-tests that referenced this issue Nov 21, 2022
@babsingh
Re-enable tests
486d52c 
eclipse-openj9/openj9#16212 was fixed by
1. eclipse-openj9/openj9#16290; and
2. eclipse-openj9/openj9#16293.

eclipse-openj9/openj9#16275 is a duplicate of
eclipse-openj9/openj9#16212.

eclipse-openj9/openj9#16229 was fixed by eclipse-openj9/openj9#16323.

FramePop/framepop02 fails with another issue, which is reported in
eclipse-openj9/openj9#16346.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
@babsingh babsingh mentioned this issue Nov 21, 2022
Merged
Mesbah-Alam pushed a commit to adoptium/aqa-tests that referenced this issue Nov 22, 2022
@babsingh
Re-enable tests (#4162)
d295e74 
eclipse-openj9/openj9#16212 was fixed by
1. eclipse-openj9/openj9#16290; and
2. eclipse-openj9/openj9#16293.

eclipse-openj9/openj9#16275 is a duplicate of
eclipse-openj9/openj9#16212.

eclipse-openj9/openj9#16229 was fixed by eclipse-openj9/openj9#16323.

FramePop/framepop02 fails with another issue, which is reported in
eclipse-openj9/openj9#16346.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LinHu2016 LinHu2016

Labels
comp:gc comp:jvmti jdk19 project:loom Used to track Project Loom related work
Projects
None yet
Milestone
Java 19 0.37
Development

Successfully merging a pull request may close this issue.

Synchronization between continuation mounting and concurrent scanning LinHu2016/openj9
5 participants
@babsingh @tajila @LinHu2016 @pshipton @dmitripivkine