cmdLineTester_criu_nonPortableRestore MethodTypeDeadlockTest intermittent hanging

[ThanHenderson]

openjdk version "11.0.22-internal" 2024-01-16
OpenJDK Runtime Environment (build 11.0.22-internal+0-adhoc..BuildJDK11x86-64linuxPersonal)
Eclipse OpenJ9 VM (build master-d15ef63, JRE 11 Linux amd64-64-Bit Compressed References 20240125_6061 (JIT enabled, AOT enabled)
OpenJ9   - d15ef63
OMR      - 269a3d2
JCL      - d37a5ac based on jdk-11.0.22+7)

While looking into #18800 I've noticed that occasionally (difficult to quantify exactly but it is rare) the MethodTypeDeadlockTest hangs indefinitely and produces a javacore file (but no core file) with the following call stack.

1XMCURTHDINFO  Current thread
3XMTHREADINFO      "main" J9VMThread:0x000000000001AA00, omrthread_t:0x00007FF33401C860, java/lang/Thread:0x00000007060F5388, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x000000070610B968)
3XMTHREADINFO1            (native thread ID:0x38FB0, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00041020)
3XMTHREADINFO2            (native stack address range from:0x00007FF339299000, to:0x00007FF3392DA000, size:0x41000)
3XMCPUTIME               CPU usage total: 0.067891052 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=12912 (0x3270)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ClassLoader.getClassLoadingLock(ClassLoader.java:1253(Compiled Code))
4XESTACKTRACE                at jdk/internal/loader/BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:776(Compiled Code))
4XESTACKTRACE                at jdk/internal/loader/BuiltinClassLoader.loadClass(BuiltinClassLoader.java:764(Compiled Code))
4XESTACKTRACE                at jdk/internal/loader/ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
4XESTACKTRACE                at java/lang/ClassLoader.loadClass(ClassLoader.java:1075(Compiled Code))
4XESTACKTRACE                at java/lang/Class.forNameImpl(Native Method)
4XESTACKTRACE                at java/lang/Class.forName(Class.java:418)
4XESTACKTRACE                at openj9/internal/criu/security/CRIUConfigurator.setCRIURestoreMode(CRIUConfigurator.java:141)
4XESTACKTRACE                at openj9/internal/criu/SecurityProviders.lambda$registerRestoreSecurityProviders$1(SecurityProviders.java:64)
4XESTACKTRACE                at openj9/internal/criu/SecurityProviders$$Lambda$23/0x0000000000000000.run(Bytecode PC:0)
4XESTACKTRACE                at openj9/internal/criu/J9InternalCheckpointHookAPI$J9InternalCheckpointHook.runHook(J9InternalCheckpointHookAPI.java:143)
4XESTACKTRACE                at openj9/internal/criu/J9InternalCheckpointHookAPI.runHooks(J9InternalCheckpointHookAPI.java:98)
4XESTACKTRACE                at openj9/internal/criu/J9InternalCheckpointHookAPI.runPostRestoreHooksSingleThread(J9InternalCheckpointHookAPI.java:115)
4XESTACKTRACE                at openj9/internal/criu/InternalCRIUSupport.checkpointJVMImpl(Native Method)
4XESTACKTRACE                at openj9/internal/criu/InternalCRIUSupport.checkpointJVM(InternalCRIUSupport.java:937)
4XESTACKTRACE                at org/eclipse/openj9/criu/CRIUSupport.checkpointJVM(CRIUSupport.java:512)
4XESTACKTRACE                at org/openj9/criu/CRIUTestUtils.checkPointJVM(CRIUTestUtils.java:77)
4XESTACKTRACE                at org/openj9/criu/DeadlockTest.methodTypeDeadlockTest(DeadlockTest.java:222)
4XESTACKTRACE                at org/openj9/criu/DeadlockTest.main(DeadlockTest.java:58)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               protectedBacktrace+0x12 (0x00007FF338B98DF2 [libj9prt29.so+0x25df2])
4XENATIVESTACK               omrsig_protect+0x239 (0x00007FF338B9D3A9 [libj9prt29.so+0x2a3a9])
4XENATIVESTACK               omrintrospect_backtrace_thread_raw+0xbe (0x00007FF338B992CE [libj9prt29.so+0x262ce])
4XENATIVESTACK               omrsig_protect+0x239 (0x00007FF338B9D3A9 [libj9prt29.so+0x2a3a9])
4XENATIVESTACK               omrintrospect_backtrace_thread+0x87 (0x00007FF338B98C77 [libj9prt29.so+0x25c77])
4XENATIVESTACK               setup_native_thread+0x1e3 (0x00007FF338B99C43 [libj9prt29.so+0x26c43])
4XENATIVESTACK               omrintrospect_threads_startDo_with_signal+0x41f (0x00007FF338B9AD9F [libj9prt29.so+0x27d9f])
4XENATIVESTACK               omrsig_protect+0x239 (0x00007FF338B9D3A9 [libj9prt29.so+0x2a3a9])
4XENATIVESTACK               _ZN18JavaCoreDumpWriter28writeThreadsWithNativeStacksEv+0x430 (0x00007FF3389D7F60 [libj9dmp29.so+0x19f60])
4XENATIVESTACK               protectedWriteThreadsWithNativeStacks+0xd (0x00007FF3389D880D [libj9dmp29.so+0x1a80d])
4XENATIVESTACK               omrsig_protect+0x239 (0x00007FF338B9D3A9 [libj9prt29.so+0x2a3a9])
4XENATIVESTACK               _ZN18JavaCoreDumpWriter18writeThreadSectionEv+0x14b (0x00007FF3389D4BFB [libj9dmp29.so+0x16bfb])
4XENATIVESTACK               protectedWriteSection+0x1d (0x00007FF3389CFA1D [libj9dmp29.so+0x11a1d])
4XENATIVESTACK               omrsig_protect+0x239 (0x00007FF338B9D3A9 [libj9prt29.so+0x2a3a9])
4XENATIVESTACK               _ZN18JavaCoreDumpWriterC2EPKcP16J9RASdumpContextP14J9RASdumpAgent+0x3f5 (0x00007FF3389D0F05 [libj9dmp29.so+0x12f05])
4XENATIVESTACK               runJavadump+0x1c (0x00007FF3389DB13C [libj9dmp29.so+0x1d13c])
4XENATIVESTACK               doJavaDump+0x42 (0x00007FF3389C3192 [libj9dmp29.so+0x5192])
4XENATIVESTACK               protectedDumpFunction+0x15 (0x00007FF3389C27D5 [libj9dmp29.so+0x47d5])
4XENATIVESTACK               omrsig_protect+0x239 (0x00007FF338B9D3A9 [libj9prt29.so+0x2a3a9])
4XENATIVESTACK               runDumpFunction+0x62 (0x00007FF3389C5FC2 [libj9dmp29.so+0x7fc2])
4XENATIVESTACK               runDumpAgent+0x15d (0x00007FF3389C614D [libj9dmp29.so+0x814d])
4XENATIVESTACK               createAndRunOneOffDumpAgent+0x184 (0x00007FF3389C6814 [libj9dmp29.so+0x8814])
4XENATIVESTACK               triggerOneOffDump+0xb5 (0x00007FF3389DD2D5 [libj9dmp29.so+0x1f2d5])
4XENATIVESTACK               setCRIUSingleThreadModeJVMCRIUException+0x7d (0x00007FF338C3B6ED [libj9vm29.so+0x386ed])
4XENATIVESTACK               old_slow_jitMonitorEntry+0x19a (0x00007FF333D098FA [libj9jit29.so+0x9d88fa])
4XENATIVESTACK                (0x00007FF333D18ABF [libj9jit29.so+0x9e7abf])

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_x86-64_linux_OpenJDK11/78

[ThanHenderson]

Testing with #18913, I am no longer seeing this hanging.

[ThanHenderson]

Closing due to merge of #18913.