Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmdLineTester_criu_nonPortableRestore MethodTypeDeadlockTest crash #18800

Closed
pshipton opened this issue Jan 23, 2024 · 10 comments · Fixed by #18913
Closed

cmdLineTester_criu_nonPortableRestore MethodTypeDeadlockTest crash #18800

pshipton opened this issue Jan 23, 2024 · 10 comments · Fixed by #18913
Labels
comp:jit criu Used to track CRIU snapshot related work segfault Issues that describe segfaults / JVM crashes test failure

Comments

@pshipton
Copy link
Member

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/588/
cmdLineTester_criu_nonPortableRestore_8 -Xgcpolicy:gencon -Xgcthreads1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/588/functional_test_output.tar.gz

Testing: Create and Restore Criu Checkpoint Image once - MethodTypeDeadlockTest
Test start time: 2024/01/22 22:06:55 Eastern Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuScript.sh /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/java " -Xgcpolicy:gencon -Xgcthreads1  -XX:+ThrowOnDelayedCheckpointOperation -XX:sleepMillisecondsForNotCheckpointSafe=20 -Xtrace:print=j9vm.731 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED  --add-exports java.base/openj9.internal.criu=ALL-UNNAMED" org.openj9.criu.DeadlockTest MethodTypeDeadlockTest 1
Time spent starting: 2 milliseconds
Time spent executing: 23557 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/bin/java -XX:+EnableCRIUSupport  -Xgcpolicy:gencon -Xgcthreads1  -XX:+ThrowOnDelayedCheckpointOperation -XX:sleepMillisecondsForNotCheckpointSafe=20 -Xtrace:print=j9vm.731 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED  --add-exports java.base/openj9.internal.criu=ALL-UNNAMED -cp /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criu.jar org.openj9.criu.DeadlockTest MethodTypeDeadlockTest 1
 [OUT] Pre-checkpoint
 [OUT] Unhandled exception
 [OUT] Type=Segmentation error vmState=0x00000000
 [OUT] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [OUT] Handler1=0000FFFF94D11E44 Handler2=0000FFFF94C77B60 InaccessibleAddress=0000000100046E00
 [OUT] R0=0000000100046E00 R1=0000000000000200 R2=0000000100046E00 R3=0000000000000005
 [OUT] R4=000000000000000E R5=0000FFFF6DE4F8E0 R6=0000FFFEF40005B0 R7=0000000000000000
 [OUT] R8=0000000000000000 R9=0000000000000000 R10=0000000000000001 R11=0000000000000000
 [OUT] R12=0101010101010101 R13=000000000000270F R14=000000000000001F R15=00000000000020D0
 [OUT] R16=0000FFFF946200D8 R17=0000FFFF9535D358 R18=0000000000000001 R19=0000FFFEF4011B60
 [OUT] R20=0000000000000200 R21=0000000100046E00 R22=0000FFFEF4011C08 R23=0000FFFEF4011C08
 [OUT] R24=00000000001D1368 R25=0000FFFF95251F50 R26=0000000000000001 R27=0000FFFF94621B08
 [OUT] R28=0005FE14E522F540 R29=0000FFFF6DE4D850 R30=0000FFFF9535D380 R31=0000FFFF6DE4D850
 [OUT] PC=0000FFFF9535D040 SP=0000FFFF6DE4D850 PSTATE=0000000040001000
 [OUT] V0 952531f0b61f2fe1 (f: 3055497216.000000, d: -8.252167e-207)
 [OUT] V1 00000000b61f2fe1 (f: 3055497216.000000, d: 1.509616e-314)
 [OUT] V2 0000ffff233a1e11 (f: 591011328.000000, d: 1.390653e-309)
 [OUT] V3 bfd00ea348b88334 (f: 1220051712.000000, d: -2.508934e-01)
 [OUT] V4 bfdff8a542b00cd5 (f: 1118833920.000000, d: -4.995511e-01)
 [OUT] V5 3fbc5e53aa362eb4 (f: 2855677696.000000, d: 1.108143e-01)
 [OUT] V6 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
 [OUT] V7 8020080280200802 (f: 2149582848.000000, d: -4.458850e-308)
 [OUT] V8 000003c10000745f (f: 29791.000000, d: 2.039238e-311)
 [OUT] V9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V16 4010040140100401 (f: 1074791424.000000, d: 4.003911e+00)
 [OUT] V17 0100000000000000 (f: 0.000000, d: 7.291122e-304)
 [OUT] V18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V19 3f9eb851eb851eb8 (f: 3951369984.000000, d: 3.000000e-02)
 [OUT] V20 3fb1eb851eb851ec (f: 515396064.000000, d: 7.000000e-02)
 [OUT] V21 0000000000000008 (f: 8.000000, d: 3.952525e-323)
 [OUT] V22 3f0000003f800000 (f: 1065353216.000000, d: 3.051759e-05)
 [OUT] V23 3fc999999999999a (f: 2576980480.000000, d: 2.000000e-01)
 [OUT] V24 3fd6666666666666 (f: 1717986944.000000, d: 3.500000e-01)
 [OUT] V25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V26 0000000000000001 (f: 1.000000, d: 4.940656e-324)
 [OUT] V27 000000000000000a (f: 10.000000, d: 4.940656e-323)
 [OUT] V28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] Module=/lib64/libc.so.6
 [OUT] Module_base_address=0000FFFF952E0000 Symbol=strnlen
 [OUT] Symbol_address=0000FFFF9535D024
 [OUT] Target=2_90_20240123_607 (Linux 4.18.0-348.7.1.el8_5.aarch64)
 [OUT] CPU=aarch64 (8 logical CPUs) (0x1dc1a0000 RAM)
 [OUT] ----------- Stack Backtrace -----------
 [OUT] #0: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x847f84) [0xffff8fca7f84]
 [OUT] #1: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x854ab0) [0xffff8fcb4ab0]
 [OUT] #2: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x124904) [0xffff8f584904]
 [OUT] #3: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x27cd4) [0xffff94c77cd4]
 [OUT] #4: function __kernel_rt_sigreturn+0 [0]
 [OUT] #5: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e56fc) [0xffff8f6456fc]
 [OUT] #6: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e5da0) [0xffff8f645da0]
 [OUT] #7: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e5eb0) [0xffff8f645eb0]
 [OUT] #8: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e6038) [0xffff8f646038]
 [OUT] #9: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1e5918) [0xffff8f645918]
 [OUT] #10: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1d1a8c) [0xffff8f631a8c]
 [OUT] #11: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1d0a5c) [0xffff8f630a5c]
 [OUT] #12: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1135b0) [0xffff8f5735b0]
 [OUT] #13: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4aa04c) [0xffff8f90a04c]
 [OUT] #14: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4ac2d4) [0xffff8f90c2d4]
 [OUT] #15: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4c9680) [0xffff8f929680]
 [OUT] #16: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x8ae6c8) [0xffff8fd0e6c8]
 [OUT] #17: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4c9680) [0xffff8f929680]
 [OUT] #18: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1006ac) [0xffff8f5606ac]
 [OUT] #19: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4d6670) [0xffff8f936670]
 [OUT] #20: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4d72e4) [0xffff8f9372e4]
 [OUT] #21: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4d2a94) [0xffff8f932a94]
 [OUT] #22: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x4fb42c) [0xffff8f95b42c]
 [OUT] #23: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x13bc00) [0xffff8f59bc00]
 [OUT] #24: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x13ca28) [0xffff8f59ca28]
 [OUT] #25: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x287ec) [0xffff94c787ec]
 [OUT] #26: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x13a738) [0xffff8f59a738]
 [OUT] #27: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x13abe8) [0xffff8f59abe8]
 [OUT] #28: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x139970) [0xffff8f599970]
 [OUT] #29: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x139e2c) [0xffff8f599e2c]
 [OUT] Unhandled exception
 [OUT] Type=Segmentation error vmState=0x0005ff04
 [OUT] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [OUT] Handler1=0000FFFF94D11E44 Handler2=0000FFFF94C77B60 InaccessibleAddress=0000000000000008
 [OUT] R0=0000FFFF6E18DC40 R1=0000000000000000 R2=0000FFFF742C7AD8 R3=0000FFFF2E7F8DC0
 [OUT] R4=0000FFFF8F6451F4 R5=0000FFFF8FF95FD8 R6=0000FFFF941EAA59 R7=0000000072687339
 [OUT] R8=000000000A6CDCA8 R9=0000000000000018 R10=0022041FE89DA093 R11=0000265A2FEDC500
 [OUT] R12=000000007FFFFFFF R13=0000FFFF742C7B00 R14=0000000000000000 R15=0000000000000001
 [OUT] R16=0000FFFF94620010 R17=0000FFFF95364950 R18=0000000000000000 R19=0000FFFF6E18DC40
 [OUT] R20=0000000000000000 R21=0000FFFF742C7B88 R22=0000FFFF2E7F8DC0 R23=0000FFFF742C7B88
 [OUT] R24=0000FFFF742C7B88 R25=0000FFFF2E7F8DC0 R26=00000000000CC0A0 R27=0000000000000000
 [OUT] R28=0000000000000000 R29=0000FFFF742CCF10 R30=0000FFFF8F645DA0 R31=0000FFFF742C7AA0
 [OUT] PC=0000FFFF8F6456FC SP=0000FFFF742C7AA0 PSTATE=0000000000001000
 [OUT] V0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V1 0000013f000000b0 (f: 176.000000, d: 6.769167e-312)
 [OUT] V2 00000130000000a1 (f: 161.000000, d: 6.450867e-312)
 [OUT] V3 ffffffffffffffff (f: 4294967296.000000, d: -nan)
 [OUT] V4 0000000200000001 (f: 1.000000, d: 4.243992e-314)
 [OUT] V5 0000001000000010 (f: 16.000000, d: 3.395193e-313)
 [OUT] V6 0000001000000018 (f: 24.000000, d: 3.395193e-313)
 [OUT] V7 0000004000000040 (f: 64.000000, d: 1.358077e-312)
 [OUT] V8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V16 4010040140100401 (f: 1074791424.000000, d: 4.003911e+00)
 [OUT] V17 aa00000000108800 (f: 1083392.000000, d: -2.180075e-106)
 [OUT] V18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V20 3fb1eb851eb851ec (f: 515396064.000000, d: 7.000000e-02)
 [OUT] V21 0000000000000008 (f: 8.000000, d: 3.952525e-323)
 [OUT] V22 3f0000003f800000 (f: 1065353216.000000, d: 3.051759e-05)
 [OUT] V23 3fc999999999999a (f: 2576980480.000000, d: 2.000000e-01)
 [OUT] V24 3fd6666666666666 (f: 1717986944.000000, d: 3.500000e-01)
 [OUT] V25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V26 0000000000000001 (f: 1.000000, d: 4.940656e-324)
 [OUT] V27 000000000000000a (f: 10.000000, d: 4.940656e-323)
 [OUT] V28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] Module=/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
 [OUT] Module_base_address=0000FFFF8F460000
 [OUT] 
 [OUT] Method_being_compiled=java/nio/Buffer.<init>(IIII)V
 [OUT] Target=2_90_20240123_607 (Linux 4.18.0-348.7.1.el8_5.aarch64)
 [OUT] CPU=aarch64 (8 logical CPUs) (0x1dc1a0000 RAM)
 [OUT] ----------- Stack Backtrace -----------
 [OUT] strnlen+0x1c (0x0000FFFF9535D040 [libc.so.6+0x7d040])
 [OUT] strncpy+0x28 (0x0000FFFF9535D380 [libc.so.6+0x7d380])
 [OUT] getTrcBuf+0x2f4 (0x0000FFFF945DEC94 [libj9trc29.so+0xec94])
 [OUT] traceV+0x170 (0x0000FFFF945DF380 [libj9trc29.so+0xf380])
 [OUT] doTracePoint+0x768 (0x0000FFFF945E1258 [libj9trc29.so+0x11258])
 [OUT] omrTrace+0x6c (0x0000FFFF945E17BC [libj9trc29.so+0x117bc])
 [OUT] walkStackFrames+0x11c (0x0000FFFF94D4F9FC [libj9vm29.so+0x6f9fc])
 [OUT] JVM_GetCallerClass_Impl+0xbc (0x0000FFFF940AA0CC [libjclse29.so+0x5a0cc])
 [OUT]  (0x0000FFFF742D0758 [<unknown>+0x0])
 [OUT] ---------------------------------------
 [OUT] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/01/22 22:06:55 - please wait.
 [OUT] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/core.20240122.220655.3605248.0001.dmp' in response to an event
 [OUT] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/core.20240122.220655.3605248.0001.dmp
 [OUT] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/javacore.20240122.220655.3605248.0002.txt' in response to an event
 [OUT] _ZN16TR_J9SharedCache28romclassMatchesCachedVersionEP10J9ROMClassRPmS2_+0x2c (0x0000FFFF8F6456FC [libj9jit29.so+0x1e56fc])
 [OUT] _ZN16TR_J9SharedCache30validateInterfacesInClassChainEP19TR_OpaqueClassBlockRPmS2_+0x70 (0x0000FFFF8F645DA0 [libj9jit29.so+0x1e5da0])
 [OUT] _ZN16TR_J9SharedCache18validateClassChainEP10J9ROMClassP19TR_OpaqueClassBlockRPmS4_+0xc0 (0x0000FFFF8F645EB0 [libj9jit29.so+0x1e5eb0])
 [OUT] _ZN16TR_J9SharedCache25classMatchesCachedVersionEP7J9ClassPm+0x138 (0x0000FFFF8F646038 [libj9jit29.so+0x1e6038])
 [OUT] _ZN16TR_J9SharedCache13rememberClassEP7J9ClassPPK24AOTCacheClassChainRecordb+0xf8 (0x0000FFFF8F645918 [libj9jit29.so+0x1e5918])
 [OUT] _ZN30TR_ResolvedRelocatableJ9Method32storeValidationRecordIfNecessaryEPN2TR11CompilationEP14J9ConstantPooli31TR_ExternalRelocationTargetKindP8J9MethodP7J9Class+0x178 (0x0000FFFF8F631A8C [libj9jit29.so+0x1d1a8c])
 [OUT] _ZThn88_N30TR_ResolvedRelocatableJ9Method24getClassFromConstantPoolEPN2TR11CompilationEjb+0x13c (0x0000FFFF8F630A5C [libj9jit29.so+0x1d0a5c])
 [OUT] _ZN2J911Compilation17canAllocateInlineEPN2TR4NodeERP19TR_OpaqueClassBlock+0xe0 (0x0000FFFF8F5735B0 [libj9jit29.so+0x1135b0])
 [OUT] _ZN2J95ARM6413TreeEvaluator14VMnewEvaluatorEPN2TR4NodeEPNS2_13CodeGeneratorE+0x9c (0x0000FFFF8F90A04C [libj9jit29.so+0x4aa04c])
 [OUT] _ZN2J95ARM6413TreeEvaluator18newObjectEvaluatorEPN2TR4NodeEPNS2_13CodeGeneratorE+0x14 (0x0000FFFF8F90C2D4 [libj9jit29.so+0x4ac2d4])
 [OUT] _ZN3OMR13CodeGenerator8evaluateEPN2TR4NodeE+0x50 (0x0000FFFF8F929680 [libj9jit29.so+0x4c9680])
 [OUT] _ZN3OMR5ARM6413TreeEvaluator16treetopEvaluatorEPN2TR4NodeEPNS2_13CodeGeneratorE+0x24 (0x0000FFFF8FD0E6C8 [libj9jit29.so+0x8ae6c8])
 [OUT] _ZN3OMR13CodeGenerator8evaluateEPN2TR4NodeE+0x50 (0x0000FFFF8F929680 [libj9jit29.so+0x4c9680])
 [OUT] _ZN2J913CodeGenerator22doInstructionSelectionEv+0x97c (0x0000FFFF8F5606AC [libj9jit29.so+0x1006ac])
 [OUT] _ZN3OMR12CodeGenPhase32performInstructionSelectionPhaseEPN2TR13CodeGeneratorEPNS1_12CodeGenPhaseE+0x70 (0x0000FFFF8F936670 [libj9jit29.so+0x4d6670])
 [OUT] _ZN3OMR12CodeGenPhase10performAllEv+0xf4 (0x0000FFFF8F9372E4 [libj9jit29.so+0x4d72e4])
 [OUT] _ZN3OMR13CodeGenerator12generateCodeEv+0x60 (0x0000FFFF8F932A94 [libj9jit29.so+0x4d2a94])
 [OUT] _ZN3OMR11Compilation7compileEv+0x65c (0x0000FFFF8F95B42C [libj9jit29.so+0x4fb42c])
 [OUT] _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x40c (0x0000FFFF8F59BC00 [libj9jit29.so+0x13bc00])
 [OUT] _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x2c8 (0x0000FFFF8F59CA28 [libj9jit29.so+0x13ca28])
 [OUT] omrsig_protect+0x21c (0x0000FFFF94C787EC [libj9prt29.so+0x287ec])
 [OUT] _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x2c4 (0x0000FFFF8F59A738 [libj9jit29.so+0x13a738])
 [OUT] _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x118 (0x0000FFFF8F59ABE8 [libj9jit29.so+0x13abe8])
 [OUT] _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x2e0 (0x0000FFFF8F599970 [libj9jit29.so+0x139970])
 [OUT] _ZN2TR24CompilationInfoPerThread3runEv+0x4c (0x0000FFFF8F599E2C [libj9jit29.so+0x139e2c])
 [OUT] _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x7c (0x0000FFFF8F599EDC [libj9jit29.so+0x139edc])
 [OUT] omrsig_protect+0x21c (0x0000FFFF94C787EC [libj9prt29.so+0x287ec])
 [OUT] _Z21compilationThreadProcPv+0x148 (0x0000FFFF8F59A2A8 [libj9jit29.so+0x13a2a8])
 [OUT] thread_wrapper+0xcc (0x0000FFFF94C173BC [libj9thr29.so+0x73bc])
 [OUT] start_thread+0x188 (0x0000FFFF954C7898 [libpthread.so.0+0x7898])
 [OUT] thread_start+0xc (0x0000FFFF953B1DDC [libc.so.6+0xd1ddc])
 [OUT] ---------------------------------------
 [OUT] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/01/22 22:07:05 - please wait.
 [OUT] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/core.20240122.220705.3605248.0003.dmp' in response to an event
 [OUT] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/core.20240122.220705.3605248.0003.dmp
 [OUT] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/javacore.20240122.220705.3605248.0004.txt' in response to an event
 [OUT] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/javacore.20240122.220705.3605248.0004.txt
 [OUT] JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/Snap.20240122.220705.3605248.0005.trc' in response to an event
 [OUT] JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/Snap.20240122.220705.3605248.0005.trc
 [OUT] JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/jitdump.20240122.220705.3605248.0006.dmp' in response to an event
 [OUT] JVMDUMP051I JIT dump occurred in 'JIT Compilation Thread-000' thread 0x0000000000015100
 [OUT] JVMDUMP049I JIT dump notified all waiting threads of the current method to be compiled
 [OUT] JVMDUMP054I JIT dump is tracing the IL of the method on the crashed compilation thread
 [OUT] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/javacore.20240122.220655.3605248.0002.txt
 [OUT] JVMDUMP052I JIT dump recursive crash occurred on diagnostic thread
 [OUT] JVMDUMP048I JIT dump method being compiled is an ordinary method
 [OUT] JVMDUMP053I JIT dump is recompiling java/nio/Buffer.<init>(IIII)V
 [OUT] JVMDUMP052I JIT dump recursive crash occurred on diagnostic thread
 [OUT] JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_aarch64_linux_Nightly_testList_0/aqa-tests/TKG/output_17059780124913/cmdLineTester_criu_nonPortableRestore_8/jitdump.20240122.220705.3605248.0006.dmp
 [OUT] JVMDUMP013I Processed dump event "gpf", detail "".
 [OUT] Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)
 [OUT] Removed test output files
 [OUT] finished script
>> Success condition was not found: [Output match: User requested Java dump using]
>> Success condition was not found: [Output match: TEST PASSED]
>> Failure condition was not found: [Output match: TEST FAILED]
>> Required condition was found: [Output match: Pre-checkpoint]
>> Success condition was not found: [Output match: Checkpoint blocked because thread]
>> Required condition was not found: [Output match: Killed]
>> Failure condition was not found: [Output match: CRIU is not enabled]
>> Failure condition was not found: [Output match: Operation not permitted]
>> Success condition was not found: [Output match: Thread pid mismatch]
>> Success condition was not found: [Output match: do not match expected]
>> Success condition was not found: [Output match: Unable to create a thread:]
>> Failure condition was not found: [Output match: TEST FAILED]
>> Failure condition was not found: [Output match: Could not dump the JVM processes, err=-70]
@pshipton pshipton added comp:jit test failure segfault Issues that describe segfaults / JVM crashes labels Jan 23, 2024
@pshipton
Copy link
Member Author

@TobiAjila pls help direct this.

@pshipton pshipton added the criu Used to track CRIU snapshot related work label Jan 23, 2024
@pshipton
Copy link
Member Author

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/433
cmdLineTester_criu_nonPortableRestore_5 -Xgcpolicy:optthruput

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/433/functional_test_output.tar.gz

Testing: Create and Restore Criu Checkpoint Image once - MethodTypeDeadlockTest
Test start time: 2024/01/24 05:58:21 Eastern Standard Time
Running command: bash /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criuScript.sh /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/jdkbinary/j2sdk-image/bin/java " -Xgcpolicy:optthruput  -XX:+ThrowOnDelayedCheckpointOperation -XX:sleepMillisecondsForNotCheckpointSafe=20 -Xtrace:print=j9vm.731 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED  --add-exports java.base/openj9.internal.criu=ALL-UNNAMED" org.openj9.criu.DeadlockTest MethodTypeDeadlockTest 1
Time spent starting: 4 milliseconds
Time spent executing: 3499 milliseconds
Test result: FAILED
Output from test:
 [OUT] start running script
 [OUT] export GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVEC,-XSAVE,-AVX2,-ERMS,-AVX,-AVX_Fast_Unaligned_Load
 [OUT] export LD_BIND_NOT=on
 [OUT] /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/jdkbinary/j2sdk-image/bin/java -XX:+EnableCRIUSupport  -Xgcpolicy:optthruput  -XX:+ThrowOnDelayedCheckpointOperation -XX:sleepMillisecondsForNotCheckpointSafe=20 -Xtrace:print=j9vm.731 --add-opens java.base/jdk.internal.misc=ALL-UNNAMED  --add-exports java.base/openj9.internal.criu=ALL-UNNAMED -cp /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/../../jvmtest/functional/cmdLineTests/criu/criu.jar org.openj9.criu.DeadlockTest MethodTypeDeadlockTest 1
 [OUT] Pre-checkpoint
 [OUT] Unhandled exception
 [OUT] Type=Segmentation error vmState=0x00000000
 [OUT] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
 [OUT] Handler1=0000FFFF9D782F44 Handler2=0000FFFF9D6E7B60 InaccessibleAddress=0000412474736580
 [OUT] R0=0000000089383CC0 R1=0000000000000008 R2=0000000000000000 R3=0000000000000004
 [OUT] R4=0000FFFF4E87DAA0 R5=0000000000000000 R6=0000FFFF9D091B08 R7=FFFFFF80FFFFFFE0
 [OUT] R8=0000FFFF4E87DC80 R9=0000FFFF4E87DC80 R10=0000FFFF4E87DC60 R11=0000000000000011
 [OUT] R12=0000FFFF4E87DC80 R13=0000FFFF4E87DC80 R14=0000000000048108 R15=00000000890B06E8
 [OUT] R16=0000FFFF97F700B0 R17=0000FFFF9DDD5480 R18=0000FFFF4E87E300 R19=0000000000208468
 [OUT] R20=0000FFFF9D91FB40 R21=0000000000000000 R22=0000FFFF9D973810 R23=0000FFFF4E87DC80
 [OUT] R24=0000FFFF9D08F000 R25=00000000FFFFFFE0 R26=0000412474736554 R27=0000000000000004
 [OUT] R28=0000FFFF9DCC1F90 R29=0000FFFF4E87D870 R30=0000FFFF9D051258 R31=0000FFFF4E87D870
 [OUT] PC=0000FFFF9D04F2EC SP=0000FFFF4E87D870 PSTATE=0000000020001000
 [OUT] V0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V1 007200650073002e (f: 7536686.000000, d: 1.602190e-306)
 [OUT] V2 004100540045004d (f: 4522061.000000, d: 1.891455e-307)
 [OUT] V3 000000490000002d (f: 45.000000, d: 1.549057e-312)
 [OUT] V4 9728f76acff8b19e (f: 3489182208.000000, d: -4.174938e-197)
 [OUT] V5 00000000a64fe1c0 (f: 2790253056.000000, d: 1.378568e-314)
 [OUT] V6 94446f0194446f01 (f: 2487512832.000000, d: -4.855772e-211)
 [OUT] V7 000e1781000e1781 (f: 923521.000000, d: 1.959708e-308)
 [OUT] V8 000003c10000745f (f: 29791.000000, d: 2.039238e-311)
 [OUT] V9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V16 0000000044000000 (f: 1140850688.000000, d: 5.636551e-315)
 [OUT] V17 0000000400000000 (f: 0.000000, d: 8.487983e-314)
 [OUT] V18 0000000000000400 (f: 1024.000000, d: 5.059232e-321)
 [OUT] V19 3f9eb851eb851eb8 (f: 3951369984.000000, d: 3.000000e-02)
 [OUT] V20 3fb1eb851eb851ec (f: 515396064.000000, d: 7.000000e-02)
 [OUT] V21 0000000000000008 (f: 8.000000, d: 3.952525e-323)
 [OUT] V22 3f0000003f800000 (f: 1065353216.000000, d: 3.051759e-05)
 [OUT] V23 3fc999999999999a (f: 2576980480.000000, d: 2.000000e-01)
 [OUT] V24 3fd6666666666666 (f: 1717986944.000000, d: 3.500000e-01)
 [OUT] V25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V26 0000000000000001 (f: 1.000000, d: 4.940656e-324)
 [OUT] V27 000000000000000a (f: 10.000000, d: 4.940656e-323)
 [OUT] V28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] V31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
 [OUT] Module=/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/jdkbinary/j2sdk-image/lib/default/libj9trc29.so
 [OUT] Module_base_address=0000FFFF9D040000
 [OUT] Target=2_90_20240124_448 (Linux 4.18.0-348.7.1.el8_5.aarch64)
 [OUT] CPU=aarch64 (8 logical CPUs) (0x1dc1b0000 RAM)
 [OUT] ----------- Stack Backtrace -----------
 [OUT] traceV+0xdc (0x0000FFFF9D04F2EC [libj9trc29.so+0xf2ec])
 [OUT] doTracePoint+0x768 (0x0000FFFF9D051258 [libj9trc29.so+0x11258])
 [OUT] omrTrace+0x6c (0x0000FFFF9D0517BC [libj9trc29.so+0x117bc])
 [OUT] walkStackFrames+0x94 (0x0000FFFF9D7C3014 [libj9vm29.so+0x73014])
 [OUT] Java_java_security_AccessController_getAccSnapshot+0x140 (0x0000FFFF97EE8DF0 [libjclse29.so+0x18df0])
 [OUT]  (0x0000FFFF7CFC34D0 [<unknown>+0x0])
 [OUT] ---------------------------------------
 [OUT] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/01/24 05:58:21 - please wait.
 [OUT] JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/core.20240124.055821.2624260.0001.dmp' in response to an event
 [OUT] JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/core.20240124.055821.2624260.0001.dmp
 [OUT] JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/javacore.20240124.055821.2624260.0002.txt' in response to an event
 [OUT] Performing CRIUSupport.checkpointJVM(), current thread name: main, Wed Jan 24 05:58:21 EST 2024, System.currentTimeMillis(): 1706093901922, System.nanoTime(): 1801422638095851
 [OUT] JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/javacore.20240124.055821.2624260.0002.txt
 [OUT] JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/Snap.20240124.055821.2624260.0003.trc' in response to an event
 [OUT] JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/Snap.20240124.055821.2624260.0003.trc
 [OUT] JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/jitdump.20240124.055821.2624260.0004.dmp' in response to an event
 [OUT] JVMDUMP051I JIT dump occurred in 'Thread-5' thread 0x0000000000208400
 [OUT] JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk17_j9_sanity.functional_aarch64_linux_OMR_testList_1/aqa-tests/TKG/output_17060928875291/cmdLineTester_criu_nonPortableRestore_5/jitdump.20240124.055821.2624260.0004.dmp
 [OUT] JVMDUMP013I Processed dump event "gpf", detail "".
 [OUT] Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)
 [OUT] Removed test output files
 [OUT] finished script
>> Success condition was not found: [Output match: User requested Java dump using]
>> Success condition was not found: [Output match: TEST PASSED]
>> Failure condition was not found: [Output match: TEST FAILED]
>> Required condition was found: [Output match: Pre-checkpoint]
>> Success condition was not found: [Output match: Checkpoint blocked because thread]
>> Required condition was not found: [Output match: Killed]
>> Failure condition was not found: [Output match: CRIU is not enabled]
>> Failure condition was not found: [Output match: Operation not permitted]
>> Success condition was not found: [Output match: Thread pid mismatch]
>> Success condition was not found: [Output match: do not match expected]
>> Success condition was not found: [Output match: Unable to create a thread:]
>> Failure condition was not found: [Output match: TEST FAILED]
>> Failure condition was not found: [Output match: Could not dump the JVM processes, err=-70]

@tajila
Copy link
Contributor

tajila commented Jan 24, 2024

This may be a regression.

@ThanHenderson can you please take a look at this

@ThanHenderson
Copy link
Contributor

ThanHenderson commented Jan 29, 2024 •
edited
Loading

I reproduced a segmentation fault for cmdLineTester_criu_nonPortableRestore_8 on my personal x86-64_linux fyre dev machine. It is very intermittent with an observed 2/1100 failures. These failures had different stack traces than what is reported here.

For cmdLineTester_criu_nonPortableRestore_5, I tried running 1000 iterations twice, both times an iteration hung before completing all iterations.

I'll investigate to see if this is indeed a recent regression or just such an intermittent problem that it hadn't shown up for a while.

@ThanHenderson
Copy link
Contributor

This is an issue with the tracing mechanism, and doesn't show up when using -Xtrace:none (passed >10,000 iterations successfully). I'm unsure exactly under which context it is happening, but I've narrowed it down to that. I'll have more insight later today.

@eclipse-openj9 eclipse-openj9 deleted a comment from ThanHenderson Jan 30, 2024
@ThanHenderson ThanHenderson mentioned this issue Jan 31, 2024
Closed
@ThanHenderson
Copy link
Contributor

ThanHenderson commented Feb 1, 2024 •
edited
Loading

This is what a normal UtThreadData looks like

UtThreadData at 0x7fffcc004eb0 {
  Fields for UtThreadData:
        0x0: struct UtDataHeader header = !utdataheader 0x00007FFFCC004EB0
        0x10: const void* id = !j9x 0x000000000001E600
        0x18: const void* synonym1 = !j9x 0x00007FFFF00D2E90
        0x20: const void* synonym2 = !j9x 0x000000000001F020
        0x28: const U8* name = !j9x 0x00007FFFCC004F50 // "(unnamed thread)"
        0x30: U8 currentOutputMask = 0x02 (2)
        0x38: struct UtTraceBuffer* trcBuf = !uttracebuffer 0x0000000000000000
        0x40: void* external = !j9x 0x0000000000000000
        0x48: I32 suspendResume = 0x00000000 (0)
        0x4c: I32 recursion = 0x00000001 (1)
        0x50: I32 indent = 0x00000000 (0)
}

But in the failing cases the UtThreadData is initialized as the first argument here

openj9/runtime/rastrace/trclog.c

Line 2140 in 5d03ee9

doTracePoint(UT_THREAD_FROM_ENV(env), modInfo, traceId, spec, var);

produces garbage data, similar to what is seen below, which causes problems at various points downstream in the tracing pipeline

UtThreadData at 0x7ffde81b0 {
  Fields for UtThreadData:
        0x0: struct UtDataHeader header = !utdataheader 0x00000007FFDE81B0
        0x10: const void* id = !j9x 0x07130003000A1700
        0x18: const void* synonym1 = !j9x 0x0100011500071400
        0x20: const void* synonym2 = !j9x 0x0600014901000178
        0x28: const U8* name = !j9x 0x00013E74696E693C
        0x30: U8 currentOutputMask = 0x03 (3)
        0x38: struct UtTraceBuffer* trcBuf = !uttracebuffer 0x694C0F000165646F
        0x40: void* external = !j9x 0x7265626D754E656E
        0x48: I32 suspendResume = 0x6C626154 (1818386772)
        0x4c: I32 recursion = 0x12000165 (301990245)
        0x50: I32 indent = 0x61636F4C (1633906508)
}

@ThanHenderson
Copy link
Contributor

After a long period of testing and a pseudo-binary search of commits not included in 0.43 (which doesn't crash), I've identified #18745 as the commit that introduces this regression.

@JasonFengJ9
Copy link
Member

After a long period of testing and a pseudo-binary search of commits not included in 0.43 (which doesn't crash), I've identified #18745 as the commit that introduces this regression.

I will have a look it.

@ThanHenderson
Copy link
Contributor

After a long period of testing and a pseudo-binary search of commits not included in 0.43 (which doesn't crash), I've identified #18745 as the commit that introduces this regression.

This is actually not the case. After that commit, they just seem to happen more frequently. Running the commit before for ~50,000 iterations I observed the failure 4 times.

@JasonFengJ9
Copy link
Member

The MethodTypeDeadlockTest crash might be related to the two UT_MODULE_LOADED(J9_UTINTERFACE_FROM_VM(vm)) calls within j9vm module.
One is

openj9/runtime/vm/jvminit.c

Line 2878 in 43f3707

UT_MODULE_LOADED(J9_UTINTERFACE_FROM_VM(vm));
The other is

openj9/runtime/vm/CRIUHelpers.cpp

Line 903 in 43f3707

UT_MODULE_LOADED(J9_UTINTERFACE_FROM_VM(vm));
which should be removed at

I was able to reproduce the crash in a fyre x64 Linux image ~10/1000 runs, removing the UT_MODULE_LOADED() call within CRIUHelpers.cpp hasn't crashed in 2000+ runs.

@ThanHenderson could you try a build in your test environment?

ThanHenderson added a commit to ThanHenderson/openj9 that referenced this issue Feb 7, 2024
@ThanHenderson
Avoid double module trace registration after CRIU move to java.base
b74a141 
This patch fixes eclipse-openj9#18800.
When the CRIUSupport APIs were moved into java.base, the line that
previously registered the j9criu module for tracing was not removed.
Therefore, the j9vm module is being double registered, causing
downstream effects in the tracing mechanism. This patch removes
the registration during the CRIU API setup phase.

Issues: eclipse-openj9#18800
Signed-off-by: Nathan Henderson <nathan.henderson@ibm.com>
This was referenced Feb 7, 2024
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp:jit criu Used to track CRIU snapshot related work segfault Issues that describe segfaults / JVM crashes test failure
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Avoid double module trace registration after CRIU move to java.base ThanHenderson/openj9
4 participants
@tajila @JasonFengJ9 @pshipton @ThanHenderson