DaaLoadTest ConcurrentScavenge segfault or assert #11616

pshipton · 2021-01-11T14:31:00Z

Changes from previous build
eclipse-openj9/openj9-omr@d2a0060...bded46c
56d0a4a...a6756ee

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.system_ppc64le_linux_xl_Nightly_testList_0/34
DaaLoadTest_daa1_ConcurrentScavenge_0

LT stderr Unhandled exception
DLT stderr Type=Segmentation error vmState=0x00000000
DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
DLT stderr Handler1=00003FFFA922EDA0 Handler2=00003FFFA9083640
DLT stderr R0=00003FFFA8921A44 R1=00003FFF110CB800 R2=00003FFFA89E5600 R3=00003FFFA403DC70
DLT stderr R4=00003FFFA47F74E8 R5=00003FFF110CBAF0 R6=00003FFF110CB8D8 R7=00003FFF110CB8E0
DLT stderr R8=00003FFF110CB8E8 R9=00003FFFA4014978 R10=00003FFFA403DBE0 R11=00003FFFA3CE1378
DLT stderr R12=0000000000004400 R13=00003FFF110D6900 R14=00003FFE6C008290 R15=00003FFFA47F5800
DLT stderr R16=00003FFF02F00038 R17=0000000000000009 R18=00003FFF8342BD90 R19=00000000001A4EE9
DLT stderr R20=0000000000000040 R21=000000000000000A R22=0000000000000000 R23=0000000000000000
DLT stderr R24=0000000000000018 R25=00003FFFA403DBE0 R26=00003FFF110CB8D8 R27=00003FFF110CBAF0
DLT stderr R28=00003FFF110CBAF0 R29=00003FFF110CB8E8 R30=00003FFF110CB8E0 R31=0000000000000000
DLT stderr NIP=00003FFFA895C354 MSR=800000010280F033 ORIG_GPR3=00000000000081C8 CTR=00003FFFA8921880
DLT stderr LINK=00003FFFA88AC5A8 XER=0000000000000000 CCR=0000000028004482 SOFTE=0000000000000001
DLT stderr TRAP=0000000000000300 DAR=0000000000000018 dsisr=0000000040000000 RESULT=0000000000000000
DLT stderr FPR0 3fa99999a0000000 (f: 2684354560.000000, d: 5.000000e-02)
DLT stderr FPR1 4059c56e60000000 (f: 1610612736.000000, d: 1.030849e+02)
DLT stderr FPR2 408f400000000000 (f: 0.000000, d: 1.000000e+03)
DLT stderr FPR3 3fc7466496cb03de (f: 2529887232.000000, d: 1.818357e-01)
DLT stderr FPR4 3fcc71c51d8e78af (f: 495876256.000000, d: 2.222220e-01)
DLT stderr FPR5 3fd2492494229359 (f: 2485293824.000000, d: 2.857143e-01)
DLT stderr FPR6 3f59f07ca0000000 (f: 2684354560.000000, d: 1.583215e-03)
DLT stderr FPR7 3ec506d200000000 (f: 0.000000, d: 2.506571e-06)
DLT stderr FPR8 0034003900390037 (f: 3735607.000000, d: 1.112585e-307)
DLT stderr FPR9 0030003000350039 (f: 3473465.000000, d: 8.900703e-308)
DLT stderr FPR10 0031003600320034 (f: 3276852.000000, d: 9.457022e-308)
DLT stderr FPR11 37f9801900000000 (f: 0.000000, d: 4.683680e-39)
DLT stderr FPR12 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
DLT stderr FPR13 401cef5faed65272 (f: 2933281280.000000, d: 7.233763e+00)
DLT stderr FPR14 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
DLT stderr FPR15 4072d07adcbbbd22 (f: 3703291136.000000, d: 3.010300e+02)
DLT stderr FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr Module=/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_ppc64le_linux_xl_Nightly_testList_0/openjdkbinary/j2sdk-image/jre/lib/ppc64le/default/libj9gc29.so
DLT stderr Module_base_address=00003FFFA86C0000
DLT stderr Target=2_90_20210108_402 (Linux 4.4.0-173-generic)
DLT stderr CPU=ppc64le (16 logical CPUs) (0x1fe090000 RAM)
DLT stderr ----------- Stack Backtrace -----------
DLT stderr (0x00003FFFA895C354 [libj9gc29.so+0x29c354])
DLT stderr (0x00003FFF110CB850 [<unknown>+0x0])
DLT stderr (0x00003FFF110CB990 [<unknown>+0x0])
DLT stderr (0x00003FFFA8921A44 [libj9gc29.so+0x261a44])
DLT stderr (0x00003FFFA870AFA4 [libj9gc29.so+0x4afa4])
DLT stderr (0x00003FFFA3CE140C [libj9jit29.so+0xac140c])
DLT stderr (0x00003FFFA920DB64 [libj9vm29.so+0x8db64])
DLT stderr (0x00003FFFA9284EF0 [libj9vm29.so+0x104ef0])
DLT stderr (0x00003FFFA9084A48 [libj9prt29.so+0x34a48])
DLT stderr (0x00003FFFA92805A4 [libj9vm29.so+0x1005a4])
DLT stderr (0x00003FFFA91513C8 [libj9thr29.so+0x113c8])
DLT stderr (0x00003FFFA9F78040 [libpthread.so.0+0x8040])
DLT stderr clone+0x98 (0x00003FFFA9E53BB0 [libc.so.6+0x123bb0])
DLT stderr ---------------------------------------

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.system_s390x_linux_xl_Nightly_testList_0/37/
DaaLoadTest_daa1_ConcurrentScavenge_0

DLT stderr Corruption in Evacuate at 000003FFB338F1A8: calculated object size 4396758518964 larger then available 3672, Forwarded Header at 000003FFB9F7DD28
DLT stderr 04:09:27.813 0x3ffb451c600    j9mm.141    *   ** ASSERTION FAILED ** at Scavenger.cpp:1482: ((false))

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.system_x86-64_linux_xl_Nightly_testList_1/34/
DaaLoadTest_daa1_ConcurrentScavenge_0

DLT stderr Unhandled exception
DLT stderr Type=Segmentation error vmState=0x00000000
DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
DLT stderr Handler1=00007FE874A22A60 Handler2=00007FE874305020 InaccessibleAddress=000000000000001A
DLT stderr RDI=00007FE87003DE40 RSI=00007FE870790DD8 RAX=00007FE870014A00 RBX=0000000000000000
DLT stderr RCX=00007FE75CEC28D8 RDX=00007FE75CEC2A50 R8=00007FE75CEC28E0 R9=00007FE87003DDB0
DLT stderr R10=0000000000000018 R11=0000000000000012 R12=00007FE75CEC28E0 R13=00007FE75CEC28E8
DLT stderr R14=00007FE75CEC2A50 R15=00007FE75CEC28D8
DLT stderr RIP=00007FE86DD8AAA8 GS=0000 FS=0000 RSP=00007FE75CEC2840
DLT stderr EFlags=0000000000010246 CS=0033 RBP=00007FE75CEC2A50 ERR=0000000000000004
DLT stderr TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=000000000000001A
DLT stderr xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr xmm1 000000003e3ab283 (f: 1044034176.000000, d: 5.158214e-315)
DLT stderr xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr xmm3 0000000041895736 (f: 1099519744.000000, d: 5.432350e-315)
DLT stderr xmm4 000000003eccce04 (f: 1053609472.000000, d: 5.205522e-315)
DLT stderr xmm5 00000000396d7337 (f: 963474240.000000, d: 4.760195e-315)
DLT stderr xmm6 3fc5f4847813d081 (f: 2014564480.000000, d: 1.715246e-01)
DLT stderr xmm7 00000000be1cc98c (f: 3189557760.000000, d: 1.575851e-314)
DLT stderr xmm8 401cef5faed65272 (f: 2933281280.000000, d: 7.233763e+00)
DLT stderr xmm9 4072d07adcbbbd22 (f: 3703291136.000000, d: 3.010300e+02)
DLT stderr xmm10 bfb745d1745d1746 (f: 1952257920.000000, d: -9.090909e-02)
DLT stderr xmm11 bfd180bb4b651000 (f: 1264914432.000000, d: -2.734822e-01)
DLT stderr xmm12 3f80ecf4c2a50800 (f: 3265595392.000000, d: 8.264458e-03)
DLT stderr xmm13 4008000000000000 (f: 0.000000, d: 3.000000e+00)
DLT stderr xmm14 4020000000000000 (f: 0.000000, d: 8.000000e+00)
DLT stderr xmm15 0000000047054a50 (f: 1191529088.000000, d: 5.886936e-315)
DLT stderr Module=/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_x86-64_linux_xl_Nightly_testList_1/openjdkbinary/j2sdk-image/jre/lib/amd64/default/libj9gc29.so
DLT stderr Module_base_address=00007FE86DB74000
DLT stderr Target=2_90_20210108_617 (Linux 4.4.0-140-generic)
DLT stderr CPU=amd64 (8 logical CPUs) (0x3eae99000 RAM)
DLT stderr ----------- Stack Backtrace -----------
DLT stderr (0x00007FE86DD8AAA8 [libj9gc29.so+0x216aa8])
DLT stderr (0x00007FE86DCF37C2 [libj9gc29.so+0x17f7c2])
DLT stderr (0x00007FE86DD5C436 [libj9gc29.so+0x1e8436])
DLT stderr (0x00007FE86E9A40D1 [libj9jit29.so+0x9570d1])

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.system_x86-64_linux_xl_Nightly_testList_0/34
DaaLoadTest_daa2_ConcurrentScavenge_0

DLT stderr Corruption in Evacuate at 00007F3537145BB8: calculated object size 15762826332667967 larger then available 1025096, Forwarded Header at 00007F336C7DCA50
DLT stderr 04:14:34.953 0x7f3538785500    j9mm.141    *   ** ASSERTION FAILED ** at Scavenger.cpp:1447: ((false))

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.system_ppc64_aix_xl_Nightly_testList_1/32/
DaaLoadTest_daa1_ConcurrentScavenge_0

DLT stderr Bad scan type for object pointer 0A00000070013BA0
DLT stderr 09:04:02.333 0x100224f2200    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_ppc64_aix_xl_Nightly/openj9/runtime/gc_glue_java/ScavengerDelegate.cpp:390: ((false))

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_extended.system_s390x_linux_xl_Nightly_testList_1/35/
DaaLoadTest_all_ConcurrentScavenge_0

DLT stderr Corruption in Evacuate at 000003FF8103F2A8: calculated object size 79087527788568 larger then available 3416, Forwarded Header at 000003FF83AFDD28
DLT stderr 06:42:56.113 0x3ff84335a00    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_s390x_linux_xl_Nightly/omr/gc/base/standard/Scavenger.cpp:1482: ((false))

https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.system_x86-64_linux_xl_Nightly_testList_1/33/
DaaLoadTest_daa1_ConcurrentScavenge_0

DLT stderr Bad scan type for object pointer 00007EFF12FE71C0
DLT stderr 07:10:51.920 0x7efeac003a00    j9mm.141    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK11_x86-6

The text was updated successfully, but these errors were encountered:

dmitripivkine · 2021-01-11T14:46:03Z

similar failure is mentioned in #11531 , I guess one of items is duplicate of another

pshipton · 2021-01-11T15:04:29Z

#11531 (comment) is a dup of this issue. It doesn't belong in #11531.

pshipton · 2021-01-11T15:21:59Z

There are more failures in subsequent builds, not detailed here.
https://ci.eclipse.org/openj9/job/Pipeline-Build-Test-All/1148/
https://ci.eclipse.org/openj9/job/Pipeline-Build-Test-All/1149/
https://ci.eclipse.org/openj9/job/Pipeline-Build-Test-All-11/422/
https://ci.eclipse.org/openj9/job/Pipeline-Build-Test-All-11/423/

dmitripivkine · 2021-01-11T18:18:40Z

@fjeremic
looks like some Jitted code disregards new format of Indexable Object Header and treated newly introduced "Data Address" slot as a first element of array. GC receives this bogus address in Read Barrier and asserts. @bragaigor has reproducible scenario and trying to figure out details.

bragaigor · 2021-01-11T18:28:39Z

Interestingly enough, I can only consistently reproduce the issue if I run the test within GDB. Outside of it, it’s intermittent. I was also able to filter the JIT methods that's causing the issue, and they are:

+ (warm) com/ibm/dataaccess/DecimalData.convertIntegerToPackedDecimal(I[BIIZ)V @ 00007FFF5814CFA8-00007FFF5814D142 OrdinaryMethod - Q_SZ=4 Q_SZI=4 QW=48 j9m=00007FFEE806BF88 bcsz=49 compThreadID=1 CpuLoad=228%(57%avg) JvmCpu=242%
+ (warm) com/ibm/dataaccess/PackedDecimal.shiftRightPackedDecimal([BII[BIIIZZ)V @ 00007FFF57F3F748-00007FFF57F3FA8C OrdinaryMethod - Q_SZ=4 Q_SZI=3 QW=36 j9m=00007FFEE806DB78 bcsz=102 compThreadID=1 CpuLoad=228%(57%avg) JvmCpu=242%
+ (warm) com/ibm/dataaccess/DecimalData.convertPackedDecimalToInteger([BIIZ)I @ 00007FFF5814FE68-00007FFF58150002 OrdinaryMethod - Q_SZ=0 Q_SZI=0 QW=15 j9m=00007FFEE806C108 bcsz=47 compThreadID=2 CpuLoad=228%(57%avg) JvmCpu=242%

Used limitFile to filter out these, and if I leave any of these methods to be compiled the test fails.

dmitripivkine · 2021-01-11T19:12:27Z

@0xdaryl @fjeremic Please let us know if we need to revert #11507

fjeremic · 2021-01-11T20:09:52Z

@0xdaryl @fjeremic Please let us know if we need to revert #11507

If we can't fix by EOD we should revert/disable since it is a blocker.

pshipton · 2021-01-11T20:33:39Z

It's causing quite a few failures in the builds, but the failure is limited to an easily identified set of tests. I'm ok to leave the code in this week, but please revert it before the next weekend if it can't be fixed.

andrew-m-leonard · 2021-01-13T09:37:58Z

@pshipton fyi
This looks to have failed and hung at Adopt last night: https://ci.adoptopenjdk.net/job/Test_openjdk15_j9_extended.system_x86-64_windows_xl/153/console
Dumps here: https://ci.adoptopenjdk.net/job/Test_openjdk15_j9_extended.system_x86-64_windows_xl/153/artifact/system_test_output.tar.gz

01:03:05  ===============================================
01:03:05  Running test DaaLoadTest_all_ConcurrentScavenge_0 ...
01:03:05  ===============================================
01:03:05  DaaLoadTest_all_ConcurrentScavenge_0 Start Time: Wed Jan 13 01:03:05 2021 Epoch Time (ms): 1610499785520
01:03:05  "C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdkbinary/j2sdk-image\\bin\\java" -Xshareclasses:destroyAll; "C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdkbinary/j2sdk-image\\bin\\java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
01:03:05  JVMSHRC005I No shared class caches available
01:03:05  JVMSHRC005I No shared class caches available
01:03:05  cache cleanup done
01:03:05  variation: -Xgcpolicy:gencon -Xgc:concurrentScavenge
01:03:05  JVM_OPTIONS:  -Xgcpolicy:gencon -Xgc:concurrentScavenge 
01:03:05  { itercnt=1; \
01:03:05  mkdir -p "C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/\\TKG\\test_output_16104995456739\\DaaLoadTest_all_ConcurrentScavenge_0"; \
01:03:05  cd "C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/\\TKG\\test_output_16104995456739\\DaaLoadTest_all_ConcurrentScavenge_0"; \
01:03:05  perl C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests///..//jvmtest\\system\\daaLoadTest/../\\stf\\stf.core\\scripts\\stf.pl -test-root="C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests///..//jvmtest\\system\\daaLoadTest/../\\stf;C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests///..//jvmtest\\system\\daaLoadTest/../\\openjdk-systemtest;C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests///..//jvmtest\\system\\daaLoadTest/../\\openj9-systemtest" -systemtest-prereqs="C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests///..//jvmtest\\system\\daaLoadTest/../\\systemtest_prereqs" -java-args=' -Xgcpolicy:gencon -Xgc:concurrentScavenge ' -results-root="C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/\\TKG\\test_output_16104995456739\\DaaLoadTest_all_ConcurrentScavenge_0" -test=DaaLoadTest; \
01:03:05  if [ $? -eq 0 ] ; then echo ""; echo "DaaLoadTest_all_ConcurrentScavenge_0""_PASSED"; echo ""; cd C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/;  else echo ""; echo "DaaLoadTest_all_ConcurrentScavenge_0""_FAILED"; echo ""; fi; } 2>&1 | tee -a "C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/\\TKG\\test_output_16104995456739\\TestTargetResult";
01:03:06  STF 01:03:05.985 - =========================   S T F   =========================
01:03:06  systemtest-prereqs has been processed, and set to: C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/systemtest_prereqsRetrieving amount of free space on drive containing C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0
01:03:06  There is 224245 Mb free
01:03:06  STF 01:03:05.999 - ==================   G E N E R A T I O N   ==================
01:03:06  STF 01:03:06.001 - Checking JVM: C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdkbinary/j2sdk-image
01:03:06  STF 01:03:06.002 - Starting process to generate scripts: C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdkbinary/j2sdk-image/bin/java  -Dlog4j.skipJansi=true -Djava.system.class.loader=net.adoptopenjdk.stf.runner.StfClassLoader -classpath C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests///..//jvmtest\system\daaLoadTest/../\systemtest_prereqs/log4j-2.13.3/log4j-api-2.13.3.jar;C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests///..//jvmtest\system\daaLoadTest/../\systemtest_prereqs/log4j-2.13.3/log4j-core-2.13.3.jar;C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/stf/stf.core/scripts/../bin net.adoptopenjdk.stf.runner.StfRunner -properties "C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0/20210113-010305-DaaLoadTest/stf_parameters.properties, , C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/stf/stf.core/config/stf.properties" -testDir "C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0/20210113-010305-DaaLoadTest"
01:03:07  GEN 09:03:07.881 - Found test. Project: 'openj9.test.load' class: 'DaaLoadTest.class' Dir: 'C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\jvmtest\system\openj9-systemtest\openj9.test.load\bin'
01:03:07  GEN 09:03:07.891 - Found test. Project: 'openj9.test.load' class: 'net.openj9.stf.DaaLoadTest'
01:03:08  GEN Classpath directories used by project 'openj9.test.load': 
01:03:08  GEN   C:\\Users\\Jenkins\\workspace\\Test_openjdk15_j9_extended.system_x86-64_windows_xl\\jvmtest\\system\\openj9-systemtest\\openj9.test.load\\bin
01:03:08  GEN   C:\\Users\\Jenkins\\workspace\\Test_openjdk15_j9_extended.system_x86-64_windows_xl\\jvmtest\\system\\stf\\stf.core\\bin
01:03:08  GEN   C:\\Users\\Jenkins\\workspace\\Test_openjdk15_j9_extended.system_x86-64_windows_xl\\jvmtest\\system\\stf\\stf.load\\bin
01:03:08  GEN   C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/systemtest_prereqs/log4j-2.13.3/log4j-api-2.13.3.jar
01:03:08  GEN   C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/systemtest_prereqs/log4j-2.13.3/log4j-core-2.13.3.jar
01:03:08  GEN   C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/systemtest_prereqs/junit-4.12/junit-4.12.jar
01:03:08  GEN   C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/systemtest_prereqs/junit-4.12/hamcrest-core-1.3.jar
01:03:08  GEN 09:03:08.325 - Using Mode NoOptions. Values = ''
01:03:08  GEN 09:03:08.491 - 
01:03:08  GEN 09:03:08.491 - Test command summary:
01:03:08  GEN 09:03:08.491 -   Step  Stage   Command           Description
01:03:08  GEN 09:03:08.492 -  -----+--------+-----------------+------------
01:03:08  GEN 09:03:08.492 -     1  execute  Run java          Run daa load test
01:03:08  STF 01:03:08.531 - 
01:03:08  STF 01:03:08.531 - Script generation completed
01:03:08  STF 01:03:08.531 - 
01:03:08  STF 01:03:08.531 - 
01:03:08  STF 01:03:08.531 - =======================   S E T U P   =======================
01:03:08  STF 01:03:08.531 - Running setup: perl C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0/20210113-010305-DaaLoadTest/setUp.pl
01:03:08  STF 01:03:08.627 - SETUP stage completed
01:03:08  STF 01:03:08.637 - 
01:03:08  STF 01:03:08.637 - ====================   E X E C U T E -   ====================
01:03:08  STF 01:03:08.637 - Running execute: perl C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0/20210113-010305-DaaLoadTest/execute.pl
01:03:08  STF 01:03:08.741 - 
01:03:08  STF 01:03:08.741 - Java version
01:03:08  STF 01:03:08.741 - Running: C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdkbinary/j2sdk-image/bin/java -version
01:03:08  openjdk version "15.0.1" 2020-10-20
01:03:08  OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9-202101122336)
01:03:08  Eclipse OpenJ9 VM AdoptOpenJDK (build master-5bd152fc4, JRE 15 Windows Server 2012 R2 amd64-64-Bit 20210112_168 (JIT enabled, AOT enabled)
01:03:08  OpenJ9   - 5bd152fc4
01:03:08  OMR      - bded46caf
01:03:08  JCL      - 0e6bc58a45 based on jdk-15.0.1+9)
01:03:08  STF 01:03:08.878 - 
01:03:08  STF 01:03:08.878 - +------ Step 1 - Run daa load test
01:03:08  STF 01:03:08.878 - | Run foreground process
01:03:08  STF 01:03:08.878 - |   Program:     C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdkbinary/j2sdk-image/bin/java
01:03:08  STF 01:03:08.878 - |   Mnemonic:    DLT
01:03:08  STF 01:03:08.878 - |   Echo:        ECHO_ON
01:03:08  STF 01:03:08.878 - |   Expectation: CLEAN_RUN within 15h
01:03:08  STF 01:03:08.878 - |
01:03:08  STF 01:03:08.879 - Running command: C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdkbinary/j2sdk-image/bin/java -Xgcpolicy:gencon -Xgc:concurrentScavenge -classpath C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\jvmtest\system\stf\stf.load\bin;C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\jvmtest\system\stf\stf.core\bin;C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/systemtest_prereqs/log4j-2.13.3/log4j-api-2.13.3.jar;C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/systemtest_prereqs/log4j-2.13.3/log4j-core-2.13.3.jar;C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/systemtest_prereqs/junit-4.12/junit-4.12.jar;C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/jvmtest/system/systemtest_prereqs/junit-4.12/hamcrest-core-1.3.jar;C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\jvmtest\system\openj9-systemtest\openj9.test.daa\bin net.adoptopenjdk.loadTest.LoadTest -resultsDir C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results -resultsPrefix 1.DLT. -dumpRequested false -reportFailureLimit 1 -abortAtFailureLimit 10 -maxTotalLogFileSpace 200M -maxSingleLogSize 1/25 -suite.daa.threadCount 2 -suite.daa.totalNumberTests 7200 -suite.daa.inventoryFile C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/TKG/test_output_16104995456739/DaaLoadTest_all_ConcurrentScavenge_0/20210113-010305-DaaLoadTest/results/1.DLT.inventory/openj9.test.load/config/inventories/daa/daaAll.xml -suite.daa.inventoryExcludeFile none -suite.daa.selection random -suite.daa.seed -1 -suite.daa.repeatCount 1 -suite.daa.thinkingTime 0ms..0ms
01:03:08  STF 01:03:08.879 - Redirecting stderr to C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/TKG/test_output_16104995456739/DaaLoadTest_all_ConcurrentScavenge_0/20210113-010305-DaaLoadTest/results/1.DLT.stderr
01:03:08  STF 01:03:08.879 - Redirecting stdout to C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/TKG/test_output_16104995456739/DaaLoadTest_all_ConcurrentScavenge_0/20210113-010305-DaaLoadTest/results/1.DLT.stdout
01:03:08  STF 01:03:08.897 - Monitoring processes: DLT
01:03:10  DLT 09:03:10.133 - Load test parameters
01:03:10  DLT 09:03:10.138 -   Time limited         = false
01:03:10  DLT 09:03:10.139 -   abortIfOutOfMemory   = true
01:03:10  DLT 09:03:10.139 -   reportFailureLimit   = 1
01:03:10  DLT 09:03:10.140 -   abortAtFailureLimit  = 10
01:03:10  DLT 09:03:10.141 -   maxTotalLogFileSpace = 209715200
01:03:10  DLT 09:03:10.141 -   maxSingleLogSize     = 8388608
01:03:10  DLT 09:03:10.142 - Parameters for suite 0
01:03:10  DLT 09:03:10.142 -   Suite name     = daa
01:03:10  DLT 09:03:10.143 -   Number threads = 2
01:03:10  DLT 09:03:10.143 -   Supplied seed  = -1
01:03:10  DLT 09:03:10.144 -   Inventory file = C:/Users/Jenkins/workspace/Test_openjdk15_j9_extended.system_x86-64_windows_xl/openjdk-tests/TKG/test_output_16104995456739/DaaLoadTest_all_ConcurrentScavenge_0/20210113-010305-DaaLoadTest/results/1.DLT.inventory/openj9.test.load/config/inventories/daa/daaAll.xml
01:03:10  DLT 09:03:10.144 -   Exclude file   = none
01:03:10  DLT 09:03:10.145 -   Number tests   = 7200
01:03:10  DLT 09:03:10.145 -   Repeat count   = 1
01:03:10  DLT 09:03:10.145 -   Thinking time  = 0ms..0ms
01:03:10  DLT 09:03:10.145 -   Selection mode = random
01:03:10  DLT 09:03:10.146 -   Actual seed    = 1610607281314
01:03:10  DLT 09:03:10.155 - Parsing inventory file. Root=C:\\Users\\Jenkins\\workspace\\Test_openjdk15_j9_extended.system_x86-64_windows_xl\\openjdk-tests\\TKG\\test_output_16104995456739\\DaaLoadTest_all_ConcurrentScavenge_0\\20210113-010305-DaaLoadTest\\results\\1.DLT.inventory File=openj9.test.load/config/inventories/daa/daaAll.xml
01:03:10  DLT 09:03:10.160 - Found included inventory. File=/openj9.test.load/config/inventories/daa/daa1.xml
01:03:10  DLT 09:03:10.161 - Parsing inventory file. Root=C:\\Users\\Jenkins\\workspace\\Test_openjdk15_j9_extended.system_x86-64_windows_xl\\openjdk-tests\\TKG\\test_output_16104995456739\\DaaLoadTest_all_ConcurrentScavenge_0\\20210113-010305-DaaLoadTest\\results\\1.DLT.inventory File=/openj9.test.load/config/inventories/daa/daa1.xml
01:03:10  DLT 09:03:10.173 - Found included inventory. File=/openj9.test.load/config/inventories/daa/daa2.xml
01:03:10  DLT 09:03:10.174 - Parsing inventory file. Root=C:\\Users\\Jenkins\\workspace\\Test_openjdk15_j9_extended.system_x86-64_windows_xl\\openjdk-tests\\TKG\\test_output_16104995456739\\DaaLoadTest_all_ConcurrentScavenge_0\\20210113-010305-DaaLoadTest\\results\\1.DLT.inventory File=/openj9.test.load/config/inventories/daa/daa2.xml
01:03:10  DLT 09:03:10.185 - Found included inventory. File=/openj9.test.load/config/inventories/daa/daa3.xml
01:03:10  DLT 09:03:10.186 - Parsing inventory file. Root=C:\\Users\\Jenkins\\workspace\\Test_openjdk15_j9_extended.system_x86-64_windows_xl\\openjdk-tests\\TKG\\test_output_16104995456739\\DaaLoadTest_all_ConcurrentScavenge_0\\20210113-010305-DaaLoadTest\\results\\1.DLT.inventory File=/openj9.test.load/config/inventories/daa/daa3.xml
01:03:10  DLT 09:03:10.194 - Final test list:
01:03:10  DLT 09:03:10.195 -   0 ArbitraryJava[net.openj9.test.simple.ConvertDecimal invokeTest]  Weighting=1 
01:03:10  DLT 09:03:10.195 -   1 ArbitraryJava[net.openj9.test.simple.MarshalUnmarshalBinary invokeTest]  Weighting=1 
01:03:10  DLT 09:03:10.195 -   2 JUnit[net.openj9.test.arithmetics.TestArithmeticOperations]  Weighting=1 
01:03:10  DLT 09:03:10.196 -   3 JUnit[net.openj9.test.arithmetics.TestArithmeticInline]  Weighting=1 
01:03:10  DLT 09:03:10.196 -   4 JUnit[net.openj9.test.arithmetics.TestArithmetics]  Weighting=1 
01:03:10  DLT 09:03:10.196 -   5 JUnit[net.openj9.test.arithmetics.TestComparisonEquals]  Weighting=1 
01:03:10  DLT 09:03:10.196 -   6 JUnit[net.openj9.test.arithmetics.TestPDComparisons]  Weighting=1 
01:03:10  DLT 09:03:10.196 -   7 JUnit[net.openj9.test.arithmetics.TestPerformance]  Weighting=1 
01:03:10  DLT 09:03:10.196 -   8 JUnit[net.openj9.test.arithmetics.TestSubExceptions]  Weighting=1 
01:03:10  DLT 09:03:10.196 -   9 JUnit[net.openj9.test.arithmetics.TestValidityChecking]  Weighting=1 
01:03:10  DLT 09:03:10.197 -   10 JUnit[net.openj9.test.binaryData.LongIntegerComparison]  Weighting=1 
01:03:10  DLT 09:03:10.197 -   11 JUnit[net.openj9.test.binaryData.TestOptimizer]  Weighting=1 
01:03:10  DLT 09:03:10.197 -   12 JUnit[net.openj9.test.decimals.TestBD2PD2BD]  Weighting=1 
01:03:10  DLT 09:03:10.197 -   13 JUnit[net.openj9.test.decimals.TestDecimalData]  Weighting=1 
01:03:10  DLT 09:03:10.197 -   14 JUnit[net.openj9.test.decimals.TestDecimalData2]  Weighting=1 
01:03:10  DLT 09:03:10.197 -   15 JUnit[net.openj9.test.PD2Primitive.TestPD2Primitives2PD]  Weighting=1 
01:03:10  DLT 09:03:10.198 -   16 JUnit[net.openj9.test.PDMoveShifts.TestShiftsAndConvert]  Weighting=1 
01:03:10  DLT 09:03:10.198 -   17 JUnit[net.openj9.test.PDMoveShifts.TestPDMove]  Weighting=1 
01:03:10  DLT 09:03:10.198 -   18 ArbitraryJava[net.openj9.test.PDMoveShifts.ShiftTestRunner invokeTest]  Weighting=1 
01:03:10  DLT 09:03:10.198 -   19 ArbitraryJava[net.openj9.test.decimals.DecimalTestRunner invokeTest]  Weighting=1 
01:03:10  DLT 09:03:10.198 -   20 ArbitraryJava[net.openj9.test.MarshallingTestRunner invokeTest]  Weighting=1 
01:03:10  DLT 09:03:10.198 -   21 ArbitraryJava[net.openj9.test.PD2Primitive.I2PD_PD2ITestRunner invokeTest]  Weighting=1 
01:03:10  DLT 09:03:10.198 -   22 JUnit[net.openj9.test.arithmetics.TestArithmeticOperations]  Weighting=1 
01:03:10  DLT 09:03:10.198 -   23 JUnit[net.openj9.test.binaryData.TestBinaryConverters]  Weighting=1 
01:03:10  DLT 09:03:10.199 -   24 JUnit[net.openj9.test.binaryData.TestByteArray2Double]  Weighting=1 
01:03:10  DLT 09:03:10.199 -   25 JUnit[net.openj9.test.binaryData.TestByteArray2Float]  Weighting=1 
01:03:10  DLT 09:03:10.199 -   26 JUnit[net.openj9.test.binaryData.TestByteArray2Integer]  Weighting=1 
01:03:10  DLT 09:03:10.199 -   27 JUnit[net.openj9.test.binaryData.TestByteArray2IntegerNumBytes]  Weighting=1 
01:03:10  DLT 09:03:10.199 -   28 JUnit[net.openj9.test.binaryData.TestByteArray2Long]  Weighting=1 
01:03:10  DLT 09:03:10.199 -   29 JUnit[net.openj9.test.binaryData.TestByteArray2LongNumBytes]  Weighting=1 
01:03:10  DLT 09:03:10.200 -   30 JUnit[net.openj9.test.binaryData.TestByteArray2Short]  Weighting=1 
01:03:10  DLT 09:03:10.200 -   31 JUnit[net.openj9.test.binaryData.TestByteArray2ShortNumBytes]  Weighting=1 
01:03:10  DLT 09:03:10.200 -   32 JUnit[net.openj9.test.binaryData.TestDouble2ByteArray]  Weighting=1 
01:03:10  DLT 09:03:10.200 -   33 JUnit[net.openj9.test.binaryData.TestFloat2ByteArray]  Weighting=1 
01:03:10  DLT 09:03:10.200 -   34 JUnit[net.openj9.test.binaryData.TestInteger2ByteArray]  Weighting=1 
01:03:10  DLT 09:03:10.201 -   35 JUnit[net.openj9.test.binaryData.TestInteger2ByteArrayNumBytes]  Weighting=1 
01:03:10  DLT 09:03:10.201 -   36 JUnit[net.openj9.test.binaryData.TestLong2ByteArray]  Weighting=1 
01:03:10  DLT 09:03:10.201 -   37 JUnit[net.openj9.test.binaryData.TestLong2ByteArrayNumBytes]  Weighting=1 
01:03:10  DLT 09:03:10.201 -   38 JUnit[net.openj9.test.binaryData.TestShort2ByteArray]  Weighting=1 
01:03:10  DLT 09:03:10.201 -   39 JUnit[net.openj9.test.binaryData.TestShort2ByteArrayNumBytes]  Weighting=1 
01:03:10  DLT 09:03:10.201 -   40 JUnit[net.openj9.test.decimals.TestED2PD]  Weighting=1 
01:03:10  DLT 09:03:10.202 -   41 JUnit[net.openj9.test.decimals.TestPD2ED]  Weighting=1 
01:03:10  DLT 09:03:10.202 -   42 JUnit[net.openj9.test.decimals.TestPD2UD]  Weighting=1 
01:03:10  DLT 09:03:10.202 -   43 JUnit[net.openj9.test.decimals.TestUD2PD]  Weighting=1 
01:03:10  DLT 09:03:10.202 -   44 JUnit[net.openj9.test.PD2Primitive.RandomTests]  Weighting=1 
01:03:10  DLT 09:03:10.202 -   45 JUnit[net.openj9.test.PD2Primitive.TestI2PD]  Weighting=1 
01:03:10  DLT 09:03:10.202 -   46 JUnit[net.openj9.test.PD2Primitive.TestPD2I]  Weighting=1 
01:03:10  DLT 09:03:10.202 -   47 JUnit[net.openj9.test.arithmetics.TestArithmeticOperations]  Weighting=1 
01:03:10  DLT 09:03:10.261 - Starting thread. Suite=0 thread=0
01:03:10  DLT 09:03:10.262 - Starting thread. Suite=0 thread=1
01:03:31  DLT 09:03:30.284 - Completed 3.9%. Number of tests started=283
01:03:51  DLT 09:03:50.253 - Completed 11.9%. Number of tests started=860 (+577)
01:04:12  DLT 09:04:10.319 - Completed 18.4%. Number of tests started=1324 (+464)
01:04:32  DLT 09:04:30.305 - Completed 26.7%. Number of tests started=1924 (+600)
01:04:51  DLT 09:04:50.298 - Completed 37.5%. Number of tests started=2698 (+774)
01:05:10  DLT 09:05:10.270 - Completed 50.7%. Number of tests started=3647 (+949)
01:05:34  DLT 09:05:30.268 - Completed 62.3%. Number of tests started=4488 (+841)
01:05:51  DLT 09:05:50.293 - Completed 72.0%. Number of tests started=5184 (+696)
01:06:14  DLT 09:06:10.326 - Completed 82.3%. Number of tests started=5923 (+739)
01:06:30  DLT 09:06:30.272 - Completed 93.9%. Number of tests started=6762 (+839)
01:06:33  DLT stderr Bad scan type for object pointer 00007FF7C1365308
01:06:33  DLT stderr 01:06:32.287 0xc1a6807f00    j9mm.141    *   ** ASSERTION FAILED ** at ../../gc_glue_java/ScavengerDelegate.cpp:390: ((false))
01:06:33  DLT stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2021/01/13 09:06:32 - please wait.
01:06:33  DLT stderr JVMDUMP032I JVM requested System dump using 'C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\core.20210113.090632.14956.0001.dmp' in response to an event
01:06:33  STF 01:06:32.407 - Found dump at: C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\core.20210113.090632.14956.0001.dmp
01:06:33  DLT stderr core file generated - C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\core.20210113.090632.14956.0001.dmp
01:06:36  DLT stderr 56.0001.dmp
01:06:36  DLT stderr Assertion failed: false && !(_currentPhaseConcurrent), file Scavenger.cpp, line 4040
01:06:36  DLT stderr JVMDUMP039I Processing dump event "abort", detail "" at 2021/01/13 09:06:35 - please wait.
01:06:36  DLT stderr JVMDUMP032I JVM requested System dump using 'C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\core.20210113.090635.14956.0003.dmp' in response to an event
01:06:36  STF 01:06:35.539 - Found dump at: C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\core.20210113.090635.14956.0003.dmp
01:06:36  DLT stderr core file generated - C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\core.20210113.090635.14956.0003.dmp
01:06:36  DLT stderr 56.0003.dmp
01:06:36  DLT stderr JVMDUMP032I JVM requested Java dump using 'C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\javacore.20210113.090635.14956.0004.txt' in response to an event
01:06:36  DLT stderr JVMDUMP010I Java dump written to C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\javacore.20210113.090635.14956.0004.txt
01:06:36  DLT stderr JVMDUMP032I JVM requested Snap dump using 'C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\Snap.20210113.090635.14956.0005.trc' in response to an event
01:06:36  DLT stderr JVMDUMP010I Snap dump written to C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\Snap.20210113.090635.14956.0005.trc
01:06:36  DLT stderr JVMDUMP032I JVM requested JIT dump using 'C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\jitdump.20210113.090635.14956.0006.dmp' in response to an event
01:06:36  STF 01:06:36.542 - Found dump at: C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\javacore.20210113.090635.14956.0004.txt
01:06:36  STF 01:06:36.557 - Found dump at: C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\Snap.20210113.090635.14956.0005.trc
01:06:37  DLT stderr javacore file generated - C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\javacore.20210113.090635.14956.0004.txt
01:06:37  DLT stderr Snap file generated - C:\Users\Jenkins\workspace\Test_openjdk15_j9_extended.system_x86-64_windows_xl\openjdk-tests\TKG\test_output_16104995456739\DaaLoadTest_all_ConcurrentScavenge_0\20210113-010305-DaaLoadTest\results\Snap.20210113.090635.14956.0005.trc
01:08:15  STF 01:08:08.164 - Heartbeat: Process DLT is still running
01:13:10  STF 01:13:08.331 - Heartbeat: Process DLT is still running
01:18:19  STF 01:18:08.499 - Heartbeat: Process DLT is still running
01:23:14  STF 01:23:08.166 - Heartbeat: Process DLT is still running
01:28:23  STF 01:28:08.333 - Heartbeat: Process DLT is still running
01:33:17  STF 01:33:08.501 - Heartbeat: Process DLT is still running
01:38:12  STF 01:38:08.167 - Heartbeat: Process DLT is still running

bragaigor · 2021-01-14T02:02:15Z

There has been some progress regarding the diagnosis of the failures. There seems to be some code related to read barrier in concurrent scavenger that thinks dataAddr is the first element of a contiguous array. @fjeremic has a lead on it (PrefetchInsertion optimization), so we'll continue to look into it

andrew-m-leonard · 2021-01-14T10:17:47Z

@pshipton i'm going to exclude this test (DaaLoadTest_all_ConcurrentScavenge_0), as it is causing numerous hung processes on Adopt machines, causing subsequent build/test issues.
@smlambert fyi

andrew-m-leonard · 2021-01-14T11:49:07Z

@pshipton @smlambert fyi, the Adopt DaaLoadTest hung Java jobs are due to the excessive DaaAllTest timeouts of 15hours, I have raised: adoptium/openj9-systemtest#121

pshipton · 2021-01-14T13:39:31Z

@andrew-m-leonard is there a PR for excluding the test? I think it's more than one test that can fail. We could just back out the problematic change. We haven't been seeing any hangs in the OpenJ9 builds, so it didn't seem much of a problem.

pshipton · 2021-01-14T13:54:14Z

I've reverted #11507 via #11653

pshipton · 2021-01-14T13:55:32Z

I'll keep the test excluded label until it's determined if anything was actually excluded.

pshipton · 2021-01-14T14:11:45Z

@fjeremic fyi, looks like the hang in #11616 (comment) is related to the jitdump. There is a JVM requested JIT dump message, but nothing indicating it finished.

pshipton · 2021-01-15T15:12:28Z

Although #11507 was reverted, there was another DaaLoadTest failure. Is this a different problem, or is this Issue not caused by 11507? @fjeremic @bragaigor

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.system_ppc64_aix_Nightly/612
DaaLoadTest_daa1_ConcurrentScavenge_0

DLT stderr Unhandled exception
DLT stderr Type=Segmentation error vmState=0x00020002
DLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000033
DLT stderr Handler1=09001000A136EC80 Handler2=09001000A1345810
DLT stderr R0=0000000000000001 R1=00000100225C8C10 R2=09001000A1265258 R3=00000100100A9558
DLT stderr R4=0000000000000001 R5=0000000000000000 R6=0000000080A004B8 R7=00000100101805D0
DLT stderr R8=0000000000000090 R9=0000000000000090 R10=0000000000000030 R11=0000000000000000
DLT stderr R12=0000000000000000 R13=00000100225D1800 R14=0000000000000000 R15=0000000000000000
DLT stderr R16=0000000000000000 R17=0000000000000000 R18=0000000000000000 R19=0000000000000000
DLT stderr R20=0000000000000001 R21=0000000000000001 R22=0000000000000000 R23=00000000000000D0
DLT stderr R24=00000000000000C0 R25=0000000000000004 R26=0000010010688450 R27=0000000000000001
DLT stderr R28=0000010010688370 R29=000000000000000C R30=0000000000000001 R31=0000010022583BF0
DLT stderr IAR=0900000013ED06FC LR=0900000013EFA270 MSR=A00000000000D032 CTR=0900000013EFCA80
DLT stderr CR=2400024B20000001 FPSCR=8202000000000000 XER=2000000182020000
DLT stderr FPR0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
DLT stderr FPR2 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
DLT stderr FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR5 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
DLT stderr FPR6 4070b1f20d4afc0c (f: 223018000.000000, d: 2.671216e+02)
DLT stderr FPR7 408f400000000000 (f: 0.000000, d: 1.000000e+03)
DLT stderr FPR8 4104161000000000 (f: 0.000000, d: 1.645460e+05)
DLT stderr FPR9 4530000000000000 (f: 0.000000, d: 1.934281e+25)
DLT stderr FPR10 408f400000000000 (f: 0.000000, d: 1.000000e+03)
DLT stderr FPR11 43300000000003e8 (f: 1000.000000, d: 4.503600e+15)
DLT stderr FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
DLT stderr FPR13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
DLT stderr Module=/home/jenkins/workspace/Test_openjdk8_j9_sanity.system_ppc64_aix_Nightly_testList_1/openjdkbinary/j2sdk-image/jre/lib/ppc64/compressedrefs/libj9gc29.so
DLT stderr Module_base_address=0900000013EA8000
DLT stderr Target=2_90_20210114_641 (AIX 7.1)
DLT stderr CPU=ppc64 (16 logical CPUs) (0x200000000 RAM)
DLT stderr ----------- Stack Backtrace -----------
DLT stderr add__34MM_OwnableSynchronizerObjectBufferFP18MM_EnvironmentBaseP8J9Object+0x44c (0x0900000013EFA270 [libj9gc29.so+0x52270])
DLT stderr scanOwnableSynchronizerObjects__27MM_MarkingSchemeRootClearerFP18MM_EnvironmentBase+0x440 (0x0900000013F77164 [libj9gc29.so+0xcf164])
DLT stderr scanClearable__14MM_RootScannerFP18MM_EnvironmentBase+0x248 (0x0900000013F6694C [libj9gc29.so+0xbe94c])
DLT stderr workerCompleteGC__18MM_MarkingDelegateFP18MM_EnvironmentBase+0x1a0 (0x0900000013F5E304 [libj9gc29.so+0xb6304])
DLT stderr markLiveObjectsComplete__16MM_MarkingSchemeFP18MM_EnvironmentBase+0x10 (0x0900000013F56514 [libj9gc29.so+0xae514])
DLT stderr run__19MM_ParallelMarkTaskFP18MM_EnvironmentBase+0x6c (0x0900000014088750 [libj9gc29.so+0x1e0750])
DLT stderr workerEntryPoint__21MM_ParallelDispatcherFP18MM_EnvironmentBase+0x22c (0x0900000014017FB0 [libj9gc29.so+0x16ffb0])
DLT stderr dispatcher_thread_proc2__FP14OMRPortLibraryPv+0x27c (0x0900000014017100 [libj9gc29.so+0x16f100])
DLT stderr omrsig_protect+0x488 (0x0900000012920BCC [libj9prt29.so+0x59bcc])
DLT stderr dispatcher_thread_proc+0x4c (0x0900000014017D30 [libj9gc29.so+0x16fd30])
DLT stderr thread_wrapper+0x524 (0x09000000128A7968 [libj9thr29.so+0x4968])
DLT stderr _pthread_body+0xf0 (0x0900000000570E14 [libpthreads.a+0x3e14])
DLT stderr ---------------------------------------

pshipton · 2021-01-15T15:14:27Z

@dmitripivkine fyi as well the previous comment, since it's crashing in the GC.

pshipton · 2021-01-15T15:15:49Z

Seems this latest crash belongs in #10489, I'll add it there.

bragaigor · 2021-01-15T15:20:23Z

#11507 was indeed causing the issue, maybe 11507 made the issue more obvious and to crash more often? I tested locally and eclipse-omr/omr#5751 seemed to have fixed 11507 problem. But not sure if this is last crash could be related or if Filip's fix would solve it.

dmitripivkine · 2021-01-15T15:43:47Z

#11507 was indeed causing the issue, maybe 11507 made the issue more obvious and to crash more often? I tested locally and eclipse/omr#5751 seemed to have fixed 11507 problem. But not sure if this is last crash could be related or if Filip's fix would solve it.

last crash is unrelated known problem #10489

Same commit as eclipse-openj9#11507 The issue related to concurrent scavenger (eclipse-openj9#11616) was solved by eclipse-omr/omr#5751 We can now safely re-introduce dataAddr again. Signed-off-by: Igor Braga <higorb1@gmail.com>

Same commit as eclipse-openj9#11507 The issue related to concurrent scavenger (eclipse-openj9#11616) was solved by eclipse-omr/omr#5751 We can now safely re-introduce dataAddr again. Conflicts: runtime/compiler/x/codegen/J9TreeEvaluator.cpp Change-Id: I827e6d63c33b982b828820f486a0e16a857cec3a Signed-off-by: Igor Braga <higorb1@gmail.com> Signed-off-by: Lin Hu <linhu@ca.ibm.com>

Same commit as eclipse-openj9#11507 The issue related to concurrent scavenger (eclipse-openj9#11616) was solved by eclipse-omr/omr#5751 We can now safely re-introduce dataAddr again. Signed-off-by: Igor Braga <higorb1@gmail.com>

pshipton added comp:gc test failure blocker segfault Issues that describe segfaults / JVM crashes labels Jan 11, 2021

pshipton mentioned this issue Jan 11, 2021

JDK8 zLinux Segmentation error vmState=0x00000000 Compiled_method=java/lang/reflect/Method.getParameterAnnotations() #11531

Closed

dmitripivkine added the comp:jit label Jan 11, 2021

dmitripivkine removed the comp:gc label Jan 11, 2021

dmitripivkine mentioned this issue Jan 11, 2021

JDK11 Mac OS : DaaLoadTest_daa1_ConcurrentScavenge_0_FAILED - Segmentation error vmState=0x00000000 at libj9gc29.dylib #11630

Closed

dsouzai mentioned this issue Jan 12, 2021

General Relocation Infrastructure Clean up #11235

Merged

pshipton added this to the Release 0.25 (Java 16) milestone Jan 12, 2021

pshipton mentioned this issue Jan 13, 2021

JDK8 AIX - j9mm.141 * ** ASSERTION FAILED ** at ../../gc_glue_java/ScavengerDelegate.cpp:390: ((false)) #11641

Closed

pshipton added the test excluded label Jan 14, 2021

This was referenced Jan 14, 2021

Revert "Add extra U_64 data field to Indexable Object Headers" #11653

Merged

Add extra U_64 data field to Indexable Object Headers #11507

Merged

pshipton removed the blocker label Jan 14, 2021

pshipton removed the test excluded label Jan 14, 2021

andrew-m-leonard mentioned this issue Jan 14, 2021

DaaAll test timeouts are too long adoptium/openj9-systemtest#121

Closed

fjeremic mentioned this issue Jan 14, 2021

Disallow prefetch insertion when read barriers are necessary eclipse-omr/omr#5751

Merged

0xdaryl closed this as completed in eclipse-omr/omr#5751 Jan 23, 2021

bragaigor mentioned this issue Jan 23, 2021

Add extra U_64 data field to Indexable Object Headers #11755

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DaaLoadTest ConcurrentScavenge segfault or assert #11616

DaaLoadTest ConcurrentScavenge segfault or assert #11616

pshipton commented Jan 11, 2021

dmitripivkine commented Jan 11, 2021

pshipton commented Jan 11, 2021

pshipton commented Jan 11, 2021

dmitripivkine commented Jan 11, 2021

bragaigor commented Jan 11, 2021

dmitripivkine commented Jan 11, 2021

fjeremic commented Jan 11, 2021

pshipton commented Jan 11, 2021

andrew-m-leonard commented Jan 13, 2021 •

edited

Loading

bragaigor commented Jan 14, 2021 •

edited

Loading

andrew-m-leonard commented Jan 14, 2021

andrew-m-leonard commented Jan 14, 2021

pshipton commented Jan 14, 2021 •

edited

Loading

pshipton commented Jan 14, 2021

pshipton commented Jan 14, 2021

pshipton commented Jan 14, 2021 •

edited

Loading

pshipton commented Jan 15, 2021

pshipton commented Jan 15, 2021

pshipton commented Jan 15, 2021

bragaigor commented Jan 15, 2021 •

edited

Loading

dmitripivkine commented Jan 15, 2021

DaaLoadTest ConcurrentScavenge segfault or assert #11616

DaaLoadTest ConcurrentScavenge segfault or assert #11616

Comments

pshipton commented Jan 11, 2021

dmitripivkine commented Jan 11, 2021

pshipton commented Jan 11, 2021

pshipton commented Jan 11, 2021

dmitripivkine commented Jan 11, 2021

bragaigor commented Jan 11, 2021

dmitripivkine commented Jan 11, 2021

fjeremic commented Jan 11, 2021

pshipton commented Jan 11, 2021

andrew-m-leonard commented Jan 13, 2021 • edited Loading

bragaigor commented Jan 14, 2021 • edited Loading

andrew-m-leonard commented Jan 14, 2021

andrew-m-leonard commented Jan 14, 2021

pshipton commented Jan 14, 2021 • edited Loading

pshipton commented Jan 14, 2021

pshipton commented Jan 14, 2021

pshipton commented Jan 14, 2021 • edited Loading

pshipton commented Jan 15, 2021

pshipton commented Jan 15, 2021

pshipton commented Jan 15, 2021

bragaigor commented Jan 15, 2021 • edited Loading

dmitripivkine commented Jan 15, 2021

andrew-m-leonard commented Jan 13, 2021 •

edited

Loading

bragaigor commented Jan 14, 2021 •

edited

Loading

pshipton commented Jan 14, 2021 •

edited

Loading

pshipton commented Jan 14, 2021 •

edited

Loading

bragaigor commented Jan 15, 2021 •

edited

Loading