TestUpcallAsync and TestAsyncStackWalk.java#default_gc test cases fail on AIX JDK22+

[smlambert]

As described in #5156 (comment), 2 jdk_foreign test cases crash/fail when run on AIX for JDK22 and JDK23. Possibly introduced by this upstream commit.

Failing testcases:
java/foreign/TestUpcallAsync.java
java/foreign/stackwalk/TestAsyncStackWalk.java#default_gc

jdk_foreign 2 testcase failures (rerun in Grinder/9213) fails, Deep history shows these tests passing on test-osuosl-aix72-ppc64-6, so run them there in Grinder/9217 - same failures below, TestAsyncStackWalk crashes

TEST: java/foreign/TestUpcallAsync.java (crashes)

13:10:04  STDOUT:
13:10:04  NativeTestHelper::DEFAULT_RANDOM.seed = -1973767948
13:10:04  Re-run with '-DNativeTestHelper.DEFAULT_RANDOM.seed=-1973767948' to reproduce
13:10:04  #
13:10:04  # A fatal error has been detected by the Java Runtime Environment:
13:10:04  #
13:10:04  #  SIGSEGV (0xb) at pc=0x0900000012ab2f48, pid=7078170, tid=10281
13:10:04  #
13:10:04  # JRE version: OpenJDK Runtime Environment Temurin-22+36 (22.0+36) (build 22+36)
13:10:04  # Java VM: OpenJDK 64-Bit Server VM Temurin-22+36 (22+36, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, aix-ppc64)
13:10:04  # Problematic frame:
13:10:04  # V  [libjvm.so+0xba1f48]  __srterm__0@AF67_2+0x8
13:10:04  #
13:10:04  # Core dump will be written. Default location: /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109545727976/jdk_custom_0/work/scratch/core or core.7078170
13:10:04  #
13:10:04  # An error report file with more information is saved as:
13:10:04  # /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109545727976/jdk_custom_0/work/scratch/hs_err_pid7078170.log
13:10:04  #
13:10:04  # If you would like to submit a bug report, please visit:
13:10:04  #   https://github.com/adoptium/adoptium-support/issues
13:10:04  # The crash happened outside the Java Virtual Machine in native code.
13:10:04  # See problematic frame for where to report the bug.
13:10:04  #
13:10:04  STDERR:
13:10:04  rerun:
13:10:04  cd /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109545727976/jdk_custom_0/work/scratch && \
13:10:04  DISPLAY=unix:0 \
13:10:04  HOME=/home/jenkins \
13:10:04  LANG=en_US \
13:10:04  PATH=/bin:/usr/bin:/usr/sbin \
13:10:04  TZ=Etc/UTC \
13:10:04  CLASSPATH=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109545727976/jdk_custom_0/work/classes/java/foreign/TestUpcallAsync.d:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/foreign:/home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/testng-7.3.0.jar:/home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/guice-5.1.0.jar:/home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/jcommander-1.82.jar:/home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Grinder/jvmtest/openjdk/jtreg/lib/jtreg.jar \
13:10:04      /home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/java \
13:10:04          -Dtest.vm.opts='-ea -esa -Xmx512m' \
13:10:04          -Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m' \
13:10:04          -Dtest.compiler.opts= \
13:10:04          -Dtest.java.opts= \
13:10:04          -Dtest.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \
13:10:04          -Dcompile.jdk=/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image \
13:10:04          -Dtest.timeout.factor=8.0 \
13:10:04          -Dtest.nativepath=/home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native \
13:10:04          -Dtest.root=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk \
13:10:04          -Dtest.name=java/foreign/TestUpcallAsync.java \
13:10:04          -Dtest.file=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/foreign/TestUpcallAsync.java \
13:10:04          -Dtest.src=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/foreign \
13:10:04          -Dtest.src.path=/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/foreign \
13:10:04          -Dtest.classes=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109545727976/jdk_custom_0/work/classes/java/foreign/TestUpcallAsync.d \
13:10:04          -Dtest.class.path=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109545727976/jdk_custom_0/work/classes/java/foreign/TestUpcallAsync.d \
13:10:04          -Dtest.class.path.prefix=/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109545727976/jdk_custom_0/work/classes/java/foreign/TestUpcallAsync.d:/home/jenkins/workspace/Grinder/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/foreign \
13:10:04          -Dtest.modules=java.base/jdk.internal.foreign \
13:10:04          --add-modules java.base \
13:10:04          --add-exports java.base/jdk.internal.foreign=ALL-UNNAMED \
13:10:04          -ea \
13:10:04          -esa \
13:10:04          -Xmx512m \
13:10:04          -Djava.library.path=/home/jenkins/workspace/Grinder/jdkbinary/openjdk-test-image/jdk/jtreg/native \
13:10:04          -XX:+IgnoreUnrecognizedVMOptions \
13:10:04          -XX:-VerifyDependencies \
13:10:04          --enable-native-access=ALL-UNNAMED \
13:10:04          -Dgenerator.sample.factor=17 \
13:10:04          com.sun.javatest.regtest.agent.MainWrapper /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109545727976/jdk_custom_0/work/java/foreign/TestUpcallAsync.d/testng.0.jta java/foreign/TestUpcallAsync.java false TestUpcallAsync
13:10:04  
13:10:04  TEST RESULT: Failed. Unexpected exit from test [exit code: 134]

TEST: java/foreign/stackwalk/TestAsyncStackWalk.java#default_gc (crashes)

[WhiteBox::VerifyFrames] Walking Frames
Native frame (sp=0x0000000115e75230 unextended sp=0x0000000115e75230, fp=0x0000000115e752f0, real_fp=0x0000000115e752f0, pc=0x0a00010007be32f4)
Compiled frame (sp=0x0000000115e752f0 unextended sp=0x0000000115e752f0, fp=0x0000000115e75360, real_fp=0x0000000115e75360, pc=0x0a000100001d345c)
Compiled frame (sp=0x0000000115e75360 unextended sp=0x0000000115e75360, fp=0x0000000115e753d0, real_fp=0x0000000115e753d0, pc=0x0a000100001d2e14)
Compiled frame (sp=0x0000000115e753d0 unextended sp=0x0000000115e753d0, fp=0x0000000115e75410, real_fp=0x0000000115e75410, pc=0x0a00010007be2174)
Compiled frame (sp=0x0000000115e75410 unextended sp=0x0000000115e75410, fp=0x0000000115e75480, real_fp=0x0000000115e75480, pc=0x0a000100001d2610)
Compiled frame (sp=0x0000000115e75480 unextended sp=0x0000000115e75480, fp=0x0000000115e754d0, real_fp=0x0000000115e754d0, pc=0x0a00010007be19c8)
C frame (sp=0x0000000115e754d0 unextended sp=0x0000000115e754d0, fp=0x0000000115e75690, real_fp=0x0000000115e75690, pc=0x0a00010007583b3c)
[WhiteBox::VerifyFrames] Done
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0900000012e5ff48, pid=30409102, tid=4884
#
# JRE version: OpenJDK Runtime Environment Temurin-22+36 (22.0+36) (build 22+36)
# Java VM: OpenJDK 64-Bit Server VM Temurin-22+36 (22+36, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, aix-ppc64)
# Problematic frame:
# V  [libjvm.so+0xba1f48]  __srterm__0@AF67_2+0x8
#
# Core dump will be written. Default location: /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109075623225/jdk_custom_0/work/scratch/core or core.30409102
#
Unsupported internal testing APIs have been used.

# An error report file with more information is saved as:
# /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_17109075623225/jdk_custom_0/work/scratch/hs_err_pid30409102.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

Deep history shows the 2 jdk_foreign test cases were last passing 10/16/2023 and first seen failing 11/06/2023 (ACTION: check what commits were introduced in between those dates).

This commit seems 'suspect' openjdk/jdk@3885dc5

[smlambert]

Also seen consistently in JDK23 runs of the jdk_foreign test target.

[suchismith1993]

I was able to reproduce the issue by using xlc 16.1. By switching over to openxlc 17.1.0 the tests are passing.
Would be good to upgrade the setup to support 17.1.0 .
I am trying to do a rca of the core dump in the meanwhile.

[sxa]

Now passes on JDK22 with the build from the AIX box using the OpenXL17 compiler as implemented in adoptium/infrastructure#3606 (Although sanity.openjdk has had some green runs

• https://ci.adoptium.net/job/Test_openjdk22_hs_sanity.openjdk_ppc64_aix_testList_0/5/testReport/java_foreign_TestUpcallAsync/java/
• https://ci.adoptium.net/job/Test_openjdk22_hs_sanity.openjdk_ppc64_aix_testList_0/5/testReport/java_foreign_stackwalk_TestAsyncStackWalk/

100 iterations in Grinder just to be sure: https://ci.adoptium.net/job/Grinder/10573/console

LGTM ✅ 👍🏻 - Closing. Thanks @suchismith1993!