Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JDK21 serviceability_jvmti_j9_0_FAILED serviceability/jvmti/vthread/SuspendResume2/SuspendResume2.java#default Fatal error: test_vthread_resume_half: error in JVMTI ResumeThread #17869

Closed
JasonFengJ9 opened this issue Jul 27, 2023 · 5 comments
Closed

JDK21 serviceability_jvmti_j9_0_FAILED serviceability/jvmti/vthread/SuspendResume2/SuspendResume2.java#default Fatal error: test_vthread_resume_half: error in JVMTI ResumeThread #17869

JasonFengJ9 opened this issue Jul 27, 2023 · 5 comments
Assignees
@babsingh
Labels
comp:jvmti comp:vm jdk21 project:loom Used to track Project Loom related work test failure
Milestone
Java 21 (0.42)

Comments

@JasonFengJ9
Copy link
Member

JasonFengJ9 commented Jul 27, 2023 •
edited
Loading

Failure link

From an internal grinder build(paix906) - launched at #17865

08:08:05  openjdk version "21-internal" 2023-09-19
08:08:05  OpenJDK Runtime Environment (build 21-internal-adhoc.jenkins.BuildJDK21ppc64aixPersonal)
08:08:05  Eclipse OpenJ9 VM (build master-c9a98669642, JRE 21 AIX ppc64-64-Bit Compressed References 20230726_28 (JIT enabled, AOT enabled)
08:08:05  OpenJ9   - c9a98669642
08:08:05  OMR      - 436bad3c438
08:08:05  JCL      - 78015256cbd based on jdk-21+31)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

11:54:29  variation: Mode150
11:54:29  JVM_OPTIONS:  -XX:+UseCompressedOops 

12:04:30  TEST: serviceability/jvmti/vthread/SuspendResume2/SuspendResume2.java#default

12:04:30  STDERR:
12:04:30  
12:04:30  Fatal error: test_vthread_resume_half: error in JVMTI ResumeThread

12:04:30  --------------------------------------------------
12:04:32  Test results: passed: 121; failed: 2
12:04:56  Report written to /home/jenkins/workspace/Grinder_iteration_2/jvmtest/openjdk/report/html/report.html
12:04:56  Results written to /home/jenkins/workspace/Grinder_iteration_2/aqa-tests/TKG/output_16904731027876/serviceability_jvmti_j9_0/work
12:04:56  Error: Some tests failed or other problems occurred.
12:04:56  -----------------------------------
12:04:56  serviceability_jvmti_j9_0_FAILED

50x internal grinder - 1/50 failing rate

https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder_iteration_3/2017/consoleFull

11:41:45  TEST: serviceability/jvmti/vthread/SuspendResumeAll/SuspendResumeAll.java#default

11:41:45  ## Java: runIt FAILED: status from native Agent: 2
11:41:45  STDERR:
11:41:45  java.lang.RuntimeException: FAILED: unexpected status: 2
11:41:45  	at SuspendResumeAll.main(SuspendResumeAll.java:78)
11:41:45  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
11:41:45  	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
11:41:45  	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
11:41:45  	at java.base/java.lang.Thread.run(Thread.java:1595)
11:41:45  
11:41:45  JavaTest Message: Test threw exception: java.lang.RuntimeException: FAILED: unexpected status: 2

12:03:30  TEST: serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

12:03:30  TEST RESULT: Error. Program `/home/jenkins/workspace/Grinder_iteration_3/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 961851ms).

FYI @babsingh
@JasonFengJ9 JasonFengJ9 added this to the Java 21 milestone Jul 27, 2023
@fengxue-IS fengxue-IS added the project:loom Used to track Project Loom related work label Sep 16, 2023
@babsingh
Copy link
Contributor

Moving to the 0.44 release since the failure is highly intermittent.

@JasonFengJ9
Copy link
Member Author

JDK21 x86-64_mac(mac11x64rt04)

java version "21.0.1-beta" 2023-10-17
IBM Semeru Runtime Certified Edition 21.0.1+12-202310261403 (build 21.0.1-beta+12-202310261403)
Eclipse OpenJ9 VM 21.0.1+12-202310261403 (build master-66e3fd1da, JRE 21 Mac OS X amd64-64-Bit Compressed References 20231026_22 (JIT enabled, AOT enabled)
OpenJ9   - 66e3fd1da
OMR      - 071c0c25b
JCL      - c06eaf638 based on jdk-21.0.1+12)

[2023-10-26T15:00:50.188Z] variation: Mode650
[2023-10-26T15:00:50.188Z] JVM_OPTIONS:  -XX:-UseCompressedOops 

[2023-10-26T15:03:11.620Z] TEST: serviceability/jvmti/vthread/SuspendResumeAll/SuspendResumeAll.java#no-vmcontinuations

[2023-10-26T15:03:11.621Z] STDERR:
[2023-10-26T15:03:11.621Z] java.lang.RuntimeException: FAILED: unexpected status: 2
[2023-10-26T15:03:11.621Z] 	at SuspendResumeAll.main(SuspendResumeAll.java:78)
[2023-10-26T15:03:11.621Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2023-10-26T15:03:11.621Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2023-10-26T15:03:11.621Z] 	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
[2023-10-26T15:03:11.621Z] 	at java.base/java.lang.Thread.run(Thread.java:1595)

[2023-10-26T15:03:30.804Z] serviceability_jvmti_j9_1_FAILED

@babsingh
Copy link
Contributor

babsingh commented Oct 28, 2023 •
edited
Loading

Moving this issue back to the 0.42 release due to the new failure.

@babsingh babsingh mentioned this issue Oct 28, 2023
Closed
@babsingh
Copy link
Contributor

re #17869 (comment): This is a new failure, which is seen for the first time. Opened #18370 for it.

babsingh added a commit to babsingh/openj9 that referenced this issue Nov 8, 2023
@babsingh
Re-fetch objects after VM access is released and reacquired
cb4f50b 
Stale objects can manifest many issues. In this case, stale objects
cause complex and intermittent synchronization issues.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
This was referenced Nov 8, 2023
Merged
Merged
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 9, 2023
@babsingh
Re-fetch objects after VM access is released and reacquired
a1cdacf 
Stale objects can manifest many issues. In this case, stale objects
cause complex and intermittent synchronization issues.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
@babsingh babsingh mentioned this issue Nov 10, 2023
Merged
@babsingh babsingh mentioned this issue Nov 13, 2023
Closed
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 13, 2023
@babsingh
Spin during VirtualThread MountBegin and UnmountBegin
feac356 
Currently, the halt flag is set in VirtualThread MountEnd if a virtual
thread is suspended via JVMTI, and in VirtualThread UnmountEnd if a
carrier thread is suspended via JVMTI.

In the above approach, the halt flag is set too late. As soon as the
continuation swaps the J9VMThread context, the thread begins execution
and is capable of triggering JVMTI events.

To avoid the above issue, the above steps are moved into VirtualThread
MountBegin and UnmountBegin. This prevents the continuation to swap
the J9VMThread context.

Currently, the halt flag is set without invoking
exitVThreadTransitionCritical. This prevents JVMTI to resume the halted
thread and cause a hang. The new approach spins, invokes
exitVThreadTransitionCritical  and releases VM access to allow JVMTI to
resume the suspended thread.

The better approach will be to fail mount if the thread is suspended
and retry later. Currently, his approach cannot be implemented because
VirtualThread.java does not support this approach.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 14, 2023
@babsingh
Update the fully mounted check in JVMTI Suspend/Resume Thread
1cd50dd 
During JVMTI suspend/resume thread operations, the current check to
determine if a thread is fully mounted is flaky. It depends on
J9VMThread->theadObject which might change after getVMThread due to
Java level changes, which are accepted as-is from the RI.

The check to determine if a thread is fully mounted is moved into
getVMThread. The check is updated to use the continuation state, which
is more stable since the Continuation implementation is maintained
within OpenJ9.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
@babsingh babsingh mentioned this issue Nov 15, 2023
Merged
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 16, 2023
@babsingh @fengxue-IS
Add JVMTI synchronization in JVM_VirtualThreadHideFrames
8a2b61e 
J9VMThread->threadObject can be modified between
JVM_VirtualThreadHideFrames's first invocation with hide=true and
second invocation with hide=false. Synchronization to prevent JVMTI
operations is acquired between these two invocations. This prevents
JVMTI functions to see an unstable J9VMThread->threadObject.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Co-authored-by: Jack Lu <Jack.S.Lu@ibm.com>
Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
@babsingh
Copy link
Contributor

Fixed via

babsingh added a commit to babsingh/openj9 that referenced this issue Nov 20, 2023
@babsingh @fengxue-IS
Add JVMTI synchronization in JVM_VirtualThreadHideFrames
38907fd 
J9VMThread->threadObject can be modified between
JVM_VirtualThreadHideFrames's first invocation with hide=true and
second invocation with hide=false. Synchronization to prevent JVMTI
operations is acquired between these two invocations. This prevents
JVMTI functions to see an unstable J9VMThread->threadObject.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Co-authored-by: Jack Lu <Jack.S.Lu@ibm.com>
Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
babsingh added a commit to babsingh/openj9 that referenced this issue Nov 20, 2023
@babsingh @fengxue-IS
Add JVMTI synchronization in JVM_VirtualThreadHideFrames
ecdbc7b 
J9VMThread->threadObject can be modified between
JVM_VirtualThreadHideFrames's first invocation with hide=true and
second invocation with hide=false. Synchronization to prevent JVMTI
operations is acquired between these two invocations. This prevents
JVMTI functions to see an unstable J9VMThread->threadObject.

Related: eclipse-openj9#17865
Related: eclipse-openj9#17869
Related: eclipse-openj9#18370

Co-authored-by: Jack Lu <Jack.S.Lu@ibm.com>
Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@babsingh babsingh

Labels
comp:jvmti comp:vm jdk21 project:loom Used to track Project Loom related work test failure
Projects
None yet
Milestone
Java 21 (0.42)
Development

No branches or pull requests
3 participants
@babsingh @fengxue-IS @JasonFengJ9