Failures occurred, TAP results and TEST TARGETS SUMMARY don't report failures

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_aarch64_mac_Nightly/44/

jenkins (correctly) reports unstable. There is a Test Result reporting failures.

Test Result (6 failures / +6)

    jdk/incubator/vector/Byte128VectorTests.java.Byte128VectorTests
    jdk/incubator/vector/Double128VectorTests.java.Double128VectorTests
    jdk/incubator/vector/Float128VectorTests.java.Float128VectorTests
    jdk/incubator/vector/Int128VectorTests.java.Int128VectorTests
    jdk/incubator/vector/Long128VectorTests.java.Long128VectorTests
    jdk/incubator/vector/Short128VectorTests.java.Short128VectorTests

The TAP results don't report any failures, and TEST TARGETS SUMMARY shows ALL TESTS PASSED

21:48:27  TEST TARGETS SUMMARY
21:48:27  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
21:48:27  DISABLED test targets:
21:48:27  	jdk_foreign_native_0
21:48:27  	jdk_foreign_native_1
21:48:27  	jdk_foreign_native_2
21:48:27  	jdk_svc_sanity_0
21:48:27  	jdk_svc_sanity_1
21:48:27  	jdk_svc_sanity_2
21:48:27  
21:48:27  PASSED test targets:
21:48:27  	jdk_lang_1 
21:48:27  	jdk_lang_native_1 
21:48:27  	jdk_math_1 
21:48:27  	jdk_util_1 
21:48:27  	jdk_foreign_1 
21:48:27  	jdk11_tier1_buffer_1 
21:48:27  	jdk11_tier1_iso8859_1 
21:48:27  	jdk_vector_float128_j9_0 
21:48:27  	jdk_vector_double128_j9_0 
21:48:27  	jdk_vector_int128_j9_0 
21:48:27  	jdk_vector_short128_j9_0 
21:48:27  	jdk_vector_byte128_j9_0 
21:48:27  	jdk_vector_long128_j9_0 
21:48:27  	jdk_vector_byte64_j9_0 
21:48:27  
21:48:27  TOTAL: 37   EXECUTED: 14   PASSED: 14   FAILED: 0   DISABLED: 6   SKIPPED: 17
21:48:27  ALL TESTS PASSED

@llxia

[pshipton]

I've also got some grinders that don't show any failures, but there are tests failing in the grinders.

https://openj9-jenkins.osuosl.org/job/Grinder/1492/consoleFull

11:08:17  PASSED test targets:
11:08:17  	jdk_vector_byte128_j9_0 
11:08:17  
11:08:17  TOTAL: 1   EXECUTED: 1   PASSED: 1   FAILED: 0   DISABLED: 0   SKIPPED: 0
11:08:17  ALL TESTS PASSED

but

11:08:17  test Byte128VectorTests.ABSMaskedByte128VectorTests(byte[i * 5], mask[i % 2]): failure
11:08:17  java.lang.AssertionError: at index #33, input = -91, mask = false expected [-91] but found [91]
11:08:17  	at org.testng.Assert.fail(Assert.java:99)
11:08:17  	at org.testng.Assert.failNotEquals(Assert.java:1037)
11:08:17  	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
11:08:17  	at org.testng.Assert.assertEquals(Assert.java:122)
11:08:17  	at org.testng.Assert.assertEquals(Assert.java:841)
11:08:17  	at Byte128VectorTests.assertArraysEquals(Byte128VectorTests.java:109)
11:08:17  	at Byte128VectorTests.ABSMaskedByte128VectorTests(Byte128VectorTests.java:5265)

Similarly in https://openj9-jenkins.osuosl.org/job/Grinder/1493/consoleFull

[pshipton]

This problem is impacting development and the ability to diagnose problems. Grinders are not correctly reporting failures that occur, they look like they are passing when tests are really failing. It needs to be fixed ASAP.

[llxia]

The test runs multiple commands. The first one failed and the last cmd was successful. As we only capture the last cmd result, the test result is successful.

We should use && to make sure the previous cmd is completed successfully. I will have a PR to fix this issue soon.

[pshipton]

Pls also check https://openj9-jenkins.osuosl.org/job/Grinder/1494/ is the same situation.
i.e. eclipse-openj9/openj9#16259 (comment)

[llxia]

re #4115 (comment), I do not think https://openj9-jenkins.osuosl.org/job/Grinder/1494/ is the same problem as vector tests.

[babsingh]

It's the test report view: https://openj9-jenkins.osuosl.org/job/Grinder/1494/testReport/. Example: PinALot.java#id0.PinALot_id0 passed but shows up as failed.

[llxia]

@babsingh are you referring to the following case that it is in the failed test section, but showed passed?

This is a different issue than vector test above. In this case, a 3rd party Jenkins JUnit plugin reads jtr files that are generated from openjdk tests. We do not do any modifications to the test report. If you refresh Jenkins test report page, the test link change from time to time (not all the time).
I got the following links for PinALot.java#id0.PinALot_id0:
https://openj9-jenkins.osuosl.org/job/Grinder/1494/testReport/java_lang_Thread_virtual_stress_PinALot/java_id0/PinALot_id0/
https://openj9-jenkins.osuosl.org/job/Grinder/1494/testReport/java_lang_Thread_virtual_stress_PinALot/java_id0/PinALot_id0_2/
https://openj9-jenkins.osuosl.org/job/Grinder/1494/testReport/java_lang_Thread_virtual_stress_PinALot/java_id0/PinALot_id0_7/
...

Also, the content on the above test link changes from time to time (see the recording below):

Screenshare.-.2022-11-04.2_34_31.PM.mp4

Just my two cents, I think it is related to the test that was executed in iteration. As a result, it had multiple jtr files with the same test name. When the Jenkins plugin renders it, it quires the database and displays the first returned result (without order), which may or may not be the failed one. I think this is something that we should report to the Jenkins JUnit plugin repo.

cc @smlambert @renfeiw

[babsingh]

are you referring to the following case that it is in the failed test section, but showed passed?

Yes. The same issue is also seen for GetStackTraceALot.java.

For actual failures (NullStackTrace.java), I don't see the error output related to the segfault:

[llxia]

It suggested check console log for any additional details in the message. You can see full result in jtf file. For example:

aqa-tests/TKG/output_16675036039469/jdk_lang_1/work/java/lang/Thread/NullStackTrace.jtr in the Grinder Artifacts

Or you can check console output for segfault.

[babsingh]

re #4115 (comment):

I generally look at the Console Output. I didn't know console log meant Console Output. Also, Agent communication error: java.io.EOFException; suggests a failure in the test framework. So, I was under the assumption that the test framework had an error.