api.static_noclient test fails with assert on jenkins

[derekbruening]

On Jenkins when I tried to enable it in the suite in PR #4470 it failed:
http://139.178.83.194:8080/job/DynamoRIO-AArch64-Precommit/1723/console

202: api.static_noclient: /var/lib/jenkins/.jenkins/workspace/DynamoRIO-AArch64-Precommit/suite/tests/api/static_noclient.c:89: test_static_decode_before_attach: Assertion `res == 0 && memcmp(&mask, &check_mask, sizeof(mask)) == 0' failed.

But it works 1000x in a row on tx1:

derek@tx1:~/dr/build$ for ((i=0; i<1000; i++)); do echo -e "\n============\n$i\n"; LD_LIBRARY_PATH=lib64/debug DYNAMORIO_OPTIONS="-stderr_mask 0xf -dumpcore_mask 0 -code_api" suite/tests/bin/api.static_noclient; done > OUT 2>&1
derek@tx1:~/dr/build$ grep Assert OUT
derek@tx1:~/dr/build$ grep -c 'all done' OUT
1000

[derekbruening]

I split this off from #1578

[derekbruening]

@AssadHashmi since it does not reproduce on the tx1 but does on Jenkins I'm hoping you can take a look

[derekbruening]

Xref #4474

[AssadHashmi]

I can't replicate the failure manually on the Jenkins machine: an Ampere eMAG 8180 running Debian GNU/Linux 9 (stretch):

assad@ampere:~/dr_i4716/dynamorio/build$ for ((i=0; i<1000; i++)); do echo -e "\n============\n$i\n"; LD_LIBRARY_PATH=lib64/debug DYNAMORIO_OPTIONS="-stderr_mask 0xf -dumpcore_mask 0 -code_api" suite/tests/bin/api.static_noclient; done > OUT 2>&1
assad@ampere:~/dr_i4716/dynamorio/build$ grep Assert OUT
assad@ampere:~/dr_i4716/dynamorio/build$ grep -i Assert OUT
assad@ampere:~/dr_i4716/dynamorio/build$ grep -c 'all done' OUT
1000
assad@ampere:~/dr_i4716/dynamorio/build$

@derekbruening have I missed something in the build and run process? I built and ran with:

cmake -DINTERNAL=ON -DDEBUG=ON -DBUILD_TESTS=ON ..
make
make test

Enabled api.static_noclient test on HEAD SHA c05ee23:

assad@ampere:~/dr_i4716/dynamorio$ git log --abbrev-commit --pretty=oneline | head -3
c05ee234 i#2666 dyn cxt: Fix early context free (#4722)
9be64f00 i#4717: Fix hang in api.static_prepop test on aarchxx (#4721)
795c8498 i#4712: Fix doxygen 1.9.1 build errors (#4713)
assad@ampere:~/dr_i4716/dynamorio$

assad@ampere:~/dr_i4716/dynamorio$ git diff
diff --git a/suite/runsuite_wrapper.pl b/suite/runsuite_wrapper.pl
index ee61581a..5145052b 100755
--- a/suite/runsuite_wrapper.pl
+++ b/suite/runsuite_wrapper.pl
@@ -259,7 +259,6 @@ for (my $i = 0; $i <= $#lines; ++$i) {
                 'code_api|client.nudge_ex' => 1,
                 'code_api|client.alloc-noreset' => 1, # i#4436
                 'code_api|api.detach_spawn' => 1, # i#2611
-                'code_api|api.static_noclient' => 1,
                 'code_api|api.static_noinit' => 1,
                 );
             $issue_no = "#2145";
diff --git a/suite/tests/CMakeLists.txt b/suite/tests/CMakeLists.txt
index 3f5cbc11..32e18de5 100644
--- a/suite/tests/CMakeLists.txt
+++ b/suite/tests/CMakeLists.txt
@@ -2942,10 +2942,8 @@ if (CLIENT_INTERFACE)
   if (NOT ANDROID AND NOT APPLE)
     tobuild_api(api.static_startstop api/static_startstop.c "" "" OFF ON)
     target_link_libraries(api.static_startstop ${libmath})
-    if (NOT AARCHXX) # TODO i#1578: Failing on AArch64.
-      tobuild_api(api.static_noclient api/static_noclient.c "" "" OFF ON)
-      target_link_libraries(api.static_noclient ${libmath})
-    endif ()
+    tobuild_api(api.static_noclient api/static_noclient.c "" "" OFF ON)
+    target_link_libraries(api.static_noclient ${libmath})
     tobuild_api(api.static_noinit api/static_noinit.c "" "" OFF ON)
     target_link_libraries(api.static_noinit ${libmath})
     tobuild_api(api.static_detach api/static_detach.c "" "" OFF ON)
assad@ampere:~/dr_i4716/dynamorio$

[derekbruening]

The Jenkins failure cited was back in Oct. Maybe one of the many fixes we have done since then addressed it? Maybe not worth the time to check out the old code and confirm: just move on? Maybe you could just commit your diff there enabling the test?

[AssadHashmi]

Enabled api.static_noclient in PR #4727 which fails AArch64 tests with:

219: api.static_noclient: /var/lib/jenkins/.jenkins/workspace/DynamoRIO-AArch64-
Precommit/suite/tests/api/static_noclient.c:89: test_static_decode_before_attach: Assertion
`res == 0 && memcmp(&mask, &check_mask, sizeof(mask)) == 0' failed.

There must be something different in the build and/or runtime environment between the manual test above and automated tests.

[AssadHashmi]

Fixed by #5470.