CI: The Job LPython CI / Check Release build (pull_request) seems flaky

[Shaikh-Ubaid]

It seems to falsely fail for me on first run. It seems to pass on second run attempt.

False Failures:

• https://github.com/lcompilers/lpython/actions/runs/5564532214/attempts/1
• https://github.com/lcompilers/lpython/actions/runs/5546663093/attempts/1
• https://github.com/lcompilers/lpython/actions/runs/5563369118/attempts/1
• https://github.com/lcompilers/lpython/actions/runs/5546663093/attempts/1

[Smit-create]

Can you describe the exact issue? or point to failure? Rarely, it fails because of some issues on the GH actions side.

[Shaikh-Ubaid]

Can you describe the exact issue? or point to failure? Rarely, it fails because of some issues on the GH actions side.

I updated the description of this issue.

[Shaikh-Ubaid]

The job Check Release build passed in the third run attempt. Something does not seem right. I will explore more about this.

[certik]

The failure seems different though:

make[2]: *** [CMakeFiles/modules_02_FAST.dir/build.make:74: modules_02_FAST.o] Illegal instruction (core dumped)
make[1]: *** [CMakeFiles/Makefile2:2838: CMakeFiles/modules_02_FAST.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

and

Test project /home/runner/work/lpython/lpython
    Start 1: test_stacktrace
1/2 Test #1: test_stacktrace ..................   Passed    0.00 sec
    Start 2: test_lpython
2/2 Test #2: test_lpython .....................***Exception: Illegal  0.13 sec

and (3x):

make[2]: *** [CMakeFiles/array_03_decl_FAST.dir/build.make:74: array_03_decl_FAST.o] Illegal instruction (core dumped)
make[1]: *** [CMakeFiles/Makefile2:568: CMakeFiles/array_03_decl_FAST.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

But all three failures seem to be caused by the lpython compiler executing some illegal instruction. That typically happens with the Release mode is compiled to a newer architecture than it runs, quite common at a CI. I think we just need to lower the optimization options, perhaps keep -O3, but remove -march=native or something like that.

[Shaikh-Ubaid]

I think the original failure is

2/2 Test #2: test_lpython .....................***Exception: Illegal  0.13 sec

This was previously not being caught because bash would not fail even if there was a failure. Therefore, the CI was failing at a later point when running reference tests.

Since #2194, the above 2/2 Test #2: test_lpython .....................***Exception: Illegal 0.13 sec is always caught now and appropriately fails.

[Shaikh-Ubaid]

The release build seems to fail too often. I request review of #2212 at the earliest convenience. If in case it does not fix the issue, we can try/find other approaches.

[Shaikh-Ubaid]

This issues seems to be fixed. It does not seem to be flaky now.