[BUG] rv-virt/citest: test_hello or test_pipe failed

[lupyuen]

Description / Steps to reproduce the issue

Since yesterday: rv-virt/citest has been failing test_hello onwards, or test_pipe onwards, hanging our CI Checks in GitHub and Build Farm. (GitHub will cancel it after 6 hours)

It might have been caused by one of these NuttX Commits:

• arch/toochain: Add toochain to gcc
• mps3-an547:ap: Disable cmd_wait() as it depends on procfs currently
• Earlier commits were OK

Or maybe one of these NuttX Apps Commits:

• nshlib/pipeline: Concat variable arguments failed
• nshlib/nsh_parse: Fix variable arguments concat error of nsh_execute()
• Earlier commits were OK

Also when one test fails: Why do the rest of the tests take a loooong time to fail, hanging our CI Checks in GitHub and Build Farm?

Fail at test_hello onwards: https://github.com/NuttX/nuttx/actions/runs/11833005280/job/32970891697#step:7:143

Configuration/Tool: rv-virt/citest
$ cd /github/workspace/sources/nuttx/tools/ci/testrun/script
$ python3 -m pytest -m 'qemu or rv_virt' ./ -B rv-virt -P /github/workspace/sources/nuttx -L /github/workspace/sources/nuttx/boards/risc-v/qemu-rv/rv-virt/configs/citest/logs/rv-virt/qemu -R qemu -C --json=/github/workspace/sources/nuttx/boards/risc-v/qemu-rv/rv-virt/configs/citest/logs/rv-virt/qemu/pytest.json

test_framework/test_cmocka.py::test_cmocka PASSED                        [  0%]
test_example/test_example.py::test_hello FAILED                          [  0%]
test_example/test_example.py::test_helloxx FAILED                        [  0%]
test_example/test_example.py::test_pipe FAILED                           [  0%]
test_example/test_example.py::test_popen FAILED                          [  0%]
test_example/test_example.py::test_usrsocktest FAILED                    [  0%]
[ Everything fails very slowly ]

Fail at test_pipe onwards: https://github.com/NuttX/nuttx/actions/runs/11850442831/job/33025374105#step:7:145

test_framework/test_cmocka.py::test_cmocka PASSED                        [  0%]
test_example/test_example.py::test_hello PASSED                          [  0%]
test_example/test_example.py::test_helloxx PASSED                        [  0%]
test_example/test_example.py::test_pipe FAILED                           [  0%]
test_example/test_example.py::test_popen FAILED                          [  0%]
test_example/test_example.py::test_usrsocktest FAILED                    [  0%]
[ Everything fails very slowly ]

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Ubuntu LTS at GitHub Actions

NuttX Version

master

Issue Architecture

[Arch: risc-v]

Issue Area

[Area: Build System]

Verification

• I have verified before submitting the report.

[lupyuen]

The Timeout Values are configured to One Minute or longer for some Python Tests. What if we reduce the Timeout Values? https://github.com/search?q=repo%3Aapache%2Fnuttx+timeout%3D+language%3APython+path%3A%2F%5Etools%5C%2Fci%5C%2Ftestrun%5C%2F%2F&type=code

Update: Nope, doesn't work: https://github.com/lupyuen/nuttx-build-farm/blob/main/run-job-macos.sh#L107-L131

Somehow the Timeout Value is hard-coded inside expect? https://github.com/apache/nuttx/blob/master/tools/ci/testrun/utils/common.py#L229-L288

[lupyuen]

For now we patched the NuttX Mirror Repo: Kill the CI Test if it exceeds 2 hours. Also for Ubuntu Build Farm and macOS Build Farm.

• [FEATURE] CI Test should always terminate after 1 hour #14680 (comment)