[BUG] ostest fails without returning the expected exit code

[tmedicci]

Description / Steps to reproduce the issue

In addition to the errors reported at #14722 (comment), there is an even critical issue regarding ostest for ESP32-S3 (and, eventually, for other archs): it fails without returning any error code after applying this PR.

How to Reproduce

Build/flash NuttX f2aeb5e56ff6ec7ba15958403289e462a8c84ab8 with Apps 0c467dc02d1f03f3f9f3defb16f36cb4f53b4c9d, with:

make -j distclean && ./tools/configure.sh esp32s3-devkit:smp && make flash EXTRAFLAGS="-Wno-cpp -Werror" ESPTOOL_BINDIR=./ ESPTOOL_PORT=/dev/ttyUSB0 -s -j$(nproc)

And run ostest :

signest_test: Simple case:
  Total signalled 1240  Odd=620 Even=620
  Total handled   1240  Odd=620 Even=620
  Total nested    3    Odd=3   Even=0  
signest_test: With task locking
  Total signalled 2480  Odd=1240 Even=1240
  Total handled   2480  Odd=1240 Even=1240
  Total nested    3    Odd=3   Even=0  
signest_test: With intefering thread
[CPU0] dump_assert_info: Current Version: NuttX  10.4.0 f2aeb5e56f-dirty Nov 12 2024 16:46:29 xtensa
[CPU0] dump_assert_info: Assertion failed (_Bool)0: at file: signest.c:555 task(CPU0): ostest process: ostest 0x420470
[CPU0] up_dump_register:    PC: 4204767e    PS: 00060322
[CPU0] up_dump_register:    A0: 80377b9d    A1: 3fc942e0    A2: 00000000    A3: 3fc8c7a0
[CPU0] up_dump_register:    A4: 20000000    A5: 3fc8b284    A6: 00000004    A7: 00000000
[CPU0] up_dump_register:    A8: 3fc8c778    A9: 3fc942d0   A10: 3fc8c778   A11: 00000000
[CPU0] up_dump_register:   A12: 3c0104bb   A13: 3fc943d0   A14: 00000004   A15: 00000000
[CPU0] up_dump_register:   SAR: 00000020 CAUSE: 3fc941d0 VADDR: 3fc941d0
[CPU0] up_dump_register:  LBEG: 40056f08  LEND: 40056f12  LCNT: 00000000
[CPU0] dump_stackinfo: User Stack:
[CPU0] dump_stackinfo:   base: 0x3fc92530
[CPU0] dump_stackinfo:   size: 00008112
[CPU0] dump_stackinfo:     sp: 0x3fc942e0
[CPU0] stack_dump: 0x3fc942c0: 82035d11 3fc94300 3c01a9cf 3fc92070 00001fb0 3fc944e0 3c01a9cf 3fc92070
[CPU0] stack_dump: 0x3fc942e0: 3fc8b3dc 3fc942e0 00000000 00000000 8204a960 3fc943f0 3c01a9cf 0000022b
[CPU0] stack_dump: 0x3fc94300: 00000000 3fc9211c 3fc9211c 42047f20 7474754e 3fc90058 00060a22 3fc92070
[CPU0] stack_dump: 0x3fc94320: 3fc943f0 3fc900d0 00000004 00000000 82050da2 3fc94370 3fc92240 3c01ad51
[CPU0] stack_dump: 0x3fc94340: 3fc94390 2e303100 00302e34 3fc8b288 3fc943f0 3fc943d0 32660004 35626561
[CPU0] stack_dump: 0x3fc94360: 66363565 7269642d 4e207974 3120766f 30322032 31203432 36343a36 0039323a
[CPU0] stack_dump: 0x3fc94380: 3fc92240 3fc943c0 00000025 65747840 0061736e 3fc943d0 00000004 00000004
[CPU0] stack_dump: 0x3fc943a0: 3fc943f0 3fc92070 3fc8c7a0 3c010a10 3c01a9cf 0000022b 00000003 00000744
[CPU0] stack_dump: 0x3fc943c0: 00000006 00060320 0000022b 00000000 3c010a10 3fc8c7a0 3c015f54 42047f20
[CPU0] stack_dump: 0x3fc943e0: 820481fe 3fc94410 00000003 00000744 3c010a10 00000000 00000003 0000026c
[CPU0] stack_dump: 0x3fc94400: 82036061 3fc94460 00000000 3fc92504 00010066 00000000 00000000 00002000
[CPU0] stack_dump: 0x3fc94420: 00000036 00000035 00000066 3fc92504 00000003 00058828 0000e428 0000002c
[CPU0] stack_dump: 0x3fc94440: 00000744 3fc8cc90 00000000 00000e88 82033920 3fc944a0 42047f20 00000005
[CPU0] stack_dump: 0x3fc94460: 0005fc38 00000001 0000001f 0005a948 000052f0 0005a948 00005458 00000000
[CPU0] stack_dump: 0x3fc94480: 00000004 3fc8cbdc 3c017b1a 3fc9251e 00000000 3fc944c0 00000000 42047f20
[CPU0] stack_dump: 0x3fc944a0: 3fc924f0 3fc92070 3fc8a2cc 00000002 00000000 3fc944e0 00000000 00000000
[CPU0] stack_dump: 0x3fc944c0: 00050020 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[CPU0] stack_dump: 0x3fc944e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  Total signalled 3720  Odd=1860 Even=1860
  Total handled   3720  Odd=1860 Even=1860
  Total nested    3    Odd=3   Even=0  
signest_test: ERROR 3 ODD signals were nested
ostest_main: Exiting with status 256
nsh> echo $?
0

The firmware asserted, but the return code of ostest is still 0! This assertion occurs only with this firmware version. So, there are two open questions to be investigated: why is this test asserting after this PR and why ostest returns 0 even after assertion.

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Manjaro

NuttX Version

f2aeb5e

Issue Architecture

[Arch: xtensa]

Issue Area

[Area: Applications]

Verification

• I have verified before submitting the report.

[yamt]

@tmedicci
i have a question.

ostest on esp32s3-devkit:smp doesn't work reliably for me, even with the change in question reverted.
it often hangs at seemingly random places. i have run it several times and it passed only once.

does it work better for you?

[yamt]

@tmedicci i have a question.

ostest on esp32s3-devkit:smp doesn't work reliably for me, even with the change in question reverted. it often hangs at seemingly random places. i have run it several times and it passed only once.

does it work better for you?

it seems it was because of an unrelated issue.
#14755

[tmedicci]

@tmedicci i have a question.

ostest on esp32s3-devkit:smp doesn't work reliably for me, even with the change in question reverted. it often hangs at seemingly random places. i have run it several times and it passed only once.

does it work better for you?

Hi @yamt , there were nested issues pending. I reported that ostest was failing because a recent change, but it seems to be fixed recently. Now (with #14751), the test runs successfully: apache/nuttx-apps#2737 (comment)

[yamt]

@tmedicci i have a question.
ostest on esp32s3-devkit:smp doesn't work reliably for me, even with the change in question reverted. it often hangs at seemingly random places. i have run it several times and it passed only once.
does it work better for you?

Hi @yamt , there were nested issues pending. I reported that ostest was failing because a recent change, but it seems to be fixed recently. Now (with #14751), the test runs successfully: apache/nuttx-apps#2737 (comment)

do you mean you have no idea how reliably it successes?

[yamt]

i've run the test several times, with and without an equivalent of #14722.
(i enabled assertions)
i couldn't reproduced the reported symptom.

[tmedicci]

i've run the test several times, with and without an equivalent of #14722. (i enabled assertions) i couldn't reproduced the reported symptom.

Did you test with the exact references of nuttx and nuttx-apps repos I reported?

[yamt]

i've run the test several times, with and without an equivalent of #14722. (i enabled assertions) i couldn't reproduced the reported symptom.

Did you test with the exact references of nuttx and nuttx-apps repos I reported?

yes. at least a few times.

[yamt]

i suspect #14807 can explain the symptom. how do you think?

[tmedicci]

Anyway, it seems we have an issue with ostest itself. Even failing one of the tests (and printing its exit code), it still returns 0 for the ostest app. That said, I think I can edit the title and remove its relation with esp32s3-devkit:smp.

Internally, we add a check for the log messages to fail in case of non-zero return code.