[FEATURE] CI Test should always terminate after 1 hour

[lupyuen]

Is your feature request related to a problem? Please describe.

CI Test will sometimes run for 6 hours (before getting killed by GitHub):

• drivers/pipes: return after short write if buffer is full #14667 (comment)

This is not so great because:

1. It will increase our usage of GitHub Runners. Which may overrun the GitHub Actions Budget allocated by ASF.
2. Suppose right after CI Test there's another build. If CI Test runs for all 6 hours, then the build after CI Test will never run.
3. We are now running our own Ubuntu PCs as a NuttX Build Farm. The PCs will hang forever until we restart the Build Jobs.

Describe the solution you'd like

CI Test should complete within 1 hour. It should gracefully terminate itself (and report an error) if the runtime exceeds 1 hour.

Describe alternatives you've considered

Right now I'm manually killing all CI Jobs that run over 3 hours. And restarting the Ubuntu PCs in our NuttX Build Farm.

Verification

• I have verified before submitting the report.

[simbit18]

@lupyuen maybe we also need to put a maximum number of minutes for a job to run.

GitHub Actions timeout
https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes

[lupyuen]

@simbit18 Yep right now it quits after 6 hours: https://github.com/NuttX/nuttx/actions/runs/11714861244

[simbit18]

@lupyuen so we should put

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 180 #  (3 hours) Decrease this timeout value as needed

[lupyuen]

@simbit18 Hmmm suppose right after CI Test there's another build. If CI Test runs for all 3 hours, then the build after CI Test will never run. So actually I prefer if CI Test could terminate itself (after 1 hour) and let other builds run.

Unless we always park CI Test at the end of the job?

[simbit18]

@simbit18 Hmmm suppose right after CI Test there's another build. If CI Test runs for all 3 hours, then the build after CI Test will never run. So actually I prefer if CI Test could terminate itself (after 1 hour) and let other builds run.
right !!!

Describe the solution you'd like
CI Test should complete within 1 hour. It should gracefully terminate itself (and report an error) if the runtime exceeds 1 hour.

This in my opinion is the right solution

The GitHub Actions timeout is only for safety and not to fall back into the tunnel #14376

[lupyuen]

It's happening again:

• [BUG] rv-virt/citest: test_hello or test_pipe failed #14808

[lupyuen]

Wonder if this will work for GitHub CI? I'm testing it for macOS Build Farm:
https://github.com/lupyuen/nuttx-build-farm/blob/main/run-job-macos.sh#L131-L144

## If CI Test Hangs: Kill it after 1 hour
( sleep 3600 ; echo Killing pytest... ; pkill -f pytest )&

## Run the CI Job
./cibuild.sh -i -c -A -R testlist/$job.dat

[lupyuen]

Yep this kills the CI Test after 2 hours! (Assuming our jobs are not supposed to exceed 2 hours)

We changed build.yml:

cd sources/nuttx/tools/ci
if [ "X${{matrix.boards}}" = "Xcodechecker" ]; then
  ./cibuild.sh -c -A -N -R --codechecker testlist/${{matrix.boards}}.dat
else
  ## Inserted this
  ( sleep 7200 ; echo Killing pytest... ; pkill -f pytest )&
  ./cibuild.sh -c -A -N -R -S testlist/${{matrix.boards}}.dat
fi

(Build Log says "Killing pytest... Terminated" and fails correctly later)

• Which is patched into NuttX Mirror Repo by this script: enable-macos-windows.sh
• Also for Ubuntu Build Farm: run-job.sh
• And for macOS Build Farm: run-job-macos.sh