[FEATURE] NuttX Dashboard for Build Farm

[lupyuen]

Is your feature request related to a problem? Please describe.

We have the First Version of the NuttX Dashboard for our Build Farm!

Try it here: https://nuttx-dashboard.org

This Dashboard consolidates the Build Logs from all Linux Build Servers (Alin, mine, GitHub CI). So it's convenient to spot any Errors and Warnings in our builds.

The Highlights Section will turn red if there are errors. (Severity 0)

We can filter the builds by Arch / SubArch / Group / Board / Config:

If you're keen to add your own Ubuntu PC or Mac to our Build Farm, please follow the steps below and tell me your GitHub Gist ID:

• "Join Ubuntu Build Farm"
• "Join macOS Build Farm"

Describe the solution you'd like

We created the Dashboard with open-source Grafana and Prometheus. Source Code is here

Describe alternatives you've considered

No response

Verification

• I have verified before submitting the report.

[lupyuen]

GitHub Actions CI Logs are now inside NuttX Dashboard

Our NuttX Dashboard now ingests the CI Build Logs from GitHub Actions! (Like an Amoeba)

• That's why our Dashboard looks Red, because of errors in the CI Build (pic below)
• When we click a Red Box, it jumps to the GitHub Actions Log (showing the error)

How It Happens:

• Daily at 00:00 UTC and 12:00 UTC: I click on NuttX Mirror Repo > Sync Fork > Discard Commits
• Run this script to Enable the macOS and Windows Builds: enable-macos-windows.sh
• Wait 2.5 Hours for the build to complete
• Then run this script to Ingest the GitHub Actions Logs into NuttX Dashboard: ingest-nuttx-builds/github.sh
• Source Code for Log Parsing is all here

What about ingesting macOS / msys2 / msvc Logs? Right now they show tooooo many ... warningses. We should probably fix the warnings before ingesting the logs into NuttX Dashboard.

[lupyuen]

NuttX Build History

If a Build Fails, how will we know what Broke the Build?

Browse to nuttx-dashboard.org, click "See the NuttX Build History".

Select the Board and Config. We'll see the History of Builds for the Board / Config.

With the Git Hash of NuttX Repo and NuttX Apps Repo, hopefully we can track down to the exact commit that failed.

🙏🙏🙏 Please join your Ubuntu PC to our Build Farm, tell me your Gist ID. So we can capture more builds, and track down quickly to the exact commit that failed! 🙏🙏🙏

[cederom]

Very cool congratz @lupyuen :-) And its already working :-) I need to buy a disk to a laptop that I can use to join The NuttX Farmers :D

[lupyuen]

Thanks @cederom! Yep hope to have your laptop in our NuttX Build Farm real soon :-)

[cederom]

What minimal disk is recommended 500GB or 1TB?

[lupyuen]

@cederom 500 GB should be more than sufficient. I'm using 51 GB right now :-)
https://gist.github.com/nuttxpr/34b8fa96537acdf7400b87d0e6794cf1#file-ci-arm-14-log-L905

[lvanasse]

@lupyuen by any chance, do you have a Dockerfile for the whole setup (I know, I've been talking about docker for a while (it'll be after the doc migration;)))

Reason why I'm asking, is that I have an Unraid server at home, which I host my media. And I was thinking maybe it would be possible to simply docker another docker on it.

[lupyuen]

@lvanasse Yep the Dockerfile is here: https://github.com/apache/nuttx/blob/master/tools/ci/docker/linux/Dockerfile

If you need the Docker Image, I can export it for you :-)

[simbit18]

Hi @lupyuen this PR #14654 removed the warnings now you can enter msys2 logs in NuttX Dashboard.

[lupyuen]

@simbit18 Awesome! Lemme work on it...

[lupyuen]

Hi @simbit18: msys2 Builds are now in nuttx-dashboard.org :-)

[simbit18]

Thank you @lupyuen

[lvanasse]

@lupyuen I'm setting up a server to run the build and to potentially add it to the build farm :). I was wondering if you just put the run-ci.sh inside a cronjob to be able to periodically run the build?

Also, I think you'll need by Gist ID? It says in your article, that it's better to have another account for security? I'm not sure if I understand why you might need to do that.

And also, is there a target group that would be best for me to set the CI to run on?

Cheers :)

[lupyuen]

Thank you so much @lvanasse!

• run-ci.sh inside a cronjob: Might work? Assuming that it can handle the sudo.
• Originally I assumed the Build PC would be compiling NuttX 24 x 7 (because New PRs will be merged all the time), but it's perfectly OK to run it sometimes
• Creating a GitHub Account: The NuttX Dashboard will be crawling all your Gists. If we're comfortable letting a bot crawl through your Gists, then it's OK :-)
• Also the gh command will be authenticated using your GitHub Account, which means it has all repo access rights, and it might be too powerful. So it depends if you're comfortable with that :-)

[lupyuen]

@lvanasse As for Target Group: How about these, we can cut down later thanks :-)

    risc-v-01 risc-v-02 risc-v-03 \
    risc-v-04 risc-v-05 risc-v-06 \
    xtensa-01 xtensa-02 \
    arm64-01 x86_64-01 \
    sim-01 sim-02 sim-03 \
    other

I'm running these right now: https://github.com/lupyuen/nuttx-release/blob/main/run-ci.sh
Alin is running these: https://gist.github.com/jerpelea

[lvanasse]

Sounds good! Then if you are not running the script in a cronjob how are you automating the process?

[lupyuen]

@lvanasse It runs on my Ubuntu Desktop all day all night, I'm not using the PC for anything else :-)

[lvanasse]

@lupyuen, sorry just so I understand correctly, so you do launch the script manually each time it finished?

[lupyuen]

@lvanasse Oops sorry, please use this script, it will run all jobs forever:

https://lupyuen.github.io/articles/ci2#build-nuttx-for-all-target-groups

## Download the scripts
git clone https://github.com/lupyuen/nuttx-release
cd nuttx-release

## Login to GitHub in Headless Mode
sudo apt install gh
sudo gh auth login

## (1) What Account: "GitHub.com"
## (2) Preferred Protocol: "HTTPS"
## (3) Authenticate GitHub CLI: "Login with a web browser"
## (4) Copy the One-Time Code, press Enter
## (5) Press "q" to quit the Text Browser that appears
## (6) Switch to Firefox Browser and load https://github.com/login/device
## (7) Enter the One-Time Code. GitHub Login will proceed.
## See https://stackoverflow.com/questions/78890002/how-to-do-gh-auth-login-when-run-in-headless-mode

## Run the Build Job forever: arm-01 ... arm-14
sudo ./run-ci.sh

[lvanasse]

Ahh ok sorry, I didn't read the whole script properly, and didn't see the infinite loop.

Thank you for the clarification, I'm setting up the server at the moment and I'll let you know my Gist ID after everything is running smoothly :).

[lvanasse]

Things seems to be running smoothly :): https://gist.github.com/lvanasse/74976165ac7605a7376a6c026ff020ee

If you want to add my ID to the Nuttx Build Farm, I would be honoured :)!

After it ran a first time through all the target groups, I might just stop it and set it up so I can forget about it and let it run on my server.

[lupyuen]

Super Awesome @lvanasse! Your PC is now in our NuttX Build Farm, thank you so much! :-)

https://nuttx-dashboard.org/d/fe2q876wubc3kc/nuttx-build-history?from=now-2d&to=now&timezone=browser&var-arch=$__all&var-subarch=$__all&var-board=$__all&var-config=$__all&var-group=$__all&var-Filters=user%7C%3D%7Clvanasse

Here's how I added your PC to our Build Farm. macOS will join us real soon :-)
lupyuen/ingest-nuttx-builds@4cbaf14

[lvanasse]

Awesome! Thank you for everything @lupyuen!

[lvanasse]

@lupyuen you might have seen some funniness in my gists. I had a couple of issue with my unraid VM, but things should be smooth from now on :).

[lupyuen]

@lvanasse haha cool no worries :-)