arm-fanned builds stalled

[lundibundi]

arm CI is stalled for some reason (3+ hours on each of the builds below)

https://ci.nodejs.org/job/node-test-pull-request/32798/
https://ci.nodejs.org/job/node-test-pull-request/32803/
https://ci.nodejs.org/job/node-test-pull-request/32802/
https://ci.nodejs.org/job/node-test-pull-request/32805/

/cc @nodejs/build

[richardlau]

It looks like the pi3's are offline as they've run out of disk space. I've logged into one of them, test-requireio_kahwee-debian10-arm64_pi3-1, and the .git folder in the node-test-binary-arm workspace was 24G. I've wiped that workspace and brought it back online and look at doing similar on the rest.

Some digging around issues here shows we have https://ci.nodejs.org/job/git-clean-rpi/ which at one point was regularly run (#893 (comment)) but it doesn't look like it is anymore -- @nodejs/build-infra anyone know if this is deliberate, or if we should be running the clean up job regularly?

[richardlau]

I've brought back what I could but more than half of the pi3's are unreachable (cannot ssh into them) and remain offline:

The ones that are back online are processing jobs, but the backlog was quite big so it may take a while to clear.

[rvagg]

I've brought some of the others back online.

The regular clean job was more helpful when we had more limited space. We now have a much larger disk that they're all sharing so it wasn't supposed to be as much of a problem .. but 24G isn't sustainable!

I don't know why the regular clean was disabled. Maybe because it would land us with a bunch of queued jobs that won't run because various Pi's are offline? I think I remember that being a problem.

The good news here is that it's been a lot longer since these little guys need cleaning up, so we shouldn't need to do it as often in the future. Maybe we need something a more more manual to remind us to do it? A bot to open an issue to tell us to clean them every month might be handy, is there such a thing?

[lundibundi]

Maybe we need something a more more manual to remind us to do it? A bot to open an issue to tell us to clean them every month might be handy, is there such a thing?

I think that will be easy to do with a monthly scheduled task on GitHub Actions if needed
(i.e. https://github.com/marketplace/actions/create-an-issue or a small script + schedule - '0 0 1 * *')