AArch runner fails frequently

[davidlattimore]

I don't necessarily have any solution for this, but wanted to raise an issue in case others had ideas.

We regularly see CI failures on the aarch64 jobs. At the moment we're running 4 aarch64 jobs (3 versions of ubuntu + openSUSE). Generally when there's a failure, it's just one or two of the jobs that fail and the others succeed.

The failure causes are varied, but mostly seem to be related to network or docker.

• Value cannot be null. (Parameter 'network')
• Error response from daemon: failed to create task for container
• Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
• Error message: Failed to connect to cdn.opensuse.org port 80 after 4001 ms: Could not connect to server

[mati865]

Sounds like AArch64 runners aren't stable yet (they are quite a new feature). I don't know how to recover from the Docker issue, but network issues should be solvable with a simple retry. Alternatively, for a more sophisticated solution, we could pre-build all images with up-to-date dependencies baked in and store them on GitHub Packages.

[mati865]

This issue stems from the combination of kernel version and hardware used by GH runners: https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/crashes.20on.20new.20aarch64.20GHA.20runners/near/498394368
So, downgrading the image version I applied is an effective workaround. Since then, there has been only one crash in zypper, but those also happen on x86_64.

[davidlattimore]

Thanks for looking into this! Sounds like we can probably close this.

[mati865]

GitHub has rolled back problematic hardware. Should we revert the workaround in #377?

[davidlattimore]

Sure. Happy to see how it goes.