macOS builders randomly stops working on GitHub Actions

[pietroalbini]

We started noticing that occasionally one of our macOS builders stop working, marking the job as either failed or canceled and providing no logs for the build. Examples of such failed builds are:

• Rollup of 6 pull requests #71934 - https://github.com/rust-lang-ci/rust/runs/648528503
• Rollup of 7 pull requests #71951 - https://github.com/rust-lang-ci/rust/runs/649952440
• Update cargo #71925 - https://github.com/rust-lang-ci/rust/runs/651724911
• upgrade chalk and use chalk-solve/chalk-ir/chalk-rust-ir #69406 - https://github.com/rust-lang-ci/rust/runs/656100319
• [WIP] Rollup of 5 pull requests #72083:
  • https://github.com/rust-lang-ci/rust/runs/661212134
  • https://github.com/rust-lang-ci/rust/runs/661326448

[pietroalbini]

New one: https://github.com/rust-lang-ci/rust/runs/656100319

[pietroalbini]

Two new ones in a row from #72083:

• https://github.com/rust-lang-ci/rust/runs/661212134
• https://github.com/rust-lang-ci/rust/runs/661326448

[RalfJung]

This broke the 2nd rollup in a row: #72187
Makes it kind of hard to land anything, really.^^

There's no error, just a red cross next to "upload artifacts to S3".

[pietroalbini]

We have a call with Github today, if there is no news we'll rollback the gate on GHA.

[pietroalbini]

GitHub is going to continue looking into this ❤️

In the meantime I'll remove the double-gate, to let the queue run more smoothly.

[pietroalbini]

Removed the gate on GHA.

[RalfJung]

@pietroalbini thanks!
Does this take effect on the already running #72202 (comment) ?

[pietroalbini]

It should.

[Aaron1011]

Possibly related: https://github.com/rust-lang-ci/rust/runs/685622726

[pietroalbini]

Just got another failure related to this:

• https://github.com/rust-lang-ci/rust/runs/834243531

[youknowone]

I am getting similar experience in other rust project, so happy to see Github is looking into this issue.

[thejoebourneidentity]

Thanks to everyone for their patience on this issue, I work on the GitHub Actions compute platform that powers our runners and wanted to share an update.

We're aware of the issue and have multiple members of our team across our entire hardware stack actively working on it. We've been able to identify a specific data center that may be the primary failure point. We're rapidly building out additional telemetry and alerting to help us find the root cause. Stay tuned, we want to fix this as soon as possible.

[matthewmccullough]

We're aware of the issue have multiple members of our team across our entire hardware stack actively working on it. We've been able to identify a specific data center that may be the primary failure point. We're rapidly building out additional telemetry and alerting to help us find the root cause. Stay tuned, we want to fix this as soon as possible.

@thejoebourneidentity Thanks for the insight into the bug investigation and reiteration that you and the team are working on it.

[alepauly]

@pietroalbini We continue looking at this and I wanted to confirm you are still seeing the exact same symptoms of disconnects. We've been able to curve the cases where the runner disconnects and are now focusing more on perf issues that might be leading to timeouts. I browsed a few workflows and noticed timeouts such as this. Is this a good illustration of the current problem or are you seeing disconnects at a high rate still? Thanks!

[pietroalbini]

@alepauly thanks for getting back to us! Indeed, it seems like the runner disconnects are not appearing anymore! I'll monitor closely the build results in the coming week to see if they appear.

Regarding the cancelled builds, like the one you mentioned, that's a mistake on our end 😅. Since our build matrix is so big we built an action to cancel builds once a new commit is pushed to the branch. While this is not a problem normally since our queue bot only pushes a new commit to the test branch when the previous one finishes, we're not gating our builds on the macOS builders yet due to this issue, so the queue bot isn't waiting for them to finish before pushing the new commit (causing the action to cancel the build).

I opened a PR to disable the build cancellation for the macOS jobs, so the cancellations should disappear as soon as the PR lands. If we don't see any other spurious failure for a while we'll try to gate our builds on macOS again!

Thanks again for looking into this issue ❤️

[pietroalbini]

It seems a runner disappeared only one time across the whole week, which is definitely an improvement!

I'll propose during the next weekly infrastructure team meeting to try and double-gate macOS on both Azure and GitHub Actions, to see what the impact would be on our queue size.

[alepauly]

It seems a runner disappeared only one time across the whole week, which is definitely an improvement!

We'll look at it on our end, we expect this to be better but not fully solved yet. Thanks for the report.

[pietroalbini]

We didn't see it anymore for months, closing this. Thanks to everyone involved in fixing this! ❤️