Internal Errors when running builds

[Brett-Best]

Description
Builds either don’t start running or fail with an internal error.

Area for Triage: Apple

Question, Bug, or Feature?:
Bug

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11.0
• Windows Server 2016 R2
• Windows Server 2019

Expected behavior
The build would complete successfully.

Actual behavior
The build is either stuck on starting or fail with an internal error.

Repro steps
A description with steps to reproduce the issue. If your have a public example or repo to share,
please provide the link.

1. https://github.com/SwiftUIX/SwiftUIX/actions/runs/457625830
2. Re-run the workflow
3. Some of the builds won’t start and some fail with internal error

EDIT:
More workflow failures on a paid org:
https://github.com/interact-technology/medCompanion-Apple/actions/runs/466140750
https://github.com/interact-technology/medCompanion-Apple/actions/runs/466644185

[nakkht]

Had the same issue. Re-running jobs however succeeded.
Could this somehow be related to long build queues when using macOS runner? (Happens on private/public repos)

[Brett-Best]

@nakkht I tried re-running the same job I think 3 times and different jobs failed each time.

[richiksc]

I'm also experiencing the same issue. My workflow has two jobs. The Ubuntu 20.04 job started immediately, while the macOS 11 one failed to start until it was automatically restarted an hour and a half later.

https://github.com/dominikWin/badlogvis/actions/runs/458499059

However, with the automatically restarted job, it didn't replace the failing PR check of the job that failed to start. Now there are three checks - two passing and one failing.

[jameslamb]

This has been affecting us in LightGBM as well. I contacted Azure support this morning about this issue (you can read what I sent here) and got this response.

We're not aware of other reports of similar problems since you submitted this & we have not experienced any monitoring alarms, so this is likely limited to your user account or organization only. Unfortunately we are unable to offer support for that here.

I will now close this outage report ticket.

[MaxDesiatov]

+1 here. All of the repositories that have GHA set up I've contributed to after #2247 was resolved have either failing jobs, or they take half an hour on average to start for macOS 11.0 agents.

[AlenaSviridenko]

Hi!
We are sorry for inconveniences. We are experiencing queuing issues with macOS 11.0, but unfortunately, currently we are unable to add more capacity for macOS 11.0 runners, so it is still in "preview" mode with only few machines, and macos-latest label still points to macOS 10.15.
Our engineers are actively investigating how we can improve the situation with queues. We will keep you posted.

Until then, please, switch to macos-10.15if it is not critical for you to use Big Sur.
Thank you for your patience.

[Brett-Best]

Thanks for the update @AlenaSviridenko, hoping macOS 11.0 moves out of “preview” mode soon given it was released ~2months ago to the public.

[MaxDesiatov]

For a few days already builds on macOS 11.0 agents no longer wait in the queue, they are just cancelled over and over, and even when restarted manually. I can't use macOS 10.15 agents as my framework explicitly requires new SwiftUI APIs for macOS, which are available only on maOS 11.0.

If the actual reason is the lack of free agents, could GitHub Actions at least avoid cancelling the builds? I'm fine with waiting in the queue, but why do builds need to be cancelled completely?

[Kaspik]

@AlenaSviridenko Thanks for the response - is there a timeline when macOS 11 will move out of preview, and hopefully will also be updated to more recent version?

I believe most of us that point to macOS11 actually have to point to them, so upgrading more of your runners to new macOS and cancel the preview support might be the only solution to this.

[richiksc]

Yes, I'm using macOS 11 runners so I can cross-compile for Apple Silicon Macs.

[jidicula]

We're using macOS 11 runners to be sure that the packages we're developing will build and pass tests on macOS 11.

[AlenaSviridenko]

Hi!
I am closing this issue, since after pool transition to private preview the situation got better, but we are still looking into the possibility to add more capacity to macOS 11.0 pools.
Please, feel free to create a new issue in case of any other questions. Thanks!

[bhaller]

Hi!
I am closing this issue, since after pool transition to private preview the situation got better, but we are still looking into the possibility to add more capacity to macOS 11.0 pools.
Please, feel free to create a new issue in case of any other questions. Thanks!

@AlenaSviridenko As the maintainer of a project that used macos-11.0 as a os target for my CI, I removed that target when it was transitioned to "private preview" since it no longer seemed to work at all for me. I was watching this bug in the hopes that it would inform me of when I could go back to using macos-11.0. What should I do now? Is there an issue that represents "users want to build for macos-11.0 but they can't"? Or can I somehow request to be included in the "private preview"? With this bug closed, we seem to be stuck in a state of limbo.

[Brett-Best]

@AlenaSviridenko I’m still seeing a lot of failures, you can check the last ~5 runs here https://github.com/SwiftUIX/SwiftUIX/actions?query=branch%3Afeature%2FXcode-12.5.

I hope that by the time Xcode 12.5 is out of beta, that the situation is improved more given it requires macOS Big Sur.