macos-14 runner: intermittent "invalid developer directory" when installing Xcode 16.0

[tsal]

Describe the bug
Our first step is to setup Xcode in our builds. Today we've been getting a large number of immediate failures, in the Xcode setup step; error message is xcode-select: error: invalid developer directory '/Applications/Xcode_16.0.app'. This normally indicates the directory is not a signed app or package folder from Apple, or it's just outright missing.

To Reproduce
Steps to reproduce the behavior:
Currently, the there is no reliable reproduction, but try installing 16.0 on a macos-14 runner a few times and you should see the error occasionally (we see it 1 in 6 right now).

Expected behavior
Since Xcode 16.0 is not the latest version, but compatible with macos-14, I would expect it to install without issue.

Runner Version and Platform

macos-14

OS of the machine running the runner? macos

What's not working?

We inject the version via secret, hence the ***, but here's the output:

Job Log Output

N/A, the job doesn't get past this first step.

Runner and Worker's Diagnostic Logs

Don't have access to a _diag folder that I can find in the log capture ZIP.

[ivard]

We sometimes experience the same issue indeed. In most cases the issue doesn't occur, but sometimes we get xcode-select: error: invalid developer directory '/Applications/Xcode_16.0.app/Contents/Developer'. Maybe there is a rolling upgrade going of the MacOS 14 runner image that breaks things.

[ronanj2]

Also intermittently getting this

2024-11-10T20:46:52.3352120Z ##[group]Runner Image
2024-11-10T20:46:52.3390050Z Image: macos-14-arm64
2024-11-10T20:46:52.3393810Z Version: 20241108.442
2024-11-10T20:46:52.3398590Z Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20241108.442/images/macos/macos-14-arm64-Readme.md
2024-11-10T20:46:52.3404040Z Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20241108.442
2024-11-10T20:46:52.3408660Z ##[endgroup]

[jamescrosswell]

We sometimes experience the same issue indeed. In most cases the issue doesn't occur, but sometimes we get xcode-select: error: invalid developer directory '/Applications/Xcode_16.0.app/Contents/Developer'. Maybe there is a rolling upgrade going of the MacOS 14 runner image that breaks things.

Same thing just started happening on our repo 😢

[BillBu]

Also getting this error today:

xcode-select: error: invalid developer directory '/Applications/Xcode_16.app/Contents/Developer'

No changes in our code or build config that would have caused it.

[stephencelis]

We're seeing this recently and regularly in our CI pipelines, as well.

[thomasbach-dk]

For me it looks like the job checks out different versions of the runner software when runs-on: macos-14 is specified.

The same workflow run uses these versions randomly:

20241108.442 (which does not include xcode 16)
20241022.361 (which includes xcode 16)

[peter-eh-easylaser]

For me it looks like the job checks out different versions of the runner software when runs-on: macos-14 is specified.

The same workflow run uses these versions randomly:

20241108.442 (which does not include xcode 16) 20241022.361 (which includes xcode 16)

Can confirm. We see the same error and this seems to be the common factor for the failing runs.

[stephencelis]

For what it's worth according to GitHub it seems to be expected behavior, and Xcode 16 is no longer supported on macos-14 runners:

actions/runner-images#10918 (comment)

The main annoyances I've observed:

• runs-on cannot be a matrix variable, so we're having to maintain copy-pasted workflow configuration.
• macos-15 runners are currently slower to pick up jobs, so generally our CI is taking longer now, even though the jobs themselves typically run faster.

[ivard]

@stephencelis Thanks for sharing the comment about the policy change. That explains a lot. The old runner image policy was the main reason we used xcode-select in our workflows in the first place, so the new policy is much better in my view.

[pmahend1]

TLDR, is there a work-around for this? Our pipelines are stuck failing and macos-15 is still in preview

[tsal]

TLDR, is there a work-around for this? Our pipelines are stuck failing and macos-15 is still in preview

We were able to switch to macos-15 today. I don't like doing this untested, but it's what got us around part of the issue.

The primary issue was the lack of communication on this to the folks who pay for build time. We could have saved probably hours of not-cheap billed time if we'd know about the PR linked above.

[ronanj2]

actions/runner-images#10703

[marcprux]

According to actions/runner-images#10929 (comment), Xcode 16 is being added back to the macos-14 images and will be available again soon.

[pmahend1]

TLDR, is there a work-around for this? Our pipelines are stuck failing and macos-15 is still in preview

We were able to switch to macos-15 today. I don't like doing this untested, but it's what got us around part of the issue.

The primary issue was the lack of communication on this to the folks who pay for build time. We could have saved probably hours of not-cheap billed time if we'd know about the PR linked above.

Thank you. I am trying to edit pipeline making macos agent, Xcode as variables. If I choose macos-15 it fails with /Users/runner/hostedtoolcache/dotnet/packs/Microsoft.Android.Sdk.Darwin/34.0.143/targets/Microsoft.Android.Sdk.Tooling.targets(31,5): error XA5205: Cannot find aapt. Please install the Android SDK Build-Tools package with the /Users/runner/Library/Android/sdk/tools/androidprogram.
This is frustrating.

[turbo25mike]

@pmahend1 - Have you found a workaround to this problem?

@marcprux - Is there a way to request a macos-14 with a specific version number that has xcode 16 installed on it?

[marcprux]

Is there a way to request a macos-14 with a specific version number that has xcode 16 installed on it?

I think you just need to wait for the rollout to complete. In the comment they mention that it may take 3-4 days. You can check the "Rollout Status" at https://github.com/actions/runner-images to see when it is complete.

[pmahend1]

@turbo25mike No work around for now. We may try downloading Android build tools but it is too much of a temporary effort just to pass builds. I have marked PR pipeline as non-mandatory for merge. And for release I am going to build manually. Hope it gets fixed once they complete rollout.

[shrimpburner]

This issue seems to have gone away as of today. Thinking they re-added Xcode 16 to the macOS-14 image and have not updated the runner-images GitHub page yet.