macos-latest cannot install homebrew packages after using this action

[ReenigneArcher]

macos-latest (i.e. macos-14) image fails to install homebrew packages AFTER using codeql-action. The same homebrew install works fine if not using the codeql-action.

I filed a report on the runner-images repo (before I realized the codeql action was causing the issue), they suggested I file a bug report here instead.
actions/runner-images#9760

This is the simplest reproduction I could create. https://github.com/ReenigneArcher/brew-error-reproduction/blob/master/.github/workflows/brew_error.yml

It works fine if not using the codeql setup step first.

Logs: https://github.com/ReenigneArcher/brew-error-reproduction/actions/runs/8870403949/job/24352158742

[aibaars]

Thanks for reporting! Could you re-run the failed job with debug logging enabled? When running in debug mode, the CodeQL Action should attach a debug artifact to the Job's summary page. That would be very helpful for diagnosing the problem.

As a workaround, have you tried running "brew install" before the github/codeql-action/init step?

[ReenigneArcher]

I have re-run it in debug mode: https://github.com/ReenigneArcher/brew-error-reproduction/actions/runs/8870403949/job/24376380948

I have not tried refactoring the workflow yet, but I know that the brew install would at least succeed if it were before the init. The actual production workflow is a bit more complex than the reproduction workflow, I'd like to avoid refactoring it, if possible.

[aibaars]

Thanks! The attached debug artifact wasn't as helpful as I expected, however, I spotted the following in the raw Actions logs:

2024-04-29T13:14:13.1521510Z ##[warning]Failed to check available disk space: Error: Command failed: df -Pk -- /Users/runner/work/brew-error-reproduction/brew-error-reproduction
dyld[4994]: terminating because inserted dylib '/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools/osx64/libtrace.dylib' could not be loaded: tried: '/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need 'arm64e')), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools/osx64/libtrace.dylib' (no such file), '/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need 'arm64e'))
dyld[4994]: tried: '/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need 'arm64e')), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools/osx64/libtrace.dylib' (no such file), '/Users/runner/hostedtoolcache/CodeQL/2.17.1/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need 'arm64e'))

Somehow CodeQL's tracer fails on the df command which seems to have an architecture with a funny e suffix. Not sure what that is about. I'll pass it on to the team.

[aibaars]

The actual production workflow is a bit more complex than the reproduction workflow, I'd like to avoid refactoring it, if possible.

Another workaround you could try is to change the runs-on: property of the workflow. You could try switching to an older version of macos or to an intel-based macos runner. It's not great, but might be easier an easier tweak than re-factoring the workflow.

[ReenigneArcher]

Re-ordering the workflow won't be feasible due to it requiring the build to run after this action is initialized: https://github.com/orgs/community/discussions/68044#discussioncomment-7109477

Mostly this is due to me using a single code-ql workflow for all my org repos, which checks for the existence of a .codeql-prebuild-<language>-<runner_OS>.sh file... and I'd rather not have multiple scripts for code-ql for each os... e.g. deps and build.

[ReenigneArcher]

Just checking in. This is still occurring: https://github.com/ReenigneArcher/brew-error-reproduction/actions/runs/9916092767/job/27397917114#step:3:66

[angelapwen]

Thank you for sharing your workflow runs — this looks like a new issue with our build tracer. We've raised this with the appropriate team internally.

[angelapwen]

After a colleague dug into this a bit more, here's what we've come up with: our build tracer forces relocation to x86, which leads to a x86 execution under Rosetta. Apparently, that x86 execution doesn't play well with Homebrew. For now, please try the suggestion in the logs, arch -arm64 brew install ... to force Homebrew back to the arm64 execution.

[ReenigneArcher]

@angelapwen Thank you for looking into this. I am pretty sure I tried the suggestion in the logs (arch -arm64 brew install ...) previously and it did not help... I will try again though and report back.

[ReenigneArcher]

Sorry for the delay... When using the suggestion from the logs (ReenigneArcher/brew-error-reproduction#2), I get the following error:

arch: posix_spawnp: brew: Bad CPU type in executable

https://github.com/ReenigneArcher/brew-error-reproduction/actions/runs/11108869334/job/30862839806?pr=2#step:3:48

Interesting since homebrew suggests to use that exact command.

[ReenigneArcher]

Just following up to hopefully keep this on the radar.

The macos-12 image is about to removed. macos-13 will likely be EOL in October 2025, so there's still a little time.

These also seem related:

• CodeQL breaks iOS builds #2347
• github/codeql-action/init@v3 failing on macos-14 runner #2109

[smowton]

This should be fixed in CodeQL 2.19.4: an arch -arm64 command would often fail because CodeQL would create and try to invoke an x86-64-only binary with the explicit architecture flag passed to posix_spawn by arch. Now we suppress that behaviour when posix_spawn forces use of arm64, though it still is not possible to trace compiler invocations reached this way, so this is useful for arch -arm64 brew install, but not arch -arm64 gcc analyse_this.c -c -o ... for example.

[ReenigneArcher]

@smowton when will this action start using 2.19.4? It looks like it's still using 2.19.3.

[smowton]

It became fully enabled by default at around 4.30pm on the 3rd Dec; try a rerun now?

[ReenigneArcher]

Yes, looks like it's using the new version now. Seems to work if forcing arm for brew, but not using the normal brew commands: https://github.com/ReenigneArcher/brew-error-reproduction/actions/runs/12160656532/job/33913434744#step:3:62

Good enough though, thank you!