hdiutil failures when creating DMGs on macOS 13 runner

[wkiefer]

Description

We use https://github.com/create-dmg/create-dmg as a small script wrapper around hdiutil to create and notarize DMG files from our macOS applications.

However, when switching to the macos-13 runner, we get the following error when calling hdiutil create

hdiutil: create: returning 49168
hdiutil: create failed - Resource busy

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Image: macos-13
Version: 20230426.3
Included Software: https://github.com/actions/runner-images/blob/macOS-13/20230426.3/images/macos/macos-13-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macOS-13%2F20230426.3

Is it regression?

https://github.com/actions/runner-images/releases/tag/macOS-12%2F20230425.1

Expected behavior

hdiutil create should not return error 49168, resource busy

Actual behavior

hdiutil create consistently returns error 49168, resource busy

Repro steps

I will try to make a public repo sample repro — right now steps are to build with Xcode 14.3 on the macOS 13 image, then run

create-dmg \
  --app-drop-link 240 332 \
  --icon "Sample.app" 240 108 \
  --icon-size 128 \
  --volname "Sample" \
  --window-pos 100 100 \
  --window-size 480 582 \
  --codesign "Your Developer ID Application" \
  "/output/path/Sample.dmg" \
  "/input/folderWithApp/"

Or simply try to run any hdiutil create command in the context of a macOS 13 runner

[Alexey-Ayupov]

Hello @wkiefer, we will take a look, however, please remember macOS 13 is still in beta version.

[vpolikarpov-akvelon]

Hi @wkiefer. I didn't succeed reproducing the issue. Both hdiutil and create-dmg work flawlessly for me. Could you provide more detailed repro steps or share a link to the failing workflow?

[geoffthemedio]

I think we have hit the same issue.
https://github.com/freeorion/freeorion/tree/MacOS_action_versions
https://github.com/freeorion/freeorion/actions/runs/4902807369/jobs/8755515334?pr=4509#step:9:426

CPack Error: Error executing: /usr/bin/hdiutil create -ov -srcfolder "/Users/runner/work/freeorion/freeorion/build/_CPack_Packages/MacOSX/DragNDrop/FreeOrion_2023-05-06.ba56d96_Test_MacOSX_10.15" -volname "FreeOrion" -fs "HFS+" -format UDZO "/Users/runner/work/freeorion/freeorion/build/_CPack_Packages/MacOSX/DragNDrop/temp.dmg"
CPack Error: Error generating temporary disk image.
hdiutil: create failed - Resource busy

CPack Error: Problem compressing the directory
CPack Error: Error when generating package: FreeOrion
Command PhaseScriptExecution failed with a nonzero exit code

** BUILD FAILED **


The following build commands failed:
	PhaseScriptExecution CMake\ PostBuild\ Rules /Users/runner/work/freeorion/freeorion/build/build/FreeOrion.build/Release/package.build/Script-571234464C2A5D0D43F1AB42.sh (in target 'package' from project 'FreeOrion')

[vpolikarpov-akvelon]

Hey @geoffthemedio, @wkiefer. Looks like the issue is gone as an image version 20230509.4 have been rolled out. Could you try to run your workflow again?

[geoffthemedio]

I reran the action and it completed successfully. I'm not sure that nothing else changed and that it's doing the exact same thing, but at least I can reproduce the issue now.

[Neverous]

I think I'm hitting similar issues or at least issues with hdiutil: hdiutil: create failed - No child processes and the newest image didn't help really 😢

Runner image:

  Image: macos-13
  Version: 20230509.4
  Included Software: https://github.com/actions/runner-images/blob/macOS-13/20230509.4/images/macos/macos-13-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-13%2F20230509.4

[vpolikarpov-akvelon]

Hi @Neverous. It looks similar indeed. But I suppose you might get this error because of the failure on the previous step:

error: /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: /Users/runner/work/efibooteditor/Qt/6.5.0/macos/lib found in: /Users/runner/work/efibooteditor/efibooteditor/build/dist/_CPack_Packages/Darwin/DragNDrop/EFIBootEditor-v0.0.0-3f25f4b38b30f8575f17628808319e6017e44bc7-macos-13-qt-6.5.0/dist/efibooteditor.app/Contents/MacOS/efibooteditor (for architecture x86_64), required for specified option "-delete_rpath /Users/runner/work/efibooteditor/Qt/6.5.0/macos/lib"

Could you ensure that it doesn't affect packaging?

[Neverous]

I get the same thing in macos 12: https://github.com/Neverous/efibooteditor/actions/runs/4908677336/jobs/8925056030#step:12:232 and 11: https://github.com/Neverous/efibooteditor/actions/runs/4908677336/jobs/8925056236#step:12:232 and they go through fine so I would guess that it doesn't, but Ill see If I can do anything about it maybe it will help.

[Neverous]

Yeah, the rpath error is just that both macdeployqt and CPack generator that I use to make the final package are trying to adjust rpaths, and both have no option to skip it from what I can tell... I think its harmless, at least for hdiutil.

Anyway, I got to very simple reproduction steps I think: https://github.com/Neverous/efibooteditor/actions/runs/4997753556/jobs/8952437874 (the issue seems to be somewhat random as just restarting the job sometimes succeeds and sometimes not 🤔 https://github.com/Neverous/efibooteditor/actions/runs/4997791581)

[vpolikarpov-akvelon]

Hey. I finally realized what causes this issue. Looks like it's XProtectBehaviorService that was introduced in macOS 13 Ventura. If you are not lucky enough it may lock your newly created dmg and hdiutil will fail. I can suggest implementing retry in you workflow to workaround this.

Another option may be shutting aforementioned service down (or just killing the process). You may use something like this:

echo killing...; sudo pkill -9 XProtect >/dev/null || true;
echo waiting...; while pgrep XProtect; do sleep 3; done;

I can't recommend doing that because it may cause issues in the future that we can't predict and avoid. Also it may have security related consequences. Use it on your own risk.

[Neverous]

Tried both, neither works for me, I must be hitting something slightly different then 🤔 (hdiutil: create failed - No child processes seems not to be so random like the Resource busy one, but so far I wasn't able to create simple reproduction steps than running the whole build)

retrying 20 times
killing XProtect

[vpolikarpov-akvelon]

You a right, Maciej, this is something different. Killing XProtect only helps when you get "Resource busy". I still suspect that previous error may somehow cause failure of hdiutil. I understand that it didn't matter on macOS 11 & 12, but something else may have changed so now it's important. Could you try to get rid of "install_name_tool" error? Just to be sure that it's really nothing.

[Neverous]

Ah should've mentioned that with macdeployqt vs CPack comment, did that (disabled macdeployqt for tests, leaving only cpack dmg generation): Neverous/efibooteditor@e7ec720#diff-1e7de1ae2d059d21e1dd75d5812d5a34b0222cef273b7c3a2af62eb747f9d20aL410 , https://github.com/Neverous/efibooteditor/actions/runs/4997683758/jobs/8952293026#step:12:234

[vpolikarpov-akvelon]

Did you try enabling debug output for hdiutil?
I suppose it may be done by setting CPACK_COMMAND_HDIUTIL option to /path/to/hdiutil -debug

[Neverous]

Don't see anything interesting unfortunately, just that diskimages-helper died 😞 logs

2023-05-25 17:08:25.255 hdiutil[11275:37357] [DIHelperProxy watchForHelperDeath] helper exited early
2023-05-25 17:08:25.255 hdiutil[11275:37357] helper died
2023-05-25 17:08:25.255 hdiutil[11275:37357] setHelperDoneWithResult: _helperDone = YES, _threadResultsError = 10

[vpolikarpov-akvelon]

Hey @Neverous. Looks like your issue may be resolved by running hdiutil using sudo.

I added this line to CMakeLists.txt and packaging works flawlessly:

set(CPACK_COMMAND_HDIUTIL "/usr/bin/sudo /usr/bin/hdiutil")

[Neverous]

Yes, thank you! It works indeed. Interesting, some permissions changed with macOS 13?, not sure why it would now require elevated access 🤔 but it packages successfully now with sudo 🙇 .

[vpolikarpov-akvelon]

Glad to hear! Well, looks like problems have been solved so i'm closing this issue for now. Feel free to reopen it if problems return.

To sum up:

• If hdiutil create fails on macOS 13 with No child processes then try running it with sudo;
• If it fails on macOS 13 with Resource busy sometimes then try killing XProtectBehaviorService.

[SanderVocke]

I also see "hdiutil: create failed - Resource busy" on macos-13.

kill -9 XProtect doesn't seem to help.

macos-12 build seems to work OK.

Same here. Unfortunately, macos-12 seems to be deprecated now.

[nuttyartist]

I'm still having issues with that. Did anyone find a solution for macOS-13?

[fredowski]

Same here: https://github.com/fredowski/osxbundler/actions/runs/12228994609/job/34108086880#step:5:9820

hdiutil create -fs HFS+ -srcfolder ./tmp-Oh1p6JiX2W -volname pspp pspp-2.0.1-2-x86_64.dmg
hdiutil: create failed - Resource busy

[nuttyartist]

If anyone curious, our solution was to keep trying until it works (max 10 times), it seems to work eventually:

https://github.com/nuttyartist/notes/blob/57542f6ca8e7a29192bb7593a6a573637248a33d/.github/workflows/macos.yml#L151-L176

[fredowski]

I was curious and tried your retry fix: fredowski/osxbundler@67d668c
It worked for me. One of my two builds needed one retry. Thanks for your hint!

[adamkewley]

@guihkx (commit) @nuttyartist (comment) , @fredowski (commit), @The-Compiler (commit) thanks for the workaround shares. A bash retry loop is preferable to (e.g.) killing a system daemon with sudo!

(edit: added links + @guihlx's workaround)

[nuttyartist]

Indeed! All credit goes to @guihkx (commit).