Support for installing macOS like AutoDMG

[jonanh]

I am one of those unlucky guys who have unfortunately discovered this project too late after Apple released Sierra 10.12.4 :-(

I am using it for prototyping my personal projects, and the price of WMware + vagrant wmware plugin made me drop the alternative WMware + AutoDMG + vfuse for now, so I took a look to how AutoDMG worked and I created a modified version of your prepare_iso.sh which uses installer -verboseR -dumplog -pkg "$MNT_ESD/Packages/OSInstall.mpkg" -target "$MNT_BASE_SYSTEM" to install macOS in a disk image that gets converted into a virtual box image by doing cat "$DISK_DEV" | VBoxManage convertfromraw stdin "$OUTPUT_DMG" "$DISK_SIZE_BYTES".

I also created a small tool to generate and export a OVF which can be used by the packer builder virtualbox-ovf. :-)

$ mkdir -p out; cd out;
$ sudo ../prepare_iso/prepare_vdi.sh -D DISABLE_REMOTE_MANAGEMENT /Applications/Install\ macOS\ Sierra.app/ .
$ ../prepare_iso/prepare_ovf.sh macOS_10.12.4_16E195.vdi
$ packer build -var provisioning_delay=30 -var source_path=macOS_10.12.4_16E195.ovf ../packer/template.json

[petemounce]

I'd like to try this out but don't understand how to. Would it be possible to include a diff to the README.md to describe where it slots in? Thanks!

[jonanh]

Sure. I will also try it with 10.12.5 as well :-)

[jonanh]

BTW, if you want I can try as well simplifying the commands and make it a way that all the scripts can be launched in a sequence without having to know the output names.

[jonanh]

I will add the following command to the README.

cd packer
sudo ../prepare_iso/prepare_vdi.sh \
  -D DISABLE_REMOTE_MANAGEMENT \
  -o macOS_10.12.vdi \
   /Applications/Install\ macOS\ Sierra.app/ \
  . && \
../prepare_iso/prepare_ovf.sh \
  macOS_10.12.vdi && \
packer build \
  -var provisioning_delay=30 \
  -var source_path="macOS_10.12.ovf" \
  template.json

[jonanh]

BTW, it worked with 10.12.5 :-)

[timsutton]

Thanks a lot for putting in the time to work on this! Busy with some other things this week but I hope to have more time next week to look at how this can fit in.

The main disadvantage I see to using the OS install here is that it imposes the requirement to build the same major OS version for the VM as what's running on the host. https://github.com/chilcote/vfuse is also a related project that is kind of a combination of what you're talking about (typically using AutoDMG's output as its input) and what this project aims to achieve in terms of the Packer templates.

[lordmuffin]

This works awesome, had to use it to get 10.12.5 working. Should really incorporate this fork.

[timsutton]

Thanks all for the feedback and to @jonanh for providing all the extra documentation! I do intend to incorporate this contribution as soon as I'm able to make the time to give it a quick review.

[jgimenez]

In High Sierra BaseSystem.dmg is not inside the InstallESD.dmg, it is in the same directory, in Installer.app/Contents/SharedSupport/BaseSystem.dmg. You could check for the existence of the file in both paths, then you would have compatibility with High Sierra as well.

[jgimenez]

It seems like this password is not correctly understood in High Sierra. System will boot, Vagrant user exists, but will never log in.

[jonanh]

Thanks @jgimenez for the info! :-) I haven't tried it yet High Sierra. I will try to test it tomorrow.

[timsutton]

About the password, this is true - High Sierra no longer supports the legacy password hash format. A tool like https://github.com/gregneagle/pycreateuserpkg could be used to programmatically generate one which can be used from 10.8 and up.

[jonanh]

I managed to get the prepare_vdi.sh generate an image. I fixed the new location of the BaseSystem.dmg, changed to APFS as the FS for the image and installed the os using the $INSTALLER_PATH/Contents/SharedSupport/InstallInfo.plist file instead of $MNT_ESD/Packages/OSInstall.mpkg package, but unfortunately the generated image doesn't boot for now. :-(

I will continue researching on it next week (I won't be able to work this weekend), but I have left the changes in a branch just in case someone wants to continue testing it.
jonanh#2

Once I get it working I will update the pull request. :-)

[jonanh]

The problem seems to be that hdiutil is not creating an EFI partition anymore in High Sierra

$ hdiutil create -size "1g" -layout GPTSPUD -type SPARSE -fs "APFS" -volname "Macintosh HD" -uid 0 -gid 80 -mode 1775 test.sparseimage
$ hdiutil attach test.sparseimage -nomount
/dev/disk6          	GUID_partition_scheme          	
/dev/disk6s1        	Apple_APFS                     	
/dev/disk7          	EF57347C-0000-11AA-AA11-0030654	
/dev/disk7s1        	41504653-0000-11AA-AA11-0030654	/Volumes/Macintosh HD

[jgimenez]

That’s curious, it did boot for me without changing anything related to the FS, only I couldn’t log in because of the other problem with the user info.

…

On Fri, Sep 8, 2017 at 8:20 AM, Jon Ander Hernández < ***@***.***> wrote: The problem seems to be that hdiutil is not creating an EFI partition anymore in High Sierra $ hdiutil create -size "1g" -layout GPTSPUD -type SPARSE -fs "APFS" -volname "Macintosh HD" -uid 0 -gid 80 -mode 1775 test.sparseimage $ hdiutil attach test.sparseimage -nomount /dev/disk6 GUID_partition_scheme /dev/disk6s1 Apple_APFS /dev/disk7 EF57347C-0000-11AA-AA11-0030654 /dev/disk7s1 41504653-0000-11AA-AA11-0030654 /Volumes/Macintosh HD — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#82 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0xwjd2Ny3Z9o9puIv-RmY1RvkXnR09ks5sgNyggaJpZM4NadrQ> .

[jonanh]

Did you create the image from a High Sierra host? Do you mind checking how does look like the partitions of the generated image?

[jgimenez]

I created the image from a Sierra host. I unfortunately deleted everything, and now I upgraded to High Sierra, so I can't reproduce it, sorry...

[jonanh]

Sorry for the delay, I have been quite busy lately :-(

In the end the problem was very simple, APFS is not supported by Virtualbox, and the EFI partition doesn't contain any boot loader like Linux does. I was assuming that the EFI BIOS wasn't able to read HFS+J partition (which turns out that they are), so I though that the installer was failing to copy something.

I updated the create_firstboot_pkg.sh script to use some of the modules used by Greg Neagle's pycreateuserpkg, and now it should work for High Sierra.

I also simplified the script a little bit, since HighSierra doesn't require the ESD to be mounted. But in the way of the simplification it stopped working with Sierra, so now I have to fix it again :D. I will make sure that it works in both and I will update the branch. :-)

[timsutton]

That's awesome @jonanh, thanks for the update. Would definitely like to incorporate your changes soon :)

[timsutton]

Thank you @jonanh so much for 1) this contribution, the effort to make it DRY along with other scripts and files already in this repo, and the update for PBKDF2 in the password hash logic, and 2) your patience while this PR has sat idle. I finally had some time to run it through and while I didn't get to the point of actually using the results for further work with Virtualbox, everything seemed to work as expected.

Is there anything left that you wanted to do for this, or any attribution in the code that we need to cover for some of the files you added for the PBKDF2 hashing support?

[jonanh]

Hey Tim, it has been a pleasure learning a little bit more about OSX and being able to contribute a small piece to your project! :-)

I borrowed those files from pycreateuserpkg, but though the copyright was cited in the headers of the files, now I am realising that neither pycreateuserpkg nor arc4random have a concrete license specified. I could contact the authors and ask them.

• pbkdf2.py
  Armin Ronacher (BSD 3-clause)
  https://github.com/mitsuhiko/python-pbkdf2

• arc4random.py
  Roland Bracewell Shoemaker (no licence specified)
  https://github.com/rolandshoemaker/py-arc4random

• shadowhash.py
  plistutils.py
  Greg Neagle (no license specified)
  https://github.com/gregneagle/pycreateuserpkg

I have some improvements to the code that I wrote, but I can open a new PR afterwards. I think that I can improve the performance quite a lot by removing some exporting/reimporting steps. Though my priority wasn't the performance, since probably most of the users will create a new base box per major macOS version.

[timsutton]

Well, this clears up the one about pycreateuserpkg - we can attribute it licensed as Apache 2.0.

[timsutton]

I also just found that for a project, I wanted to source the create_firstboot_pkg from a script which was run via Packer directly, and which lived outside this repo. I added the following logic to it so that it would set SCRIPT_DIR differently if it had been sourced:

f0869e0

If you're good for me to merge this PR now I can just add this change back on top of master.

[jonanh]

Sorry for the delay. Thanks for the review and merging! :-)

I wasn't contacted back from the author of arc4random.py about the licence, so I will try to port the BSD's arc4random this weekend.

[timsutton]

Oh, thanks! I'd forgotten about that one. I'll see about trying to fix up attributions meanwhile.