macOS Ventura: Support for running x86-64 binaries with Rosetta 2

[reppners]

Problem Description

Currently on Apple Silicon x86 binaries/containers only can be run with qemu but qemu sometimes crashes or glitches out.

Proposed Solution

With macOS Ventura it becomes possible to apply Rosetta 2 in virtualized environments.
See https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta?language=objc

Additional Information

See the same request in context of Docker Desktop: docker/roadmap#384

[jandubois]

It should probably go into Lima (I already mentioned it in lima-vm/lima#889 (comment)). It is definitely on my radar, as I expect to have a machine running Ventura soonish.

Moved the comment to it's own issue for better visibility: lima-vm/lima#1088

[AkihiroSuda]

Currently on Apple Silicon x86 binaries/containers only can be run with qemu but qemu sometimes crashes or glitches out.

Is Rosetta 2 more stable than QEMU?

[vjanelle]

at least on OSX, you can certainly run things like the JVM on it.

[danielhodder]

Seems like all the required changes have now landed in Lima. Testing the docker engine inside a Lima VM seems to show much better compatibility.

Very much looking forward to when this lands in Rancher Desktop.

[kalavt]

LIMA has support to Virtualization.framework from release v0.14.0

[danielhodder]

Is there anything the community at large could help with to get this into the next release of Rancher Desktop? I am sure there are a lot of people who really are looking forward to this feature, and would be happy to help out if we knew what to do.

[jandubois]

Latest status: #3537 (comment)

[reppners]

I've installed Ventura, Rosetta2 and RD 1.8.0 with settings

rdctl set --experimental.virtual-machine.type vz
rdctl set --experimental.virtual-machine.use-rosetta

and tested with docker run -it --platform linux/amd64 ubuntu to check if rosetta is being used, however I see

root@95a56ee40f11:/# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 10:46 pts/0    00:00:00 /usr/bin/qemu-x86_64 /bin/bash /bin/bash
root        14     1  0 10:47 ?        00:00:00 ps -ef

My guess now is I'm being too early and something is still missing. If otherwise, please let me know 😅

[jandubois]

@reppners I cannot reproduce this (on M1):

$ sw_vers
ProductName:		macOS
ProductVersion:		13.2.1
BuildVersion:		22D68

$ rdctl set --experimental.virtual-machine.type vz
Status: reconfiguring Rancher Desktop to apply changes (this may take a while).

$ rdctl set --experimental.virtual-machine.use-rosetta
Status: UI is currently busy, but will eventually be reconfigured to apply requested changes.

$ rdctl set --experimental.virtual-machine.use-rosetta
Status: no changes necessary.

$ ps -ef|grep qemu
  501 60015 59175   0  9:37am ttys000    0:00.00 grep qemu

$ ps -ef|grep Virtualization
  501 59397     1   0  9:37am ??         0:35.55 /System/Library/Frameworks/Virtualization.framework/Versions/A/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine
  501 60085 59175   0  9:38am ttys000    0:00.00 grep Virtualization

Running amd64 images also works as expected:

$ docker run -it --platform linux/amd64 ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
2ab09b027e7f: Pull complete
Digest: sha256:67211c14fa74f070d27cc59d69a7fa9aeff8e28ea118ef3babc295a0428a6d21
Status: Downloaded newer image for ubuntu:latest
root@411d74350965:/# uname -a
Linux 411d74350965 5.15.96-0-virt #1-Alpine SMP Sun, 26 Feb 2023 15:14:12 +0000 x86_64 x86_64 x86_64 GNU/Linux
root@411d74350965:/# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@411d74350965:/#

However, I cannot tell if this is actually using Rosetta, or emu-static, which is pre-installed in our Alpine ISO image.

However, I find the whole VZ setup not yet ready for real usage; I run into bugs and performance issues that I believe we inherit at least partially from upstream. I just file #4258 and #4259 for 2 of those issues.

[nathanpiper]

I confirmed that out of the box, with the experimental settings enabled QEMU is still used. I'm not sure how Rancher Desktop calls lima to configure the VM, but it seems you need to also set "$LIMA_CIDATA_ROSETTA_BINFMT=true".

It looks like binfmt in the VM isn't configured to register rosetta for handling x86_64 binaries.

Running the following worked for me:

First shell into the VM:
rdctl shell

Then run the following on the VM:

sudo -s
service qemu-binfmt stop
echo ':rosetta:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/mnt/lima-rosetta/rosetta:OCF' >/proc/sys/fs/binfmt_misc/register

Unfortunately, this config isn't persisted after restarting Rancher Desktop / the VM.

[nathanpiper]

On further investigation, it looks like it is configured properly but the boot script (05-rosetta-volume.sh) is running before procfs has started?

In /var/log/lima-init.log:

/mnt/lima-cidata/boot/05-rosetta-volume.sh: line 13: can't create /proc/sys/fs/binfmt_misc/register: nonexistent directory

[getsomebread]

I confirmed that out of the box, with the experimental settings enabled QEMU is still used. I'm not sure how Rancher Desktop calls lima to configure the VM, but it seems you need to also set "$LIMA_CIDATA_ROSETTA_BINFMT=true".

It looks like binfmt in the VM isn't configured to register rosetta for handling x86_64 binaries.

Running the following worked for me:

First shell into the VM: rdctl shell

Then run the following on the VM:

sudo -s
service qemu-binfmt stop
echo ':rosetta:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/mnt/lima-rosetta/rosetta:OCF' >/proc/sys/fs/binfmt_misc/register

Unfortunately, this config isn't persisted after restarting Rancher Desktop / the VM.

Can confirm this is the only way I've been able to get Rosetta working with RD 1.8.1. This Rosetta emulation successfully runs Microsoft SQL Server x86 container successfully on M1, whereas it fails every time with qemu. This does not persist across reboots. A possible solution for persistence was given here, but didn't work for me: lima-vm/lima#1443 (comment)

[LaurentLesle]

ps -ef

@jandubois - Can you run that command in the ubuntu container? Looks like from your screenshot you executed the command on your host and not within the container?

I can reproduce

And on my MAC M1

[LaurentLesle]

@getsomebread can you expand or share the steps you did to get it working?

[getsomebread]

@getsomebread can you expand or share the steps you did to get it working?

Sure, it requires a combination of steps from two earlier posts here. I did this with Rancher Desktop 1.8.1.

# Ensure you are on macOS 13/Ventura
sw_vers

# Enable experimental features, then wait for Rancher Desktop's VM to restart
rdctl set --experimental.virtual-machine.type vz
rdctl set --experimental.virtual-machine.use-rosetta

# Extra REQUIRED steps, must be done every time the VM restarts until properly fixed by developers
# Open shell into the RD VM
rdctl shell
# Become root
sudo -s
# Stop the qemu service
service qemu-binfmt stop
# Manually set this obscure setting so Rosetta works for the VM
echo ':rosetta:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/mnt/lima-rosetta/rosetta:OCF' >/proc/sys/fs/binfmt_misc/register

[LaurentLesle]

echo ':rosetta:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/mnt/lima-rosetta/rosetta:OCF' >/proc/sys/fs/binfmt_misc/register

For some reasons the echo is failing on my machine...

  ~  sw_vers
ProductName:		macOS
ProductVersion:		13.3.1
BuildVersion:		22E261
   ~  ~/.rd/bin/rdctl set --experimental.virtual-machine.type vz    
Status: no changes necessary.
   ~  ~/.rd/bin/rdctl set --experimental.virtual-machine.use-rosetta
Status: no changes necessary.
   ~  ~/.rd/bin/rdctl shell                                         
lima-rancher-desktop:/Users/$ sudo -s
lima-rancher-desktop:/Users/# service qemu-binfmt stop
 * Unregistering QEMU binaries in binfmt misc ...                                                                                                                                                                                                                                                                                                                                                             [ ok ]
lima-rancher-desktop:/Users/# echo ':rosetta:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/mnt/lima-rosetta/rosetta:OCF' >/proc/sys/fs/binfmt_misc/register
ash: write error: File exists

[valente]

ash: write error: File exists

fyi I've been following the thread and in my first attemp I got the same error. I restarted rancher and retried and it worked. Now I see the Virtualization.framework at the top when I run the image. 🎉

[LaurentLesle]

Yes after a restart of rancher-desktop I can sudo, stop and perform the copy. I then restart the qemu-binfmt but still get the quemeu when running ps ef

docker exec -it f1a2142e95f3 /bin/zsh -c "ps ef"
PID TTY STAT TIME COMMAND
1 pts/0 Ssl+ 0:00 /usr/bin/qemu-x86_64 /usr/bin/tfc-agent

[jandubois]

Will be fixed by lima-vm/alpine-lima#103, lima-vm/lima#1580, and #4881.

[reppners]

Thanks for making this happen! Could be quite a big quality of life improvement for my daily work 👍

❯ docker run -it --platform linux/amd64 ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
6b851dcae6ca: Pull complete 
Digest: sha256:6120be6a2b7ce665d0cbddc3ce6eae60fe94637c6a66985312d1f02f63cc0bcd
Status: Downloaded newer image for ubuntu:latest
root@29b7e990e566:/# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 13:21 pts/0    00:00:00 /mnt/lima-rosetta/rosetta /bin/bash
root        10     1  0 13:21 pts/0    00:00:00 /usr/bin/ps -ef
root@29b7e990e566:/#