How's the disk performance ?

[yaronsumel]

Thank you for this project. It appears that containerization is the crucial component missing to transform the Macs into viable servers.

Here are a few points:

1. Historically, bind mounting to the host has been extremely slow on Macs for an extended period. While it is less problematic without bind, it is still slow. I am curious to know how the disk performance will be with new containerization. Will it match native NVMe speeds?
2. Perhaps it would be beneficial to include a benchmark in the project that compares Docker/Lima to containerization in various aspects.
3. What about communications between containers on host 1 and containers on host 2? It would be unfortunate not to leverage TB5 80Gbps for such networking.

Thank you!

[afbjorklund]

The performance between Containerization and Lima should be similar, since they are both using Virtualization.

Starting a MicroVM takes around 10x longer than a container, but it is around 100x faster than a VM. But YMMV.

[jglogan]

Historically, bind mounting to the host has been extremely slow on Macs for an extended period. While it is less problematic without bind, it is still slow. I am curious to know how the disk performance will be with new containerization. Will it match native NVMe speeds?

The "bind mounts" (which aren't bind mounts in the Linux sense) are virtiofs mounts. As @afbjorklund implies, those will pretty much identically to the same mounts in Lima/Colima or any other tool using Virtualization.framework.

The named volumes that were added in #362 provide better performance than virtiofs shares from the host, but work slightly differently than those in Docker and others where you can just Linux bind mount from the common VM into the containers. Instead, a named volume is a sparse block file with an ext4 image that mounts into the container. What this implies is that you can either RW mount the volume in exactly one container, or RO mount the volume into multiple containers.

[afbjorklund]

In Lima, those are called "disks". Again, they will be very similar since they are using the same Virtualization.framework

# Lima disks to attach to the instance. The disks will be accessible from inside the
# instance, labeled by name. (e.g. if the disk is named "data", it will be labeled
# "lima-data" inside the instance). The disk will be mounted inside the instance at
# `/mnt/lima-${VOLUME}`.
# 🟢 Builtin default: []
additionalDisks:
# disks should either be a list of disk name strings, for example:
# - "data"
# or a list of disk objects with extra parameters, for example:
# - name: "data"
#   format: true
#   fsType: "ext4"

The named volumes in nerdctl.lima are just bind-mounted from a guest directory, same as it works in docker/podman...

https://docs.docker.com/engine/storage/bind-mounts/

So they have the same performance as the disk used.

[yaronsumel]

By "same performance" you mean as the host disk allows or referring to similar options like docker ? Because I can't get close to any host nvme speeds within containerize solution

[afbjorklund]

I meant that if you mount a /var directory you will get the same performance as the OS disk, if you mount a /Users directory you will get virtiofs performance - or whatever other mount type is being used (like 9p) and if you mount a /mnt directory you will get the same performance as data disk (probably very similar to the OS unless using a different filesystem).

Are you comparing the Darwin host filesystem performance with the Linux guest filesystem performance? That overhead should be up to Virtualization.framework and the virtio paravirtualized driver, I don't what the difference in raw I/O speed is. Most people are interested in network filesystem performance, between the guest and the host, and there we used virtiofs.

[yaronsumel]

@afbjorklund thanks makes sense, I did test raw speed between Darwin to VM linux. Maybe i got bit over-excited for this new container platform to do stuff differently as its native - but as it seems the backbone still is the same Virtualization.framework with all the pros and cons.

[afbjorklund]

You can test with Asahi Linux, if you want native OS

But avoiding the Rosetta should at least give you native Arch