Filesystem Sharing #20

tarik02 · 2021-05-19T07:42:04Z

Hello. I would like to give some ideas and advices about filesystem sharing.

So, since we are using QEMU, lets see which options do we have:

VirtioFS. Looking very cool, seems to have really good performance, but works only on Linux hosts. It is very optimised for using in virtual machines, it even uses DAX (direct access) for files, so there's no need to copy files over network, they're just in the shared RAM between VM and host.
VirtFS (9P). I've tried to use it, but it's incredible slow. Really. Using just git status in shared directory with middle size project takes at least half a minute. I would rather just place files in VM and access them via some remote file access protocol and use vscode with remote access (sad, but they're proprietary). I think that it is so slow because it is sync. Whenever you read some file, do stat call etc. you have to wait for this operation to end.
Just sync filesystem state between VM and host.
Write custom FUSE driver with asynchronous protocol and multithreading support. Theoretically this can be more performant than 9P, but not sure.

The text was updated successfully, but these errors were encountered:

AkihiroSuda · 2021-05-19T09:38:29Z

My current plan:

Step 1: Use 9p over virtio serial: Remove dependency on apt-get install sshfs #1
Step 2: Implement vsock support in QEMU for Darwin, and switch away from virtio serial to vsock
Step 3: Implement virtiofs support in QEMU for Darwin

AkihiroSuda · 2021-05-19T09:40:07Z

In parallel, we can consider supporting mutagen integration as well. IIUC it is used by Docker for Mac as well.

https://mutagen.io/

tarik02 · 2021-05-19T11:57:26Z

Step 3: Implement virtiofs support in QEMU for Darwin

Do you think this is possible? It requires to be able to share mmap'ped memory pages between host and guest. It this is possible, it would be cool.

mroi · 2021-05-21T08:33:54Z

Just so you know: There is a patch for Darwin 9p support developed within the Nixpkgs project. I have adapted it to current QEMU from a patchset originally by Keno Fischer.

The people over at Nixpkgs would certainly be interested in any effort to accelerate file sharing using VirtioFS, which is why I’m now subscribed here.

AkihiroSuda · 2021-05-21T08:38:03Z

Thanks @mroi !

The patch seems proposed to the upstream in 2018, but what's the current status of upstreaming?
Is there / will there be a binary with the patch, that can be installed without nix? For both Intel and ARM. (Would it be possible to crosscompile ARM target on an Intel host?)

AkihiroSuda · 2021-05-21T08:40:49Z

@mroi Aside from 9p, do you know whether somebody is working on supporting vsock?
For reimplementing the even notification system that is currently implemented by running ssh -L /run/user/<UID>/lima-guestagent.sock:ga.sock.

mroi · 2021-05-21T08:50:35Z

I am preparing to propose the revised patch upstream. Have not had sufficient time to push this forward, but I hope to do this next week. It would certainly be good to have this upstream.

Regarding binaries: Nix has a binary cache, but the packages do not run standalone (they depend on other Nix packages). The patch however should apply to the vanilla QEMU sources, so you should be able to just recompile QEMU 6 with it.

I’m not aware of vsock developments. Unfortunately I’m not familiar with QEMU development internals at all. I just happened to have a need for 9p and worked on this patch.

AkihiroSuda · 2021-05-21T08:54:05Z

Thanks!

AkihiroSuda · 2021-05-22T07:05:41Z

The patch seems to have issues 😞 NixOS/nixpkgs#122420 (comment)

AkihiroSuda · 2021-06-10T09:13:55Z

Until we can get virtio-9p-pci for macOS hosts in the QEMU upstream, I was planning to use 9P over virtserial, but it doesn't seem easy as expected 😞

diff --git a/pkg/qemu/qemu.go b/pkg/qemu/qemu.go
index d9b0778..19f0f75 100644
--- a/pkg/qemu/qemu.go
+++ b/pkg/qemu/qemu.go
@@ -154,7 +154,7 @@ func Cmdline(cfg Config) (string, []string, error) {
        // Parallel
        args = append(args, "-parallel", "none")
 
-       // Serial
+       // Legacy Serial
        serialSock := filepath.Join(cfg.InstanceDir, "serial.sock")
        if err := os.RemoveAll(serialSock); err != nil {
                return "", nil, err
@@ -167,7 +167,17 @@ func Cmdline(cfg Config) (string, []string, error) {
        args = append(args, "-chardev", fmt.Sprintf("socket,id=%s,path=%s,server,nowait,logfile=%s", serialChardev, serialSock, serialLog))
        args = append(args, "-serial", "chardev:"+serialChardev)
 
-       // We also want to enable vsock and virtfs here, but QEMU does not support vsock and virtfs for macOS hosts
+       // vport for 9p
+       vportSock := filepath.Join(cfg.InstanceDir, "vport.sock")
+       if err := os.RemoveAll(vportSock); err != nil {
+               return "", nil, err
+       }
+       const vportChardev = "char-vport"
+       args = append(args, "-device", "virtio-serial")
+       args = append(args, "-chardev", fmt.Sprintf("socket,id=%s,path=%s,server,nowait", vportChardev, vportSock))
+       args = append(args, "-device", fmt.Sprintf("virtserialport,chardev=%s,name=lima", vportChardev))
+
+       // TODO: use virtio-9p-pci when QEMU supports it for macOS hosts
 
        // QEMU process
        args = append(args, "-name", "lima-"+cfg.Name)

package main

import (
        "fmt"
        "os"
        "syscall"
)

func main() {
        if err := xmain(); err != nil {
                fmt.Fprintln(os.Stderr, err)
                os.Exit(1)
        }
}

func xmain() error {
        devPath := "/dev/vport5p1"
        mntPath := "/mnt/foo"

        devFd, err := syscall.Open(devPath, syscall.O_RDWR|syscall.O_NONBLOCK, 0600)
        if err != nil {
                return err
        }
        return syscall.Mount(
                "",
                mntPath,
                "9p",
                0,
                fmt.Sprintf("trans=fd,rfdno=%d,wfdno=%d", devFd, devFd),
        )
}

# go run main.go
invalid argument
exit status 1

# dmesg
...
[27353.823239] kernel write not supported for file /vport5p1 (pid: 2243 comm: kworker/3:1)

The error is happening because the vport device somehow wires up both write_iter and write: https://github.com/torvalds/linux/blob/v5.11/fs/read_write.c#L540-L545

I guess I have to inject some pipe as a "shim" fd, or use ssh instead of virtserial.

AkihiroSuda · 2021-06-23T12:35:03Z

virtserial seems slower than ssh :(

virtserial

qemu flags: -device virtio-serial -chardev socket,id=foo,path=foo.sock,server,nowait -device virtserialport,chardev=foo.sock,name=foo

[host]$ socat file:urandom-1G unix-connect:foo.sock

[guest]$ time sha256sum /dev/virtio-ports/foo
real    0m10.022s

ssh

[host] $ cat urandom-1G | lima time sha256sum -
real    0m7.673s

jandubois · 2021-06-23T16:52:54Z

virtserial seems slower than ssh

Given that we seem to be stuck with ssh, at least for now, did you check if selecting a different cipher makes any difference for throughput? It seems like we are using the default:

debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none

I remember reading that aes128-ctr was twice as fast as chacha20-poly1305@openssh.com when running on a CPU with AES instructions, but not sure how much that affects throughput.

I'm somewhat confused though, as even when I add -c aes128-ctr to the ssh command, it still ends up using chacha20-poly1305@openssh.com, so I don't know how to configure it.

AkihiroSuda · 2021-06-28T09:03:25Z

Given that we seem to be stuck with ssh, at least for now, did you check if selecting a different cipher makes any difference for throughput?

Haven't looked into. Help wanted 🙏

AkihiroSuda · 2021-06-28T09:04:55Z

Btw QEMU Samba seems roughly two times faster than sshfs

[root@lima-archlinux ~]# time sha256sum /mnt/smb/ubuntu-21.04-desktop-amd64.iso 
fa95fb748b34d470a7cfa5e3c1c8fa1163e2dc340cd5a60f7ece9dc963ecdf88  /mnt/smb/ubuntu-21.04-desktop-amd64.iso

real    0m15.578s
user    0m7.054s
sys     0m6.259s

[root@lima-archlinux ~]# time sha256sum /tmp/lima/ubuntu-21.04-desktop-amd64.iso 
fa95fb748b34d470a7cfa5e3c1c8fa1163e2dc340cd5a60f7ece9dc963ecdf88  /tmp/lima/ubuntu-21.04-desktop-amd64.iso

real    0m29.707s
user    0m9.244s
sys     0m6.370s

diff --git a/pkg/qemu/qemu.go b/pkg/qemu/qemu.go
index 796c6ee..1524a24 100644
--- a/pkg/qemu/qemu.go
+++ b/pkg/qemu/qemu.go
@@ -193,7 +193,7 @@ func Cmdline(cfg Config) (string, []string, error) {
        // CIDR is intentionally hardcoded to 192.168.5.0/24, as each of QEMU has its own independent slirp network.
        // TODO: enable bridge (with sudo?)
        args = append(args, "-net", "nic,model=virtio")
-       args = append(args, "-net", fmt.Sprintf("user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:%d-:22", y.SSH.LocalPort))
+       args = append(args, "-net", fmt.Sprintf("user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:%d-:22,smb=/tmp/lima", y.SSH.LocalPort))
 
        // virtio-rng-pci acceralates starting up the OS, according to https://wiki.gentoo.org/wiki/QEMU/Options
        args = append(args, "-device", "virtio-rng-pci")

Needs Homebrew/homebrew-core#80171 to be merged (which may take time)

AkihiroSuda · 2021-07-19T08:42:13Z

Samba is now available on homebrew (https://github.com/Homebrew/homebrew-core/blob/master/Formula/samba.rb), so I'm planning to replace sshfs with samba soon.

Samba will be executed with slirp_add_exec: https://github.com/qemu/qemu/blob/4cc10cae64c51e17844dc4358481c393d7bf1ed4/net/slirp.c#L964

So Samba will not listen on any actual TCP port.

cc @jandubois

markomitranic · 2021-07-23T22:43:23Z

@AkihiroSuda That is awesome to hear. From the benchmarks i have seen so far, it is supposedly 2x faster with most workloads.

What would it take for us to support inotify events?

afbjorklund · 2021-09-11T16:14:20Z

It would be interesting to see some benchmarks between sshfs and virtio-9p-pci and nfs and cifs/smb...

The virtfs authors seemed convinced: https://www.kernel.org/doc/ols/2010/ols2010-pages-109-120.pdf

markus-geiger · 2021-11-11T10:10:04Z

On Linux QEMU using virtfs is a lot faster than sshfs and even 9p. If anything is slowed down on macOS due to kernel lockdown/security don't take it as given for Linux users!

-–› Please always consider Linux users in mind since lima itself becomes more and more a tool for cross-platform development (macOS, Linux, WSL/Windows). Especially k8s/k3s is really nyce scenerio! Don't make your decisions based on macOS only! Thank you!

Docker for Desktop on Windows uses 9p (they switched away from SMB which they used before).

Docs for linux: https://www.linux-kvm.org/page/9p_virtio

Aslo check out how wimpy uses hardware-acceleration/virt*-kernel modules in Linux w/ qemu v6.x (yup 6.0+!):
https://github.com/wimpysworld/quickemu/blob/master/quickemu

deviantintegral · 2021-11-11T12:32:11Z

It's not Docker per se, but WSL2 that uses 9P for any WSL VM: https://devblogs.microsoft.com/commandline/a-deep-dive-into-how-wsl-allows-windows-to-access-linux-files/

I've been testing out WSL2 integration for other team members, and the filesystem performance is still not to a point where it's usable for large projects. For example, PHPStorm hangs and crashes trying to index a basic Drupal 9 codebase over the \\wsl$ share: ddev/ddev#3366

To date, mutagen (which is built into ddev) is giving our team the best experience. That work came about because Docker Desktop dropped their alpha mutagen support, and I think it would be really nice if support for it was at the lima (or colima) layer instead of at the next layer up.

I'm not sure it's worth the development effort to only get a relatively small performance improvement. If performance is any more than 20-50% slower than native IO, it's still too slow for most of our uses.

abiosoft · 2022-02-21T07:20:33Z

Looks like there has been some progress/success with 9p and Qemu containers/podman#8016 (comment).

afbjorklund · 2022-02-21T11:04:14Z

It is still enroute to be merged for 7.0, and still available as a patch for 6.x - so no real news from last year, except that more people are actually trying it now

AkihiroSuda · 2022-03-08T06:29:06Z

After the 9p patch (https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg02090.html) gets merged we can switch the default from reverse sshfs to 9p, however, probably we will have to continue maintain reverse sshfs for EL8 family 😞

[root@lima-rocky ~]# grep -i 9p /boot/config-4.18.0-348.el8.0.2.x86_64 
# CONFIG_NET_9P is not set

afbjorklund · 2022-03-08T06:33:15Z

They are waiting for virtiofs support (or nfs over vsock). Waiting, waiting, waiting. It's been years already.

https://access.redhat.com/discussions/1119043

We discussed it as a replacement for VirtualBox Shared Folders (vboxsf), but the outcome was: sshfs

Add 9p support for mounting /home dhiltgen/docker-machine-kvm#2

AkihiroSuda · 2022-03-08T13:13:09Z

The 9p patch is now merged in the master branch of QEMU qemu/qemu@f45cc81

afbjorklund · 2022-03-08T16:25:13Z

Note that it is the 9p for Darwin that is "new", the 9p for Linux has been there for ages

afbjorklund · 2022-03-13T12:02:31Z

I added a "mountType" selector. Theoretically it could also include something like NFS.

Add virtfs/9p mounts, instead of sshocker/sshfs #726

mountType: "reverse-sshfs" (sshocker)

mountType: "9p" (virtfs)

Saeger · 2022-03-14T10:30:01Z

Note that it is the 9p for Darwin that is "new", the 9p for Linux has been there for ages

Yeah, I guess most of us are interested in the Darwin version as we're trying to mount @ MacOS for replacing Docker Desktop for good.

afbjorklund · 2022-03-14T11:27:45Z

The 9p patch is now also merged in brew (qemu 6.2.0_1), so hopefully that will be enough for most Mac users too...

willcohen · 2022-03-14T19:41:50Z

That's right -- the 9p patch has been backported to 6.2 in both brew and nixpkgs at this point and there's no reason why both will have any other issues making it to 7.0 regardless, so I'd consider it widely available.

afbjorklund · 2022-03-14T20:18:16Z

The brew package is insanely large, since it includes some 30+ architectures. But that's another story... (One can compile and install just amd64/arm64 and their firmware, if needed)

0x3333 · 2022-03-15T15:27:25Z

Docker Desktop Edge has a VirtioFS option that is looking very good.

drpebcak · 2022-03-15T19:04:48Z

Docker Desktop Edge has a VirtioFS option that is looking very good.

Looks like that dropped to stable! https://docs.docker.com/desktop/mac/release-notes/#docker-desktop-460

AkihiroSuda · 2022-03-16T01:05:47Z

Docker Desktop Edge has a VirtioFS option that is looking very good.

VirtioFS is still unavailable in the macOS version of QEMU.

Apple's Virtualization.framework supports VirtioFS, but it is not the best fit for Lima, as it lacks support for firmware, qcow2, slirp, and a lot of features of QEMU. And yet it is proprietary.

reneoelke · 2022-04-26T16:10:00Z

Docker Desktop Edge has a VirtioFS option that is looking very good.

Looks like that dropped to stable! https://docs.docker.com/desktop/mac/release-notes/#docker-desktop-460

Yes, it is available a an experimental feature and will be replace the gRPC FUSE implementation (hopefully). Instead of the hypervisor.framework it requires the new virtualization.framework which is available Big Sur. But VirtioFS in Docker Desktop for Mac is incredibly fast. With that feature the experience with developing inside containers is very comfortable. Would be very nice to have it in lima (perhaps optionally with additional libs / runtime options).

ColCh · 2022-09-06T10:31:00Z

There is a new project which provides fuse via nfs, please follow these links for more info:

https://news.ycombinator.com/item?id=32726166
https://www.fuse-t.org/

IMHO, looks very promising. May be, it's would be very nice for lima ?

afbjorklund · 2022-09-06T11:20:48Z

It has some similar licensing issues as macfuse, it seems ?

jandubois · 2022-09-06T17:35:26Z

IMHO, looks very promising. May be, it's would be very nice for lima ?

It isn't open-source, and it isn't even free to use for commercial uses, so FUSE-T is not an option for Lima.

Also nfs is not really that simple to use on macOS: you need to be root to configure /etc/exports to secure the exported directories.

guild-jonathan-kaczynski · 2022-10-11T23:59:49Z

Hello, some interesting conversation in this thread. I was wondering, looking at this from a pragmatic lens, if there was an immediate solution (maybe the 9p patch getting folded in) while the longer term solution is in the works (which sounds like VirtioFS)? Myself, and others, would be thrilled to use Rancher Desktop on a regular basis, which is a downstream package that uses lima, but there's one thing stopping us: rancher-sandbox/rancher-desktop#1209 (mounting volumes). Appreciate any thoughts or feedback.

ryancurrah · 2022-11-22T05:43:33Z

Looks like 6 days ago Lima documented using virtiofs c18ae23

AkihiroSuda · 2022-11-22T06:59:57Z

c18ae23

Yes, this is now available in v0.14.0-beta.0.

henrik242 · 2023-02-28T08:52:45Z

Any updates to this? Looking for a way to solve the missing inotify events in docker-compose volume mounts in Colima and Lima on macOS.

AkihiroSuda · 2023-02-28T08:54:50Z

Any updates to this? Looking for a way to solve the missing inotify events in docker-compose volume mounts in Colima and Lima.

inotify is discussed here:

File system change events (inotify) support for mounted directories #615

This adds support for using virtiofs to mount filesystems on Linux hosts, via QEMU's vhost-user-fs-pci device + the Rust implementation of virtiofsd. In a simple "benchmark" running sha256sum on a copy of the Windows 11 ARM64 VHDK (because it's a large file I randomly had lying around): - reverse-sshfs took ~21s - 9p took ~13-15s - virtiofs took ~6-7s (For comparison, running it directly on the host system took ~5s.) This is marked as "experimental" because it has undergone testing by...me and relies on additional tools installed other than just QEMU. Unfortunately, this does *not* include support for DAX, because that's not merged into upstream QEMU yet, making it rather difficult to test. Ref. lima-vm#20.

This adds support for using virtiofs to mount filesystems on Linux hosts, via QEMU's vhost-user-fs-pci device + the Rust implementation of virtiofsd. In a simple "benchmark" running sha256sum on a copy of the Windows 11 ARM64 VHDK (because it's a large file I randomly had lying around): - reverse-sshfs took ~21s - 9p took ~13-15s - virtiofs took ~6-7s (For comparison, running it directly on the host system took ~5s.) This is marked as "experimental" because it has undergone testing by...me and relies on additional tools installed other than just QEMU. Unfortunately, this does *not* include support for DAX, because that's not merged into upstream QEMU yet, making it rather difficult to test. Ref. lima-vm#20. Signed-off-by: Ryan Gonzalez <git@refi64.dev>

AkihiroSuda added the enhancement New feature or request label May 19, 2021

AkihiroSuda added the priority/high label May 21, 2021

AkihiroSuda mentioned this issue May 21, 2021

Remove dependency on apt-get install sshfs #1

Closed

r2r-dev mentioned this issue May 25, 2021

9pfs directory sharing utmapp/UTM#2551

Closed

AkihiroSuda changed the title ~~Filesystem Sharing~~ Filesystem Sharing (9P) Jun 11, 2021

AkihiroSuda removed the priority/high label Jun 11, 2021

AkihiroSuda mentioned this issue Jun 23, 2021

guestagent: run as the root #72

Closed

AkihiroSuda changed the title ~~Filesystem Sharing (9P)~~ Filesystem Sharing Jun 28, 2021

AkihiroSuda added the expert label Aug 28, 2021

AkihiroSuda mentioned this issue Sep 2, 2021

Replace reverse SSHFS with Samba #188

Closed

afbjorklund mentioned this issue Sep 5, 2021

Support native source folder for volume mount in remote model containers/podman#8016

Closed

afbjorklund mentioned this issue Mar 13, 2022

Add virtfs/9p mounts, instead of sshocker/sshfs #726

Merged

ggustafsson mentioned this issue Jun 14, 2022

chown error on bind mount when trying to launch postgres via docker compose rancher-sandbox/rancher-desktop#1209

Open

refi64 mentioned this issue Jun 15, 2023

qemu: Add virtiofs support on Linux #1628

Merged

Filesystem Sharing #20

Filesystem Sharing #20

Comments

tarik02 commented May 19, 2021

AkihiroSuda commented May 19, 2021

AkihiroSuda commented May 19, 2021

tarik02 commented May 19, 2021

mroi commented May 21, 2021

AkihiroSuda commented May 21, 2021 • edited Loading

AkihiroSuda commented May 21, 2021

mroi commented May 21, 2021

AkihiroSuda commented May 21, 2021

AkihiroSuda commented May 22, 2021

AkihiroSuda commented Jun 10, 2021

AkihiroSuda commented Jun 23, 2021

virtserial

ssh

jandubois commented Jun 23, 2021

AkihiroSuda commented Jun 28, 2021

AkihiroSuda commented Jun 28, 2021

AkihiroSuda commented Jul 19, 2021

markomitranic commented Jul 23, 2021

afbjorklund commented Sep 11, 2021

markus-geiger commented Nov 11, 2021 • edited Loading

deviantintegral commented Nov 11, 2021

abiosoft commented Feb 21, 2022

afbjorklund commented Feb 21, 2022

AkihiroSuda commented Mar 8, 2022

afbjorklund commented Mar 8, 2022 • edited Loading

AkihiroSuda commented Mar 8, 2022

afbjorklund commented Mar 8, 2022

afbjorklund commented Mar 13, 2022 • edited Loading

Saeger commented Mar 14, 2022 • edited Loading

afbjorklund commented Mar 14, 2022 • edited Loading

willcohen commented Mar 14, 2022

afbjorklund commented Mar 14, 2022 • edited Loading

0x3333 commented Mar 15, 2022

drpebcak commented Mar 15, 2022

AkihiroSuda commented Mar 16, 2022

reneoelke commented Apr 26, 2022

ColCh commented Sep 6, 2022

afbjorklund commented Sep 6, 2022

jandubois commented Sep 6, 2022

guild-jonathan-kaczynski commented Oct 11, 2022

ryancurrah commented Nov 22, 2022

AkihiroSuda commented Nov 22, 2022

henrik242 commented Feb 28, 2023 • edited Loading

AkihiroSuda commented Feb 28, 2023

AkihiroSuda commented May 21, 2021 •

edited

Loading

markus-geiger commented Nov 11, 2021 •

edited

Loading

afbjorklund commented Mar 8, 2022 •

edited

Loading

afbjorklund commented Mar 13, 2022 •

edited

Loading

Saeger commented Mar 14, 2022 •

edited

Loading

afbjorklund commented Mar 14, 2022 •

edited

Loading

afbjorklund commented Mar 14, 2022 •

edited

Loading

henrik242 commented Feb 28, 2023 •

edited

Loading