Support Qemu accelerated podman machine on Windows hosts

[arixmkii]

/kind feature

Description

Support Qemu accelerated podman machine on Windows hosts.

• gvproxy supports windows in its latest releases https://github.com/containers/gvisor-tap-vsock/releases/tag/v0.3.0
• There are Qemu windows builds, which supports Hyper-V based acceleration https://qemu.weilnetz.de/w64/ so, -machine q35,accel=whpx:tcg should perform snappy on Hyper-V enabled hosts.

I'm pretty new to podman machine, so, I might be not aware of other external components needed to run podman machine.

I saw that there is support for running podman machine in WSL2 being merged to 4.0.0 RC2. Why this idea still makes sense.

• It will be Fedora CoreOS based;
• It will be fully featured Linux VM;
• It will more precisely match behavior of the podman machine on MacOS;
• It will allow running at least some arm64 images with qemu syscalls for guest architecture on windows hosts.

The downside from WSL2 option would be less efficient host resource management, but in some cases it is not of the concern.

What is missing right now is the command line support for choosing between multiple available vendors for podman machine on a single platform, which I don't think is a thing already (at least from my studying of the source tree).

[rhatdan]

I am fine with this, but we need community to work on it. If you want to implement this and it is simply a CLI change, I would be all for it.

[afbjorklund]

@arixmkii

The main problem was the lack of unix sockets, for the Windows build. Then there were a lot of other "little things", to fix.

Here some old code, if you want to continue with it, https://github.com/afbjorklund/podman/commits/machine-windows

Things like missing "xzcat.exe" and missing $HOME, and so on and so forth.

The official Podman support would be for WSL2, so it (Windows) was abandoned.

• Introduce Windows WSL implementation of podman machine #12503

Like you say, qemu itself should support windows and co-exist with Hyper-V.

[arixmkii]

@afbjorklund Thank you for your input! I have interest to at least try to make it work, but I need to study the code in more details, it will take time.

The official Podman support would be for WSL2

Feel free to close this issue then with this resolution, it could be reopened/recreated later if there is some actual work to base it on.

[afbjorklund]

There was windows support in podman-machine (v1) but it mostly used VirtualBox so the QEMU driver was bare-bones...

It should be possible to use "fifo files" instead of "unix sockets", but it will take some adaption of the go code and qemu flags.

[albertdb]

This would be really nice to have. Currently the only alternative, where WSL2 is not feasible, is using Minikube, which at the current version includes Podman v2.

[afbjorklund]

You could also DIY, using something like Vagrant for the provisioning. But otherwise I think "everyone" is using WSL2 now.

https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/about/supported-guest-os

[arixmkii]

The main problem was the lack of unix sockets

Go has built in support for unix sockets on windows https://go-review.googlesource.com/c/go/+/125456/ which I expect could work just fine for gvproxy and podman binaries.

Mingw added afunix.h last summer (pretty fresh and probably will need to build it from sources to get it into toolchain) https://github.com/mingw-w64/mingw-w64/blob/43e87a27fdb97ae562f7c1e2017c8ce58fef9ee1/mingw-w64-headers/include/afunix.h I believe then Qemu sources could be patched to allow unix sockets in windows build at least behind conditional switch.

Windows support lacks DATAGRAM for unix sockets, but I'm not ready to say if this will be a blocker or not.

missing "xzcat.exe"

Download latest windows build of xzutils https://tukaani.org/xz/ and copy or remane xz.exe to xzcat.exe and it will behave as xzcat, because it checks the command line first argument to choose the behavior. More natural with symlinks, but this way it will work on windows - checked SHA256 of processed qcow2.xz on Win and MacOS hosts.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[arixmkii]

There is a patch submitted to QEMU for enabling builds with af_unix support on Windows https://lists.gnu.org/archive/html/qemu-devel/2022-08/msg00221.html

[arixmkii]

This also looks useful https://lists.gnu.org/archive/html/qemu-devel/2022-07/msg04098.html as the functionality of file descriptors for unix sockets on windows seems to be limited in Go lang/runtime (probably platform limitations).

It will be needed to replace gvproxy qemu_wrapper pattern from podman.

[arixmkii]

Worked with 9p Windows support for QEMU developers. We traced down 3 bugs in the current implementation. There is a hotfix ready for tests. Will try to publish build with FS mount support soon.

[arixmkii]

Example of QEMU Pomdan machine with p9 on Windows

Creating the machine

podman machine init --image-path testing --username core --cpus 4 --memory 8192 -v C:\Temp\PodmanStorage;/home/core/ps;rw -v C:\Temp\PodmanReadonly;/home/core/pr;ro

Here I modified command line argument parsers to use os.PathListSeparator (';' on Windows platform) to simplify usage for demo purposes, I will create a separate issue to work on the parser, which will be acceptable for the mainlining. 2 directories mounted. One RW to /home/core/ps and another RO to /home/core/pr (now I understand that this naming is bad for the demo, but let it stay)

Then start machine as normal (I did with debug logs just in case, but this should not matter)

Then launching interactive terminal inside container with both mounts

podman run -it --rm -v /home/core/ps:/mnt/write -v /home/core/pr:/mnt/read busybox

Work with readonly FS

### Using RO FS

~ # cd /mnt/read/
/mnt/read # cat ReadMe.txt
RO text
/mnt/read # echo "added" > ReadMe.txt
sh: can't create ReadMe.txt: Read-only file system
/mnt/read # echo "added" > Other.txt
sh: can't create Other.txt: Read-only file system

And the with read-write FS

### Using RW FS

/mnt/read # cd /mnt/write/
/mnt/write # cat ModifyMe.txt
sample
/mnt/write # echo "new sample" >> ModifyMe.txt
/mnt/write # cat ModifyMe.txt
sample
new sample
### This is actually a bug
/mnt/write # echo "only sample" > ModifyMe.txt
sh: can't create ModifyMe.txt: Invalid argument
/mnt/write # echo "other sample" > Other.txt
/mnt/write # cat Other.txt
other sample
/mnt/write # rm Other.txt
/mnt/write # cat Other.txt
cat: can't open 'Other.txt': No such file or directory
/mnt/write # mkdir inner
/mnt/write # ls -l
total 0
-rw-rw-rw-    1 nobody   nobody          18 Jan 12 20:55 ModifyMe.txt
drwxrwxrwx    1 nobody   nobody           0 Jan 12 20:59 inner
/mnt/write # ls -l inner
total 0

There discovered bug with the permissions will be reported to people working on QEMU side. There are more known issues with the current implementation:

• content of directories with symlinks (Windows symlinks) can't be enumerated
• content of directories with AF_UNIX sockets (Windows Unix domain sockets) can't be enumerated
• file enumeration is not safe, when content of the directory is modified during enumeration
• can't overwrite the file (this one was discovered during this demo)

Summary would, that in general it works almost like on macOS. Ok, one can't really use -v "$HOME:$HOME" for the machine and then magical -v "$PWD:$PWD" for the container, when working directory is under user home, but this was always like "hiding the actual complexity" thing, not a real feature (though looking like the one).

Will prepare and publish demo builds (both QEMU and Podman) in the coming days.

[arixmkii]

Finally managed to publish pre-built packages https://github.com/arixmkii/qcw/releases/tag/v0.0.8 and updated README file to reflect recent changes.

[arixmkii]

I would say that now we have all the bricks ready and just need to put them together.

• LookPath like impelementations for OS specific executable searches common#1131 - and then need to wait for the new common version to be pulled into Podman tree
• Support for Windows paths in the source position of the volume mounts #17113 - nice to have, but 9pfs implementation will be only be included in future version of QEMU, for now this change will only work if one builds QEMU with patches
• Implement Unix domain socket support for VLAN #17473 - main and most significant part, review pending
• [Feature]: Support Podman Machine provider selection using containers.conf #17116 - will require one more PR, when new common version is pulled into the tree. Based on this commit arixmkii@ce9e930

More in the pipeline:

• Consider adding machine provider prefix to ssh keys #16710 - nice to have usability fix, but is not a blocker.

[arixmkii]

Published pre-built packages including all PRs mentioned above https://github.com/arixmkii/qcw/releases/tag/v0.0.9 There are hopes, that the next time Podman package will be built from the unmodified source tree 🤞

[arixmkii]

Published latest QEMU and Podman builds under https://github.com/arixmkii/qcw/releases/tag/v0.0.11

All included changes applied to Podman are now also submitted as PRs and are waiting for being reviewed:

• Implement Unix domain socket support for VLAN #17473
• Use dynamic identity file names by default #18487
• Enable QEMU Podman machine on Windows #18488

Unfortunately 9pfs on Windows didn't make it to QEMU 8.0.0, so, the patched QEMU build (provided from the release I mentioned) is still needed to experiment with it.

[arixmkii]

Some noteworthy updates:

• managed to make it run nested (with SMP enabled and 4 CPUs) inside Hyper-V Win 11 machine (only tested with Insider Preview for now)

[arixmkii]

I published new builds based on Podman 4.7.0-dev and QEMU 8.1.0 with support for running inside Hyper-V machine https://github.com/arixmkii/qcw/releases/tag/v0.0.12

[arixmkii]

Another incremental release with patches rebased (and ready for review) to latest Podman 4.8.0-dev and QEMU 8.1.1
https://github.com/arixmkii/qcw/releases/tag/v0.0.13

[arixmkii]

New build, now code changes are validated with the newly introduced cross platform e2e Podman Machine test suite.

All future builds will be validated with that test suite. Going forward I will just update the link in this comment to point to latest rebuild if there are no significant functional changes.

https://github.com/arixmkii/qcw/releases/tag/v0.0.16

Updated Jan, 06, 2024: v0.0.16 released moving to QEMU 8.2.0 and Podman 5.0.0-dev

[jgresham]

@arixmkii First, thanks for your work on this! It would be nice for me to remove the WSL2 install flow to use podman in my app on Windows.

How is the stability & performance of Podman on QEMU + Windows looking? It seems like most users home laptops don't have HyperV, so will that be a problem here?

[arixmkii]

@jgresham I can confirm that it works on Win 11 Home with "Virtual Machine Platform" feature enabled and there is no requirement of the full Hyper-V feature installed.

As for performance, I can't comment on this, because after recent struggles I moved all my development into VMs on my Windows laptop, so, I'm running Podman machine inside Hyper-V VM on a portable HW and it is not as snappy as bare metal for sure. You can share some ideas of benchmarks you would like to see and I may be able to run them on my desktop machine, but no promises of timeline here.

As for stability. I use it for assisting me doing my dev work on Windows, which is relatively lightweight scenario, as I'm on Windows only for my personal projects. I try to fix issues, when I encounter them, but I admit I had never stress tested the QEMU machine. I'm running full set of Podman machine e2e tests on the commit I place my published releases on, to at least prevent obvious regressions. I'm also using the same QEMU changes related in Podman app on my MacOS setup just to have it slightly more tested.

[arixmkii]

Everything is moved to the new Podman Machine 5 refactoring, this required rework to multiple parts. The first preview build is provided here (QEMU is still building and will be added in couple of hours): https://github.com/arixmkii/qcw/releases/tag/v0.0.17 I plan to do another build after Podman 5 is released, because the source tree around the machine code is receiving multiple changes and fixes now.

There are still some hopes, that this change #21594 could be accepted to Podman Machine 5, which will dramatically reduce the amount of additional code needed to support Windows.

[hgkamath]

@jgresham :

It seems like most users home laptops don't have HyperV, so will that be a problem here?

just penning for FYI: ANS: Most likely, No, it won't be a problem. For Windows Home edition, there are batch-files that can be found online that augment the feature set of the Windows Home Edition. The batch files just download and install the components that provide the respective feature, which are normally only bundled with the Professional-Edition. The package lists are determined from%SystemRoot%\servicing\Packages\*.mum files, and packages are downloaded by the Windows tool DISM from official sources. The WinOS becomes a configuration unsupported by Microsoft, but it works. The WHPX hypervisor can be used with qemu and virtualbox. WSL also works. WRT the HyperV manager and HyperV-Manager-Manged-Machines, I haven't fully pushed them to the max, but for small experiments that I did with it some years ago, the type-1 and type-2 hyperV-VM-s worked. On Win11-Home, the WSA (which as of this writing is being sunset) also works.

• gpedit-enabler.bat Group Policy Editor
• Hyper-V-Enabler.bat HyperV and HyperV manager

So 'Windows Home' users who want to use WHPX virtualization just have this extra step. One, could also argue that 'Windows Home' users elected to use a non-virtualization OS, but one could also argue that virtualization based software is not only for business and enterprise users, but necessary for home users too. The processor/motherboard has virtualization capabilities that the WinOS Home-edition disallows to use. Microsoft should provide more to its home edition users for a more functional home-use laptop.

[arixmkii]

I finally got time to update my QEMU supporting builds for Windows. Recent changes in the codebase removed 9p filesystem support for QEMU machine, so, there will be no support for filesystem volume mounts (there is no point to restore it as QEMU contributor, who created 9p implementation for Windows don't have throughput to continue and making it mainline supported https://github.com/arixmkii/qcw/releases/tag/v0.0.18). I will still include the patch in my QEMU rebuilds to experiment outside of Podman related activities.

As there is no intention to make it mainline from my side and that changes since 5.0 refactoring has been stabilized, I would probably publish releases only when either QEMU or Podman newer release is available and apply patches onto release revision.

@ podman maintainers - may be it is worth to put the issue in "closed/won't fixed" if this will look better in your journals. Thank you for all the cooperation in working on this :)

Current latest build is available at:
https://github.com/arixmkii/qcw/releases/tag/v0.0.18

[hgkamath]

arixmkii, thanks v much for your contributions, especially the parallel fork with the 9p patches.

just requesting clarification,

• So presently :
  • podman 5.2 official : + qemu-9.0 official installed separately
    • ✅WinOS+whpx+wsl+podman
    • 🗹❓WinOS+whpx+qemu+podman (but no 9p filesystem volume-mounts)
  • arixmkii/qcw (podman+qemu) :
    • ✅WinOS+whpx+qemu+podman+9pfs

(Q) Is the above correct, can I remove the question-mark ?
(A) [EDIT], nope, as of today, can't be done, hence struck-out.

• The qemu 9p maintainer, needs a contribution/patches that addresses mitigating corner case host memory usage concerns.
  • See discussion in gitlab issue 974 [3]

Ref :

1. Default qemu flags for Windows amd64
   Default qemu flags for Windows amd64 #15851
2. Documenting Hyper-V QEMU acceleration settings
   Documenting Hyper-V QEMU acceleration settings #16609
3. gitlab issue 974 Enable virtio-9pfs on windows hosts
   https://gitlab.com/qemu-project/qemu/-/issues/974

[arixmkii]

@hgkamath appreciate your interest in QCW builds! I will try to breakdown all the changes I applied to the releases (I have all patches listed in the repository and the build scripts in a form of GH actions are also available). Applying changes I'm trying to never break or remove any of the original functionality, because it would make me uncomfortable, so, far I could get away w/o changing the base apps, but only extending them. I only do amd64 Windows releases, because this is HW platform I needed them for, I can't confidently comment or compare with other platforms.

There are multiple apps I experimented with, but recently only focused on QEMU and Podman, as they were my priority.

QCW QEMU:

• 9.0.2 Windows amd64
  • with the updated patch for 9p support (it needed rebase since the original one from mailing list was published against older version)
  • with the patch for allowing loading UEFI BIOS with pflash instead of -bios (this is needed for experiments with Lima on Windows)
  • with a small patch to save some cycles during build, no effect on the production code

QCW Podman:

• 5.2.0-dev Windows amd64 - the snapshot of what will become 5.2.0, It was cut around 5.2.0-rc1 and I plan to re-release when 5.2.0 GA is available
  • with the patch to update some Windows related parts in QEMU package, which shows compilation errors in VS Code (but the code is evicted in production build, because QEMU is not added to the list of provider, so no compilation errors in Podman Windowws builds),
  • with the patch to actually enable QEMU provider on Windows
  • with another pending patch to implement support for -v "" arguments in machine init for empty filesystem volume mounts

What providers are available in the rebuild:

• Hyper-V - supports host volume mounts via external process running 9p server on the host and communicating via vsock
• QEMU - doesn't support host volume mounts at all, will show error that virtiofsd server is not available if any mounts are requested, the user is responsible to add -v "" during podman machine init or edit container.conf file to always have empty list of mounts by default.
• WSL - I don't have a lot of experience with this one as it is definitely not my cup of tea, but it should be absolutely the same as in the original Podman releases.

Regarding testing. I don't do testing on the patched QEMU because one would need non trivial setup and good understanding of the project to test it reasonably. I always manually do e2e Podman machine tests for QEMU provider from the same source base, which will be used to cut the release, but I have to do it on my local machine, because one needs virtualization support and it is not available in free GH agents. If I have spare time or feel like I have to do it I will also do tests for Hyper-V or WSL provider just in case, but this is an optional step, because every e2e test run is time consuming w/o having it automated.

I hope this will make things at least slightly more clear what to expect from the builds published under QCW project.

[hgkamath]

hmmm, Thankyou for your detailed reply.

• By comparing pkg/machine/provider/platform_windows.go in both git repos, I conclude that the enabling of the "QemuProvider" for whpx windows platform is not upstreamed. So to answer the question in my previous comment, I strike out the line.
• I think Arthur's idea of having qemu as an alternative to a wsl based podman system is of value.
• In am by no means an expert in podman, I have dabbled a little on using podman on Linux, but penning the following thoughts.
• Essentially, for non-production purposes, I think live host-folder sharing is a luxury that's not always a necessity, but providing a storage to mount is, A storage mount is usually necessary for (a) providing /etc/srvetc/ like configuration for the pod (b) /var/srvdata/ for the pod to save data to
• I think live host folder sharing can be worked around. In order enable the qemuProvider, but without 9pfs mounts, perhaps podman project could consider the following 3 ways to go forward
  • (w0) address the v9fs security problem so that it can be mainlined into windows-qemu
    a dev volunteer with kernel/qemu dev could augment the 9p protocol and the kernel v9fs driver with a new caching mode so that the host memory consumption issue and other security issues are mitigated.
  • (w1) Instead of 9p host-mounts, just attach vdisk images
    • unsure what image fileformat podman uses, I would guess it uses a raw or a qcow2 image. It's possible to schedule a downtime and copy data in and out of a image,
    • (w1b) Additionally, if a VHDX-vdisk with a NTFS partition is used instead, it will make offline data transferring easier as Windows diskmgmt can mount VHDX vdisk files.
  • (w2) Attempt CIFS samba based network filesystem mounts
    • Windows does CIFS natively and easy to configure network shares. In Linux the CIFS mount drivers have long since been mainlined, and CIFS shares can be mounted via the fstab file.
    • this will require the user to have first configured sharable network folder with security
    • configure the container's fstab with the network folder access password
    • and the kernel that boots up withing the container to first get the network up, mount the CIFS volume, then proceed to bring up the rest of the pod/containers.
    • I am guessing that the reason this is unfavorable is because it requires changes to downloaded pods, which perhaps defeats a feature of pods in that they are download, fire-up and forget. The other reason may be that the inside of pods is not a single known layout of Linux distribution, and come in flavors, it could be alpine, fedora, etc. So one can't know from the outside how to configure.
  • (w3) podman could manage a Linux VM with attached images that acts like a network attached storage (NAS). The Linux running inside this VM can share 9p over the network. A software component like nbdkit/winnbd could convert a 9p network service port to a container mount. If host sharing is required, one could ether use samba to share the folder back to the hostOS windows or schedule a downtime to detach the image to do offline data transfer. Additionally, if the image is VHDX with a NTFS partition, it is advantageous as previously mentioned in (w1b).
• Hope that's something to think over.

[arixmkii]

There is definitely a value of having more features, but product maintainers should be realistic about what they can actually support to the full extent. QEMU machine support has been recently removed for masOS hosts. It might look appealing to continue supporting it as requested here #23262 because it definitely has value, but I respect decision of the team to focus on priority supported targets. For Windows these are WSL2 and full Hyper-V. Hyper-V experience is very close to what one can get with QEMU out of the box (it requires elevated shells for some operations, but this is not a blocker for many people. It also requires Windows Pro version to consider full Hyper-V officially supported, but this is for different discussion why some feature are SKU locked). QEMU could be superior for people, who like QEMU and like tinkering with QEMU command line for some specific additional goods, but this is not the target audience.

The Windows support files for QEMU implementation in this repo are more like leftovers, I expect at some point brave enough person will remove them (and I will have to add them back in my patches) as it happened with QEMU files for Darwin platform.

Summarizing the thoughts. I see added value in having QEMU in addition to WSL2 and Hyper-V on Windows, but I can agree that maintaining it will put additional burden on the team. It was sometimes very noisy in the tracker, because of the issues with QEMU releases on Apple silicon Macs and there is no guarantees that it will not happen for Windows hosts. If you are official team behind the project you can't all the time get away saying "this feature is less supported than others", you will be forced to go full in with the support or drop it. It's tough, but it is what we get at the end. No hard feelings. :)

[arixmkii]

Some updates:

• it will not be possible to use 5.2.0 GA release as a baseline as it will be missing some fixes, so, I will continue publishing from 5.3.0-dev at least until 5.3.0 GA is releases.
• I managed to add full machine e2e test run to my github action for building Podman in QCW project, so, now every release published will be verified for machine tests in CI instead of my local machine

[arixmkii]

Starting new release train for Podman 5.3.0-dev and QEMU 9.1.x series. There are hopes that with 5.3.0 GA it will be not needed anymore to go to dev revisions.

• https://github.com/arixmkii/qcw/releases/tag/v0.0.24

With this published I will also clear all old branches in my Podman fork and ask anyone to reference patches in QCW repository for replicating builds. This will also make a number of links posted by me broken in older messages, but the code is very different from them, so, I believe it is for good.

Latest release: v0.0.24 with Podman 5.3.1 (release) with QEMU provider available at install time and QEMU 9.1.2 and support for docker compose and podman compose for QEMU provider