Cannot start VMs on WSL2

[andrelop-zz]

Using a Debian 10/buster VM running under WSL2 here :

root@DESKTOP-G37QQGG:~/ignite# cat test-vm.yaml
apiVersion: ignite.weave.works/v1alpha1
kind: VM
metadata:
  name: test-vm
spec:
  image:
    ociClaim:
      ref: weaveworks/ignite-ubuntu
  cpus: 2
  diskSize: 3GB
  memory: 800MB
root@DESKTOP-G37QQGG:~/ignite#
root@DESKTOP-G37QQGG:~/ignite# ignite run --config test-vm.yaml
INFO[0000] Removed VM with name "test-vm" and ID "b24918cd3b168a3b"
command ["dmsetup" "create" "ignite-b24918cd3b168a3b-base"] exited with "device-mapper: reload ioctl on ignite-b24918cd3b168a3b-base  failed: Invalid argument\nCommand failed.\n": exit status 1
root@DESKTOP-G37QQGG:~/ignite#

Also, dmes gives me this error while the previous command is tried :

[Wed Jul 10 18:03:37 2019] device-mapper: table: 253:0: zero: unknown target type
[Wed Jul 10 18:03:37 2019] device-mapper: ioctl: error adding target to table

These are all the devicemapper related kernel config options provided by the Microsoft WSL2 kernel :

CONFIG_BLK_DEV_DM_BUILTIN=y
# CONFIG_DM_MQ_DEFAULT is not set
# CONFIG_DM_DEBUG is not set
# CONFIG_DM_UNSTRIPED is not set
# CONFIG_DM_CRYPT is not set
# CONFIG_DM_SNAPSHOT is not set
# CONFIG_DM_THIN_PROVISIONING is not set
# CONFIG_DM_CACHE is not set
# CONFIG_DM_WRITECACHE is not set
# CONFIG_DM_ERA is not set
# CONFIG_DM_MIRROR is not set
# CONFIG_DM_RAID is not set
# CONFIG_DM_ZERO is not set
# CONFIG_DM_MULTIPATH is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_UEVENT is not set
# CONFIG_DM_FLAKEY is not set
# CONFIG_DM_VERITY is not set
# CONFIG_DM_SWITCH is not set
# CONFIG_DM_LOG_WRITES is not set
# CONFIG_DM_INTEGRITY is not set
andrelop@DESKTOP-G37QQGG:~$

My guess is that the WSL2 kernel is not enabling some devicemapper related config.

[twelho]

Try to run modprobe dm_zero and see if it fixes the issue, but based on the Kconfig you most likely need CONFIG_DM_ZERO and CONFIG_DM_SNAPSHOT set. I don't know if the WSL2 kernel can be recompiled to enable these options.

[luxas]

Most probably you need those kernel options. If you can reproduce with a new kernel (as per microsoft/WSL#4165 (comment)), we can add it to https://github.com/weaveworks/ignite/blob/master/docs/dependencies.md#host-requirements

Would be super exciting to get Ignite working on WSL 😄

[andrelop-zz]

Hello

Most probably you need those kernel options. If you can reproduce with a new kernel (as per microsoft/WSL#4165 (comment)), we can add it to https://github.com/weaveworks/ignite/blob/master/docs/dependencies.md#host-requirements

Would be super exciting to get Ignite working on WSL 😄

Yes, I had to rollback on using Windows Insider Preview temporarily due to some issues I was having with VirtualBox VMs not working at all on my laptop (Hyper-V being enabled for WSL2 and VirtualBox not working when VBox is enabled).

I'll try to get into Insider Preview to enable WSL2 again this weekend and see if I can get a customized kernel with the needed options compiled in.

[liwei]

Doesn't ignite relies on KVM? AFAIK, WSL2 is a lightweight hyper-v vm by itself and it means hyper-v support nested virtualization?

[luxas]

Doesn't ignite relies on KVM?

Yes, Firecracker is a KVM implementation.

AFAIK, WSL2 is a lightweight hyper-v vm by itself and it means hyper-v support nested virtualization?

I'm pretty sure that Hyper-V supports nested virtualization, but I'm unsure if that knob is exposed in WSL2. That is indeed a requirement.

[andrelop-zz]

Hello,

Try to run modprobe dm_zero and see if it fixes the issue, but based on the Kconfig you most likely need CONFIG_DM_ZERO and CONFIG_DM_SNAPSHOT set. I don't know if the WSL2 kernel can be recompiled to enable these options.

Both options are not enabled by default in the kernel provided by the WSL2 Debian VM :

root@DESKTOP-G37QQGG:~# zcat /proc/config.gz | egrep 'DM_(ZERO|SNAPSHOT)'
# CONFIG_DM_SNAPSHOT is not set
# CONFIG_DM_ZERO is not set
root@DESKTOP-G37QQGG:~#

I'll try to figure out how one could provide a custom compiled kernel to the WSL2 VM with these options enabled.

[andrelop-zz]

I tried building a custom kernel but the WSL2 VM cannot start after I change the C:\Windows\System32\lxss\tools\kernel using the vmlinux kernel produced by the kernel build process.

What I did :

• Downloaded the last release on microsoft/WSL2-Linux-Kernel github repo (https://github.com/microsoft/WSL2-Linux-Kernel/archive/v4.19.57.tar.gz)
• Unpacked it and used the provided kernel config on https://github.com/microsoft/WSL2-Linux-Kernel/blob/master/Microsoft/config-wsl
• Ran make pointing to this config (ie. using make KCONFIG_CONFIG=config.wsl)
• At the end of the build, renamed the existing C:\Windows\System32\lxss\tools\kernel file to something else (ie. C:\Windows\System32\lxss\tools\kernel.old) and renamed/moved the vmlinux generated file as C:\Windows\System32\lxss\tools\kernel
• Forced all the WSL2 VMs shutdown using wsl --shutdown on PowerShell as Administrator
• Started the WSL2 VM again, but it did not boot
• Restored the previously renamed file in order to get the VM booting again

Not sure what else I could try or if what I did was really what needed to be done.
Will try to ask the WSL2 guys on microsoft/WSL#4165 and report back.

[PatrickLang]

Well, I tried kvm-ok under WSL2 and got this:

INFO: Your CPU does not support KVM extensions
KVM acceleration can NOT be used 

Just a FYI - I'm not actively pursuing this I just followed from WSL#4165 to see what ignite was

[chanwit]

I did this egrep -c '(svm|vmx)' /proc/cpuinfo and it returned 0.
Hopefully Microsoft's Taylor Brown could look into it for us.

https://twitter.com/chanwit/status/1158673722309410816

[gaetanww]

I've been able to use kvm with WSL2 with the last windows insider build. But ignite is not working when running the command:

ignite run weaveworks/ignite-ubuntu \
                --cpus 2 \
                --memory 2GB \
                --ssh \
                --name my-vm

I also get the error command ["dmsetup" "create" "ignite-2686ebc1815eb94c"] exited with "device-mapper: reload ioctl on ignite-2686ebc1815eb94c failed: No such device\nCommand failed.\n": exit status 1.
@twelho I verified that CONFIG_DM_ZERO and CONFIG_DM_SNAPSHOT are enabled.

I also get the following warning from containerd that seems related:
WARN[2020-06-10T10:26:40.083542000+01:00] failed to load plugin io.containerd.snapshotter.v1.devmapper error="devmapper not configured"

[stealthybox]

Thanks all for your comments. I began getting ignite working in WSL2.

Windows Fast Ring Build 19631+ is needed to use the undocumented nestedVirtualization option in .wslconfig:
microsoft/WSL#4193 (comment)

I'm using windows build 20150 on an Intel CPU. This build also supports AMD nested virt.

I'm able to build ignite in WSL2 using an Ubuntu distro and docker-desktop for Windows' WSL2 backend.
The devicemapper errors when creating an ignite VM are reproducible.

Checking dmesg for errors roughly indicated that CONFIG_DM_ZERO needed to be enabled.
We also thought CONFIG_DM_SNAPSHOT should be included:

CONFIG_DM_SNAPSHOT=y
CONFIG_DM_ZERO=y

These edits can be made on top of your existing WSL2 kernel config: zcat /proc/config.gz

After compiling our configured kernel from https://github.com/microsoft/wsl2-linux-kernel and configuring .wslconfig to boot nestedVirtualization=true on our new kernel, we were still getting errors from dmsetup, because ignite relies on /dev/mapper to contain symlinks of the device names to their numbered counterparts, /dev/dm1,dm2,etc...

WSL2 does not bind mount /dev/mapper from the host into the distro containers so the named symlinks are not available.
We were able to create these symlinks manually and get the ignite VM's created with the proper backing devices.

While the symlinks are not available in WSL2, the /dev/mapper/control device is.
It's possible to calculate these device name:id mappings using ioctls on the control device.
We can modify ignite in the future to do this.
This will probably remove the dmsetup dependency.
It will also solve some issues with running ignite in containers in general. (#400 (comment))

The last problems were actually starting the VM's.
I found that running containerd as root wasn't quite enough using an Ubuntu 20.04 WSL2 container.
Containerd could not find a cgroup hierarchy in /sys/ or network namespace info in /proc/.
Running the ignite client and containerd inside of a systemd namespace fixes these problems.
I created my systemd namespace using: https://github.com/damionGans/ubuntu-wsl2-systemd-script

There may be a more minimal way to meet these /sys and /proc requirements without running systemd.
We could look at the docker-desktop WSL2-backend source for inspiration.

Once we remove the dependency on /dev/mapper symlinks, ignite should work in WSL2 as well other privileged containers provided by docker/podman/kubernetes/etc.

[stealthybox]

I've discovered that these /dev/mapper symlinks are maintained as a convenience by the userspace udevd.
This usually runs on the host, but you can also run it in containers (such as a WSL2 distro).

While looking at removing our dependency on these symlinks, I discovered that we call dmdetup in the ignite-spawn sandbox and our capabilities and LSM config may not be allowing that to work, so that's one related blocker.

I also found two container focused libraries for devicemapper; one uses ioctl's and the other exec's dmsetup:

• https://github.com/moby/moby/tree/805452a/pkg/devicemapper
• https://github.com/containerd/containerd/blob/3887053/snapshots/devmapper/dmsetup/dmsetup.go

Here is a justification for wrapping dmsetup: containerd/containerd#2988 (comment)

[stealthybox]

These patches remove the main blockers/caveats with running ignite under containers (like WSL2 distros):

• Exec blkid to getUUID() instead of relying on udevd symlinks #652
• Fix in-container device nodes for containerd runtime #653
• Configure dmsetup to fallback to managing device nodes without udevd #654

We will work on making the experience of booting a modified WSL2 kernel easier (either by publishing a build or maintaining sources) and work on documenting the setup more fully.

The gist is that you currently need:

• Fast Ring Windows Build for nestedVirt /w WSL2
• Updated WSL2 kernel enabling devicemapper deps
• cgroup mount modifications + sys and proc
• backgrounded/daemonized containerd

If you meet these main environment dependencies, ignite should run or be very close to running .
I'm currently developing ignite with this setup, and it works well :)

[DarkGhostHunter]

Does Ignite works with WSL2 at time of writing?