Libvirt driver, for hypervisors on remote machines #2031

afbjorklund · 2023-11-25T12:00:42Z

Description

You can use libvirt for talking to the local libvirtd, but it can also open a remote connection:

https://libvirt.org/remote.html

This makes it interesting for starting the QEMU machine on a local server, instead of localhost.

There is also the qemu://session, but it doesn't really add anything over running qemu-system?

Thinking of adding a new "virt" driver, for using libvirt. The goal is to run qemu remotely.

qemu:///system connects to a system mode daemon. (like rootful)
qemu:///session connects to a session mode daemon. (like rootless)

The libvirt libraries bring a lot of dependencies though, so hide them in an optional .so plugin...

Alternatives

The alternative (to qemu+ssh://) is to ssh to the remote host, and run limactl from there...

https://libvirt.org/uri.html

The libvirt driver would be more interesting when coupled with oVirt, to allocate a VM ?

But that would probably be better as a separate "cloud" driver, using the go-ovirt-client

The text was updated successfully, but these errors were encountered:

afbjorklund · 2023-11-25T12:04:50Z

Initial PR:

Support for Virt driver #2032

Mostly to test the plugin system, it can successfully call the libvirt API without adding to limactl.

INFO[0000] Starting the instance "virt" with VM driver "virt" 
INFO[0000] VERSION: 6000000                             
INFO[0000] Version: 6000000

29M	_output/bin/limactl
4,7M	_output/lib/lima/plugin/virt.so

$ ldd _output/bin/limactl 
	linux-vdso.so.1 (0x00007fff965a5000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f9fc6270000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9fc624d000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9fc6247000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9fc6055000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9fc62b3000)
$ ldd _output/lib/lima/plugin/virt.so 
	linux-vdso.so.1 (0x00007ffd183dd000)
	libvirt-lxc.so.0 => /lib/x86_64-linux-gnu/libvirt-lxc.so.0 (0x00007f96c9eb8000)
	libvirt-qemu.so.0 => /lib/x86_64-linux-gnu/libvirt-qemu.so.0 (0x00007f96c9eb3000)
	libvirt.so.0 => /lib/x86_64-linux-gnu/libvirt.so.0 (0x00007f96c9a5d000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f96c9a3a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f96c9848000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f96c96f9000)
	libcap-ng.so.0 => /lib/x86_64-linux-gnu/libcap-ng.so.0 (0x00007f96c96ef000)
	libyajl.so.2 => /lib/x86_64-linux-gnu/libyajl.so.2 (0x00007f96c96e3000)
	libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f96c96c0000)
	libaudit.so.1 => /lib/x86_64-linux-gnu/libaudit.so.1 (0x00007f96c9694000)
	libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 (0x00007f96c9629000)
	libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f96c961c000)
	libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007f96c960f000)
	libxml2.so.2 => /lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f96c9455000)
	libgio-2.0.so.0 => /lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007f96c9274000)
	libgobject-2.0.so.0 => /lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007f96c9214000)
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f96c90ea000)
	libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f96c90cd000)
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f96c90a0000)
	libapparmor.so.1 => /lib/x86_64-linux-gnu/libapparmor.so.1 (0x00007f96c908b000)
	libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x00007f96c903a000)
	libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f96c8e65000)
	libcurl-gnutls.so.4 => /lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f96c8dd5000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f96c8dd0000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f96c8dc8000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f96c8dad000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f96ca3c4000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f96c8d80000)
	libicuuc.so.66 => /lib/x86_64-linux-gnu/libicuuc.so.66 (0x00007f96c8b9a000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f96c8b7e000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f96c8b53000)
	libgmodule-2.0.so.0 => /lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007f96c8b4d000)
	libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x00007f96c8aed000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f96c8ad1000)
	libffi.so.7 => /lib/x86_64-linux-gnu/libffi.so.7 (0x00007f96c8ac5000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f96c8a52000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f96c89bf000)
	libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f96c8910000)
	libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f96c87da000)
	libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f96c87b9000)
	libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f96c8637000)
	libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f96c8621000)
	libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 (0x00007f96c85e5000)
	libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 (0x00007f96c85ae000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f96c852a000)
	libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f96c8501000)
	librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f96c84e1000)
	libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007f96c8473000)
	libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f96c845e000)
	libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f96c8411000)
	libldap_r-2.4.so.2 => /lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f96c83bb000)
	liblber-2.4.so.2 => /lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f96c83aa000)
	libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007f96c839c000)
	libicudata.so.66 => /lib/x86_64-linux-gnu/libicudata.so.66 (0x00007f96c68d9000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f96c66bf000)
	libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f96c6668000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f96c665e000)
	liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f96c663d000)
	libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f96c651d000)
	libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f96c6247000)
	libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f96c616a000)
	libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f96c6139000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f96c6132000)
	libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f96c6121000)
	libgssapi.so.3 => /lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f96c60dc000)
	libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007f96c60b9000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f96c6096000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f96c608f000)
	libheimntlm.so.0 => /lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f96c6081000)
	libkrb5.so.26 => /lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f96c5fee000)
	libasn1.so.8 => /lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f96c5f48000)
	libhcrypto.so.4 => /lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f96c5f10000)
	libroken.so.18 => /lib/x86_64-linux-gnu/libroken.so.18 (0x00007f96c5ef7000)
	libwind.so.0 => /lib/x86_64-linux-gnu/libwind.so.0 (0x00007f96c5ecd000)
	libheimbase.so.1 => /lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f96c5eb9000)
	libhx509.so.5 => /lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f96c5e6b000)
	libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f96c5d42000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f96c5d07000)

afbjorklund · 2023-11-25T12:13:04Z

`LIBVIRT_DEFAULT_URI=qemu:///session`

Adding support for qemu-session is useful for development and debugging:

But it uses a lot of qemu-specific hacks, rather than the actual libvirt API...

`LIBVIRT_DEFAULT_URI=qemu:///system`

The qemu-system is more "real", but it (libvirtd) cannot access ~/.lima directly.

So that means that all images have to be copied to /var/lib/libvirt/images , etc.

https://libvirt.org/storage.html

afbjorklund · 2023-11-25T12:30:08Z

The actual implementation is mostly an exercise in XML templating. With some API exercises.

https://libvirt.org/formatdomain.html

https://libvirt.org/formatnetwork.html

https://pkg.go.dev/libvirt.org/go/libvirt

vsoch · 2023-11-28T03:49:31Z

@afbjorklund I have a (possibly dumb) question. If lima can support remote machines, could it support connecting more than one remote machine? E.g., bringing up VMs on different hypervisor hosts and then networking them all?

And actually, what if I just brought up llima VMs on separate physical machines, can I expose their ip addresses in some way to connect and have a cluster?

afbjorklund · 2023-11-28T07:12:03Z

I suppose it is possible, but it sounds like a job that would be better for norouter.io or similar dedicated software?

https://norouter.io/docs/getting-started/first-example/

The out-of-the-box experience is that you get one SSH connection to each VM, everything beyond that is "extra"...

And it seems oVirt is deprecated ? (For KubeVirt)

vsoch · 2023-11-28T08:09:24Z

For context, I want to get the same setup (usernetes with flux and VMs) on a small multi-node cluster. The colleague that is helping me setup has been using ovirt with ansible, and instead of starting fresh figuring that out I was wondering if Lima could still be used.

should we try KubeVirt instead for this use case? Still with ansible or something else?

I greatly appreciate your insights, still fairly new to automating setup of VMs beyond APIs provided by cloud builders and Vagrant.

afbjorklund · 2023-11-28T14:48:20Z

I should have mentioned that both oVirt and KubeVirt are out of scope for this issue, about libvirt support...

vsoch · 2023-11-28T16:39:28Z

I thought it might be related to remote machines, so I chose it.

afbjorklund · 2023-11-29T12:13:09Z

The main difference is that they (oVirt/KubeVirt providers) will choose a VM host for you, from "the cloud"...

With this libvirt driver, you will specify an URI like:

uri_aliases = [
  "hail=qemu+ssh://root@hail.cloud.example.com/system",
  "sleet=qemu+ssh://root@sleet.cloud.example.com/system",
]

And it would talk to one machine at a time ($LIBVIRT_DEFAULT_URI).

So the workaround is that we ssh to the server, and talk to qemu-system on it (similar to qemu:///session)

ssh user@rain.cloud.example.com limactl ...

But then it would be a different lima list, there.

It would not blend in with the local instances.

afbjorklund · 2023-11-30T17:13:34Z

Will do another attempt of PoC implementation, but using virsh exec instead of libvirt pkg.

afbjorklund mentioned this issue Dec 5, 2023

Support for Virt driver #2054

Closed

afbjorklund closed this as not planned Won't fix, can't repro, duplicate, stale Jul 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Libvirt driver, for hypervisors on remote machines #2031

Libvirt driver, for hypervisors on remote machines #2031

afbjorklund commented Nov 25, 2023 •

edited

Loading

afbjorklund commented Nov 25, 2023

afbjorklund commented Nov 25, 2023 •

edited

Loading

afbjorklund commented Nov 25, 2023 •

edited

Loading

vsoch commented Nov 28, 2023 •

edited

Loading

afbjorklund commented Nov 28, 2023 •

edited

Loading

vsoch commented Nov 28, 2023

afbjorklund commented Nov 28, 2023

vsoch commented Nov 28, 2023

afbjorklund commented Nov 29, 2023 •

edited

Loading

afbjorklund commented Nov 30, 2023

Libvirt driver, for hypervisors on remote machines #2031

Libvirt driver, for hypervisors on remote machines #2031

Comments

afbjorklund commented Nov 25, 2023 • edited Loading

Description

Alternatives

afbjorklund commented Nov 25, 2023

afbjorklund commented Nov 25, 2023 • edited Loading

LIBVIRT_DEFAULT_URI=qemu:///session

LIBVIRT_DEFAULT_URI=qemu:///system

afbjorklund commented Nov 25, 2023 • edited Loading

vsoch commented Nov 28, 2023 • edited Loading

afbjorklund commented Nov 28, 2023 • edited Loading

vsoch commented Nov 28, 2023

afbjorklund commented Nov 28, 2023

vsoch commented Nov 28, 2023

afbjorklund commented Nov 29, 2023 • edited Loading

afbjorklund commented Nov 30, 2023

afbjorklund commented Nov 25, 2023 •

edited

Loading

afbjorklund commented Nov 25, 2023 •

edited

Loading

`LIBVIRT_DEFAULT_URI=qemu:///session`

`LIBVIRT_DEFAULT_URI=qemu:///system`

afbjorklund commented Nov 25, 2023 •

edited

Loading

vsoch commented Nov 28, 2023 •

edited

Loading

afbjorklund commented Nov 28, 2023 •

edited

Loading

afbjorklund commented Nov 29, 2023 •

edited

Loading