Support for FreeBSD guests ?

[afbjorklund]

Description

There was some question in Slack, about supporting also FreeBSD guests.

I guess it would be "freema" or something, and not just "lima", but it is doable...

There are qcow2 images for all architectures, but it does involve adding OS.

I was just doing some experiments, so thought I would raise the question first.

[afbjorklund]

Example:

os: "FreeBSD"

images:
- location: https://download.freebsd.org/ftp/snapshots/VM-IMAGES/14.0-CURRENT/amd64/Latest/FreeBSD-14.0-CURRENT-amd64.qcow2.xz
  arch: "x86_64"
- location: https://download.freebsd.org/ftp/snapshots/VM-IMAGES/14.0-CURRENT/aarch64/Latest/FreeBSD-14.0-CURRENT-arm64-aarch64.qcow2.xz
  arch: "aarch64"

video:
  display: "default" 

[afbjorklund]

It is supposed to have support for OCI runtime:

https://github.com/samuelkarp/runj

There should be support, since containerd 1.5:

https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-with-containerd-1-5/

[jay7x]

Hi! I was the original requestor on Slack. My need is to run a Puppet module acceptance tests on a FreeBSD. Lima is really attractive here because it's headless and scriptable. People are using VirtualBox (with or w/o vagrant) to do this usually. But after I got the M1-based Macbook I lost the ability to do this. In my case there is no real urgency but it'd be really nice to have. I'll try it on a coming week and will share the experience here.

[afbjorklund]

The support for linux containers (on freebsd) seems to have merged as well, in 1.7.

https://productionwithscissors.run/2022/09/04/containerd-linux-on-freebsd/

[afbjorklund]

Short list of portability issues:

• There is no /run, only /var/run (added variable)
• There is no /bin/bash, only /usr/local/bin/bash. (added variable)

And of course: s/linux/freebsd/

The LocalPorts support for guestagent can be implemented using sockstat.
Note: the output of the sockstat command is subtly different, between the OS.

• Investigating guestagent on FreeBSD, for containerd with runj #1509

The sshfs options for "fusefs" does not have allow_other, unfortunately:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215797

Not sure if -o slave works, but there is some -o passive flag instead ?
Also the kernel module is not loaded by default, needs /boot/loader.conf

But otherwise, it seems to be working:

   sudo mkdir -p /mnt/lima-cidata
   sudo mount -t cd9660 /dev/cd0 /mnt/lima-cidata
   sudo cp /mnt/lima-cidata/meta-data /run/lima-ssh-ready
   sudo install -m 755 /mnt/lima-cidata/lima-guestagent /usr/local/bin/lima-guestagent
   sudo -b lima-guestagent daemon
   sudo cp /mnt/lima-cidata/meta-data /run/lima-boot-done

Created the user manually, for now.

$ limactl shell freebsd uname
To see the output from when your computer started, run dmesg(8).  If it has
been replaced with other messages, look at /var/run/dmesg.boot.
		-- Francisco Reyes <lists@natserv.com>
FreeBSD

Probably want to disable those "tips"...

# $FreeBSD$
#
# .login - csh login script, read by login shell, after `.cshrc' at login.
#
# See also csh(1), environ(7).
#

# Query terminal size; useful for serial lines.
if ( -x /usr/bin/resizewin ) /usr/bin/resizewin -z

# Display a random cookie on each login.
if ( -x /usr/bin/fortune ) /usr/bin/fortune freebsd-tips

[XVilka]

Asked this some time ago: #1096

Also, since QEMU 8.0 there is a guest agent support for *BSD systems:

• https://www.qemu.org/docs/master/interop/qemu-ga.html
• https://wiki.qemu.org/ChangeLog/8.0#Guest_agent

[afbjorklund]

Thanks for the reminder. I used QEMU 4 ☺️

• *BSD support (VM) #1096

[afbjorklund]

The support is quite rough, but nerdctl and friends (containerd and runj) are available in the default pkg install.
It does require setting up a ZFS disk and pool, but this can be done using the limactl disk feature (/dev/vtbd1)

[anders@freebsd ~]$ sudo nerdctl version
WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH 
WARN[0000] unable to determine runc version: exec: "runc": executable file not found in $PATH 
Client:
 Version:	v1.2.1
 OS/Arch:	freebsd/amd64
 Git commit:	<unknown>
 buildctl:
  Version:	

Server:
 containerd:
  Version:	1.7.0
  GitCommit:	unknown
 runc:
  Version:	
[anders@freebsd ~]$ runj --version
runj version v0.0.1-dev ()
go: go1.20.3
[anders@freebsd ~]$ sudo nerdctl info
Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: 1.7.0
 Storage Driver: native
 Logging Driver: json-file
 Cgroup Driver: 
 Cgroup Version: 
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: zfs native
 Security Options:
 Kernel Version: 14.0-CURRENT
 Operating System: FreeBSD 14.0-CURRENT
 OSType: freebsd
 Architecture: amd64
 CPUs: 0
 Total Memory: 0B
 Name: freebsd
 ID: ebcf5f96-01a3-4f40-a063-e3f64163c948

You can run native (freebsd) containers, but so far I have not been able to run linux containers (issues mounting the fake /proc ) even though linux binaries (like limactl) do work on the VM (after enabling linux service and centos-7)

[afbjorklund]

https://www.freebsd.org/status/report-2023-01-2023-03/#_freebsd_as_a_tier_1_cloud_init_platform

One could probably do a simple user-data parser calling pw etc, before the python version is up

[afbjorklund]

Here is an example of a freebsd container image:

public.ecr.aws/samuelkarp/freebsd:13.1-RELEASE

[afbjorklund]

Currently the workaround for missing cloud-init is to open a GUI console.

video:
  display: "default"

From there, it should be possible to log in as root once the system is booted.

The template is using FreeBSD 14.0-CURRENT which is the upcoming release.

---

Telling lima that the ssh setup is complete:
#1508 (comment)

Instructions for adding additional users:
https://docs.freebsd.org/en/books/handbook/basics/#users-synopsis

Instructions for adding containerd packages:
https://docs.freebsd.org/en/books/handbook/ports/

Instructions for setting up a ZFS disk pool:
https://docs.freebsd.org/en/books/handbook/zfs/

[afbjorklund]

Similar work for Podman:

• Allow specifying a guest OS in podman machine init. containers/podman#19939

[fwilhe2]

Hi @afbjorklund, are there any updates on this? I think this would be a great way to tinker with FreeBSD.

[afbjorklund]

I don't have any further updates, but could do a rebase of the existing branch (#1509) if it helps anyone?

I guess 14.0 is out now, but I don't know if runj is more mature... Or if it is going to be in containerd 2.0

https://www.freebsd.org/releases/14.0R/announce/ (2023-11-20)

https://www.freebsd.org/status/report-2024-01-2024-03/cloud-init/

[glloydNAT]

I'm pretty excited about this idea but... what do you think about adding support for netbsd as well?

[afbjorklund]

Not sure there are any official cloud images, would need to use e.g. https://bsd-cloud-image.org/

What would a NetBSD image be used for? For FreeBSD, there was the concept of testing runj etc

If it is just to run a virtual machine with BSD, then you could use https://mac.getutm.app/gallery/

[glloydNAT]

I realize there may not be an OCI-compatible runtime native/exclusive to NetBSD and that, therefore, it may fall outside the purview of this project...

I am interested, however, in NetBSD's veriexec functionality as it could relate to running containerized apps. Similarly, a lock-down mechanism like securelevel.

I don't know.. just a thought.