Feature Discussion: distro subsystem

[dcantah]

Background

Due to the popularity of projects like WSL, Lima and many others in this realm, it's clear that there's appetite for a container/VM hybrid solution that plops you in a Linux environment as fast as possible, while being effortless to get started, highly customizable, and integrated with your host system in clever ways.

While our project is currently tailored around running single containers, and that will continue to be the main focus, we've been mulling over a new experimental surface that would complement this functionality and that we already have most of the building blocks for from the rest of the project. We've been thinking of naming this new feature distro.

What is distro?

The distro feature would provide a more full featured VM-like experience, giving you (subject to change, but our current thoughts):

• Instant Linux environments: Boot into Ubuntu, Debian, Alpine, or any other distribution in seconds. The distributions would just be container images like we're all used to, but the main difference being the set to choose from would be more full featured images with non-stripped down userlands, and we'd by default try and run any init system in the image (/sbin/init).

• Persistent workspaces: Unlike ephemeral containers, distros maintain state between sessions. Writes are persistent in other words.

• Seamless integration:

  1. DNS name by default on every distro, making talking to network services inside a breeze.
  2. Host filesystem access by default. /Users exposed in the distro as /Users (or wherever you please). Read-only by default, but we would have ample flags to tweak where to expose this, r/o vs. r/w, and many others.
  3. Automatic ssh agent forwarding.
  4. Automatic user creation. Every distro launched would have a default user created that is named after your macOS username.
  5. (TBD) Run macOS binaries in the distro. Through some clever tricks with binfmt and a process on the host that we can talk to, we can launch macOS binaries on the host and relay the output to your distros shell so you don't need to leave the terminal pane you're working in to run a quick binary on the host.

• Simple management: Straightforward commands to manage your Linux environments.

Interface brainstorm

We're envisioning commands like the following:

container distro list                                                # List available distributions
container distro create mycoolregistry.foo/ubuntu:24.04 ubuntu24.04  # Create a new distro from an image.
container distro run --name ubuntu24.04                              # Start a distro (with no arguments gives a login shell)
container distro run --name ubuntu24.04 bash                         # Run a single command in a distro
container distro shutdown ubuntu24.04                                # Gracefully shutdown the distro and virtual machine
container distro remove ubuntu24.04                                  # Remove a distro
container distro set-default alpine                                  # Set the default distro used for `run` if `--name` isn't provided

Questions for the Community

We'd love your feedback:

1. Is this something you'd use? Would this be valuable to your workflow?
2. What features are must-haves?

• Automatic port forwarding?
• Shared filesystem mounts?
• Snapshot/backup capabilities?

3. How would you use this differently from containers? What use cases would this unlock for you?

---

Please share your thoughts! Whether you'd be an eager early adopter, or you think this doesn't fit the project's direction 😄, we want to hear from you.

[afbjorklund]

This feature is useful in WSL (distro integration, port forwarding, filesystem sharing), and something also Lima does.

In fact you can run Lima with the WSL2 driver, and it will do the user creation and those other missing pieces for you...
You can do it yourself, but then you also have to do the provisioning and forwarding those other things in the template.

Nowadays WSL even does GUI (with WSLg), and that would be the real killer feature for this "distro" integration as well?

Meanwhile we have an experimental AC/DC driver so that you can use Lima to customize your "container" (Apple or Kata)
When you do the container run or docker run you get root by default, so we create the user and the mounts and so on.

• The Lima container drivers lima-vm/lima#3829

[afbjorklund]

Since virtiofs mounts were broken in container at the time I was trying it out, I went with the default reverse-sshfs mounts (that only require fuse on client and sftp on host). But now it should be able to just do -v /Users:/mnt/Users instead.

Also most external drivers are not fully working yet after the major upgrade from Lima 1.2 to Lima 2.0 that just happened, so only the WSL2 driver is fully operational when it comes to containers. The QEMU and VZ drivers do work for virtual machines.

[saehejkang]

If distro is meant to be a "container/VM hybrid", would having some sort of "consoling" connection be in scope of this work? If the user spins up the a container with something like a CentOS image, should there not be a way to connect to the console in the web browser? Maybe using websockets and noVNC? This is all dependent on if the container images actually have GUI support and I know that it was mentioned above for WSL.

Nowadays WSL even does GUI (with WSLg), and that would be the real killer feature for this "distro" integration as well?

But what about GUI support for lima? I found that there is a project called lima-gui. Funny enough, I realized that @afbjorklund is the creator of it so maybe he can speak to it more on how feasible integration is?

[afbjorklund]

It is possible to display a SSH terminal connection in a web browser, using something like xTerm.js and ssh...
There is a prototype (non-production) integration here: https://github.com/afbjorklund/go-ssh-web-client (WebSocket)

You can also show a VNC graphical connection in the web browser, using something like noVNC and vnc...
At one point I was thinking of bundling these tools with Lima, but in the end we did not. lima-vm/lima#1003 (comment)

So currently we just use the external ssh and vncviewer.

The user can still start the web clients, instead of those.

But what about GUI support for lima?

There is built-in support, for opening a display in the Linux VM. You can either use the default from the hypervisor, which looks like a regular window. But on some platforms, that might consume a lot of resources (so the default is to keep it hidden).
Then there is support for starting a VNC server instead. The Lima GUI application has a built-in SSH client and a built-in VNC viewer, so it connects using these. But it will be in a window, and the graphic will be started from the client (not the server).

• Providing graphical lima example lima-vm/lima#1002

"Hello World": limactl start template:experimental/vnc

With the built-in support in "WSLg", you can start an X11 application from the Linux command line and the application will show up next to the native Windows host applications. There is no RDC or anything needed, it is integrated with the desktop GUI.

https://github.com/microsoft/wslg

It's not a normal "container" feature.

[afbjorklund]

I don't see why you need to use new commands, for these "distro" containers?

container distro run --name ubuntu24.04 # Start a distro (with no arguments gives a login shell)
container distro run --name ubuntu24.04 bash # Run a single command in a distro

container distro shutdown ubuntu24.04 # Gracefully shutdown the distro and virtual machine
container distro remove ubuntu24.04 # Remove a distro

Shouldn't these just use "stop" and "rm" commands, like with any other container.

Commands such as "list" and "create" are still convenient wrappers, to the usual.

[jglogan]

It's a good point. It's really create that contains the specialization for distro.

And distro run is just a wrapper that invokes the distro create instead of the normal one.

Seems the important consideration is whether start needs to do anything special. Ideally everything unique to distro can be housed in container creation, and in a specialized runtime helper (container-runtime-linux-distro) if necessary.

[afbjorklund]

Every distro launched would have a default user created that is named after your macOS username.

A simple approach to get started with the idea, would be to prompt the user (after starting the container):

https://learn.microsoft.com/en-us/windows/wsl/setup/environment#set-up-your-linux-username-and-password

But a typical container would have a custom image for the user, with the account and the home mountpoint dir:

RUN groupadd -g $gid $group && useradd -u $uid -g $gid -d $home -s $shell -m -N $user \
&& rm -f /etc/.pwd.lock
VOLUME $home
WORKDIR $home
USER $user
CMD ["$shell"]

To not have to do it later at runtime, similar to how packages are installed in the image and not in the container?

Lima uses cloud-init and cloud-config.yaml for the regular cloud images, but it is overkill for container images...