Add OCI image support

[stgraber]

While we have no plans to turn Incus into a full Docker or Kubernetes competitor, we do see quite a few cases where folks either need to run Docker alongside Incus for just a couple of small services or are running Docker inside of an Incus container to achieve the same.

This is particularly common with things like IoT services which come in extremely trivial application containers as their primary distribution mechanism.

To better handle this, Incus should get the ability to create containers from OCI image registries.

The general expectation then would be to do something like:

incus remote add docker https://registry-1.docker.io/v2 --protocol=oci --public
incus launch docker:hello-world --ephemeral --console

This would be roughly equivalent to the traditional docker run hello-world.

To handle this, at minimum we'll need:

• Minimal OCI client in our client package
• Have the OCI client handle registry authentication and retrieval/combination of OCI layers
• Add support for using the oci protocol to retrieve container images on the server side
• Add some parsing of the metadata to create the container's initial configuration:
  • Environment (converted to environment.XYZ)
  • Cmd, WorkingDir, Entrypoint, StopSignals and User (internally handled)
  • ExposedPorts (converted to proxy devices)

We'll also likely want to add an extra config key to allow setting a restart policy.

For system containers, that's not really needed as init usually doesn't die and if it does, you usually want to know about it and not blindly restart it, but for application containers, it's a bit of a different story and much more common to need a restart policy of some kind.

It would also be good to have some kind of extra key in the API to tell us whether a container is a system container or an application container. That would then let us show something like:

stgraber@dakara:~$ incus list
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
|    NAME      |  STATE  |          IPV4           |                    IPV6                     |      TYPE       | SNAPSHOTS |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| centos3      | STOPPED |                         |                                             | CONTAINER       | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| centos4      | STOPPED |                         |                                             | CONTAINER       | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| fga          | STOPPED |                         |                                             | VIRTUAL-MACHINE | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| hydroqc2mqtt | RUNNING | 10.10.10.6 (eth0)       | fd42:1234:1234:fd42:0:242:a0a:a06 (eth0)    | CONTAINER (APP) | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| incus-ui     | RUNNING | 172.17.250.243 (eth0)   | 2602:fc62:c:250:216:3eff:fec9:ae37 (eth0)   | CONTAINER       | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| kernel-test  | RUNNING | 172.17.250.162 (enp5s0) | fd42:252a:1e48:9675::1 (incusbr0)           | VIRTUAL-MACHINE | 0         |
|              |         | 10.222.239.1 (incusbr0) | 2602:fc62:c:250:216:3eff:fe75:7941 (enp5s0) |                 |           |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| keybase      | STOPPED |                         |                                             | CONTAINER       | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| mkosi        | RUNNING |                         |                                             | VIRTUAL-MACHINE | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| rtl433mqtt   | RUNNING | 10.10.10.4 (eth0)       | fd42:1234:1234:fd42:0:242:a0a:a04 (eth0)    | CONTAINER (APP) | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| rtl-usb      | RUNNING | 10.10.10.3 (eth0)       | fd42:1234:1234:fd42:0:242:a0a:a03 (eth0)    | CONTAINER (APP) | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| speedtest    | RUNNING | 172.17.250.143 (eth0)   | 2602:fc62:c:250:216:3eff:feb9:39d4 (eth0)   | CONTAINER       | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| sysinfo      | RUNNING | 172.17.250.148 (eth0)   | 2602:fc62:c:250:216:3eff:fe1b:2f8d (eth0)   | CONTAINER       | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| v2           | RUNNING |                         | 2602:fc62:c:250:216:3eff:fe62:169b (eth0)   | VIRTUAL-MACHINE | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| win11        | STOPPED |                         |                                             | VIRTUAL-MACHINE | 1         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| zigbee2mqtt  | RUNNING | 10.10.10.5 (eth0)       | fd42:1234:1234:fd42:0:242:a0a:a05 (eth0)    | CONTAINER (APP) | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+
| zwave-js     | RUNNING | 10.10.10.2 (eth0)       | fd42:1234:1234:fd42:0:242:a0a:a02 (eth0)    | CONTAINER (APP) | 0         |
+--------------+---------+-------------------------+---------------------------------------------+-----------------+-----------+

[stgraber]

@cyphar @hallyn @tych0 so I'm guessing the way to go for the OCI image retrieval and conversion into a format we can handle (just merging all layers together, we'll do something more optimized later) would be to use umoci's Go packages?

[tych0]

umoci doesn't support retrieval, for that I used containers/image, which is a redhat library that backs skopeo. It is a bit annoying because it brings in a bunch of shared object dependencies, most of which you can disabel.

For extraction and manipulation once you have them downloaded, though, umoci is the way to go.

[tych0]

I think it is also worth exporting the exit code of these app containers somehow, as it's a thing people care about. I was intending to do that, but have gotten distracted by other things :)

[stgraber]

Ah yeah indeed, I remember us talking about the exit code and adding some liblxc logic to expose that to the stop/post-stop hooks.

[stgraber]

stgraber@dakara:~$ incus launch docker:hello-world demo-oci -c raw.lxc=lxc.init.cmd=/hello
Launching demo-oci
stgraber@dakara:~$ incus console --show-log demo-oci

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

[stgraber]

So I've basically got support for OCI images throughout the codebase and rely on skopeo and umoci to access image servers and convert things into image files we can consume.

stgraber@dakara:~$ incus image info 266b1
Fingerprint: 266b191e926f65542fa8daaec01a192c4d292bff79426f47300a046e1bc576fd
Size: 0.00MiB
Architecture: x86_64
Type: container
Public: no
Timestamps:
    Created: 2023/05/02 12:49 EDT
    Uploaded: 2024/06/05 01:14 EDT
    Expires: never
    Last used: 2024/06/05 01:20 EDT
Properties:
Aliases:
Cached: yes
Auto update: enabled
Source:
    Server: https://docker.io
    Protocol: oci
    Alias: hello-world
Profiles:
    - default

This is all working fine and will even get automatic background updates on images and everything, same as what we have with simplestreams.

It's missing a lot of optimizations though, the current process downloads a ton of stuff to disk just to repack them and send them back over the network. Ideally we should be able to do all of that as data is streaming through.

But this basically unblocks getting the rest of the work done, which is having the LXC driver detect application containers and automatically run them with all the right settings.

[stgraber]

A few more things I've noticed that we'll need to sort out:

• Network configuration. Containers don't do that, so we need our init shim to handle it.
• --console on incus launch or incus start doesn't work on very short lived containers like hello-world as the container is stopped by the time we try to connect to it. To fix this we should both have the CLI handle the stopped case by pulling the log instead AND have the init shim wait a couple of seconds before executing the entry point, giving us time to connect.
• We need to do smarter parsing/escaping of lxc.init_cmd to handle commands with arguments containing spaces
• Definitely need to have GetImageFile return at least some kind of progress information given how slow the process can be

[stgraber]

stgraber@dakara:~$ incus launch docker:hello-world --ephemeral --console
Launching the instance
Instance name is: primary-fawn

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

[awalvie]

Woah! That was fast!

[stgraber]

The devil's always in the details, so while we have something fun and flashy right now, it's likely going to take a bit longer to sort out all those small issues...

[parallax-home]

I have questions. :)

How would docker-compose files be handled? Or would the user define the container launch parameters via command line? Will this handle .env files, otherwise potentially passwords are going to be sloshing about in clear text?

As a general comment, the ability to launch containers/VM from a config file held in a git would be great. Then a hook to trigger a redeployment of the container after a change to the config would be chef's kiss

incus launch docker:hello-world --ephemeral --console --configfile git:hello-world --poll 5m

[stgraber]

How would docker-compose files be handled? Or would the user define the container launch parameters via command line? Will this handle .env files, otherwise potentially passwords are going to be sloshing about in clear text?

I don't expect we will be handling docker-compose files, at least not initially.
Our focus is really on running OCI images natively and not on trying to achieve parity with Docker itself or its tools ecosystem.

For the .env, we support feeding YAML directly to incus create or incus launch so that's likely how you'd avoid passing your environment variables through the globally visible command line.

Our ability to do incus launch docker:hello-world --ephemeral --console < config.yaml is also how someone would likely implement your second comment. Having us directly understand how to interact and monitor git feels out of scope, but the fact that we support reading a full YAML config from stdin already makes scripting something like that pretty easy.

[3XX0]

Nice! @stgraber you've probably looked into it already, but we hacked a LXC OCI template back in the days: https://github.com/lxc/lxc/blob/main/templates/lxc-oci.in
Not sure if some of the logic could be reused here.

[stgraber]

Yep, that's what I used to figure out most of the logic and tools needed.

Incus doesn't use LXC templates and works very differently in the way it handles images and creates the storage part of the container so we can't straight up use the script but it made for a good starting point and a good way to make sure I didn't miss anything.

[3XX0]

Ok yeah makes sense, just checking.

Depending on your kernel requirements you could also avoid umoci altogether with unpriv overlay (i.e. parallel aufs2ovlfs + tar / mksquash the mount), this should provide significant speedup.

The DHCPv4 hook was kind of a pain, but I guess for this one you can bundle a small Go dhcpclient.

[stgraber]

Yeah, I'm currently working on the network configuration side of things and already have a minimal DHCP client written in Go, just sorting out the hooks to have it called and be able to safely configure the network in the container.

[stgraber]

I published the PR for OCI support if anyone wants to follow along :)

Since last time, I've basically added:

• Basic env variable support (needs to be moved to image unpack)
• Usage of liblxc's execute rather than start method (needed a go-lxc fix)
• Proper handling of entry point containing spaces or being relative
• Fixed the console handling
• DHCP client (IPv4 now working)

So the main bits to do at this point are:

• Make it possible to distinguish app vs system containers in the API and CLI
• Cleanup DHCP client to avoid failures on less common network setups
• Move environment handling to image unpack (set environment config keys)
• Add port handling during image unpack (create proxy devices)
• Add config keys for auto-restart of the instance
• Improve performance of image conversion and add progress reporting

[jamescvbn]

This is what I need. Thank you!