polymorphic OCI OS images with Kairos

[j-landru]

abcd : Alpine Based Cos Derivative

Keywords : Kairos derivative; polymorphic OS images; immutability; instal-lless OS image; execution context agnostic; "rootless" Containerized OS (sandboxing).

I share here some personal notes about Kairos derivatives (sorry maybe a bit long), hoping it can be useful for others or maybe receive some feedbacks or guidelines from the Kairos community.

_

Disclaimers : This is a work in progress, not yet fully completed, some issues still need to be investigated (see at end of page...).
_

polymorphic OCI OS image with Kairos

Context

Orphan of K3OS as a base system for CIRRUS (Cloud Iac Reservoir de Resources pour Sessions de tp) a pedagogical lab cluster. I successively turned to C3OS, then elemental-toolkit and finally naturally Kairos to build base system ISO images for lab environments. For the fourth version of VIMINAL (VIrtual Model for Ip Network Architecture Lab), initially used as autonomous installless all in one LiveDVD network lab plateform, now essentially played as live ISO on HVM Hypervisor, I plan to switch from Gentoo catalyst live iso generator to the lighter Kairos/Aurora iso builder. I'll switch from Gentoo to Alpine distribution, which was the one used on K3OS, for its neat lightness and as a more than 20 years Gentoo user I'm more familar with its openrc init process. Prefix name of these images is ABCD which means Alpine Based Cos Derivative (Cos stands for "Containerized Opernating System"). But all that's describe in this post can, with no doubt, be translated to more common systemd like Linux distributions as Kairos is "Meta-Distribution, distro agnostic".

Essential properties for that goal :

• immutability : induced on initial liveDVD context that's a key property of Kairos environnment for ephemeral renewable lab with almost no surprises at lab session startup, as well for node startup for k3s CIRRUS cluster. Persistency, not yet managed on VIMINAL, will be based on COS_PESISTENCE auxilliary block strorage capability of Kairos.
• installless system : although A/B automatic install mode of Kairos is the right way to manage immutable always up-to-date base system. I'd like to stay in installless mode.
  • For VIMINAL, lab sessions are ephemeral, on common computer of lab rooms so :
    • no config changes on host ("You are requested to return the machine in the state in which you found it when booting")
    • and usually no admin rights on the common computer lab hosts.
  • For CIRRUS cluster, on core control nodes, I'd like to dedicate all disks for the distributed ROOK/Ceph storage backend, so system drive will be on auxilliary USB key drive not on disk and config or system persistence managed on COS_PERSITENT labeled auxilliary usb drive. Ephemeral compute nodes, will be on diskless PXE or even better IKU boot.
    _

Disclaimers : abcd4desktop derivatives for VIMINAL are full desktop systems (SDDM/KDE/Plasma). I'm aware that such systems are not Kairos system target. But Polymorphic OCI image property described on this post could also be useful as well for the infrastructure system OS images.
_

Polymorphic OS image running on all major common execution contexts

In that context of networked lab execution, the pure performance of lab system nodes is less important than the flexibility of execution environments that typically run nested. The nodes on which the students carry out manipulations are generally virtualized in the form of an appliance (all in one VM), as the first virtualization level, and the entire laboratory platform (VIMINAL, but also the dominant GNS3 system image in the context of teaching network infrastructure) is itself executed in virtualized mode (second virtualization level aka nested). At MOOCs (Massive Open On line Courses) scale, that laboratory platform must be able to be run for the entire subscriber population. It needs to be distributed as a simple VM image with the minimum of runtime dependencies, to be run on common HVM hypervisors such Virtualbox (available on the 3 dominant systems, i.e. GNU/Linux, Windows and IOS), VMWare or Hyper-V, as well as libvirt/KVM (but for the latter only for Linux subscribers).

As OSaC (Operating System as Code) environment, basically based on Dockerfile format, Kairos use standard OCI system image as its immutable rootfs packaging for baremetal or HVM-VM install. Several years ago, VIMINAL network lab model environment, as its third version, switched from nested HVM (libvirt/KVM) execution context to an LXD (lightvisor) execution context. So idea arose, why not try running Kairos system image on the common LXD container runtime as well as dominant Docker or Podman. In that way the same single base system image could be run on, not all but, almost all major common execution contexts : baremetal (liveUSB ISO), HVM hypervisor (KVM/libivrt, VirtualBox, VMWare, Hyper-V, XVisor,... in liveISO mode) and container runtime (LXD, Docker, Podman, ...) paving the way for VDI (Virtual Desktop Infrastrucure) scaling up to today's dominant cloud envrionment i.e. K8S.
_

Disclaimers : I know that containerized os images break the primitive dogma enacted by Docker ("one process per container you'll execute"), that's why Canonical designed LXD as "lightvisor" to containerize full systems. But on k8s clusters LXD runtime is not common.
_

inheritance and abcd lineage

OCI container images stack several storage layers at build time, one layer per step of the dockerfile. Multiple stacking levels can be aggregated when importing a previously built image. So one system image can share some common genes of ancestor system. The system image can thus be managed by inheritance, as in the OOP programming language. The polymorphic image adds an upper layer to a common functional image, whose elements are adapted to the target execution context. The aim is to ensure that, whatever the target execution context, the same functionalities, user interface and user experience are preserved.

So, for a basic ABCD image, these directives for the upper layer are set inside dedicated dockerfiles :

• abcdic : abcd Inside Container to build container runtime conformant system image ;

• abcdik : abcd with Initrd and Kernel : add initrd and kernel elements which will be used by Aurora to build iso full system image bootable on bare metal or on full virtalization hypervisors (HVM).

• abcdib : (just a crazy idea not yet implemented) abcd Inside Browser : Another lab platform called L.I.B.E.R.A.L. (Linux Inside Browser to Experiment Remotely Activities and Labs) is an in browser remote lab use case based on Fabrice Bellard JSLinux javascript CPU emulator. LIBERAL was conceived at IMT Nord Europe during that unusual COVID19 pandemic period where containment forced to play some labs remotely at student's home. Based on a javascript hardware emulator like jor1k or JSLinux, this kind of platform aimed at playing basic Linux system lab directly inside the browser with no system dependencies nor hypervisor install. On such Javascript emulator performance of the virtualized system is modest, but sufficient for shell cli manipulations. JSLinux emulates X86 or RISC-V architecture, so abcdib dockerfile would be the recipe for building LIBERAL lab images, but need to investigate Kairos on RISC-V, which would undoubtedly be another exciting story...

abcd family tree

                                                    +------+
                                                    | abcd | _kairos Alpine base_ 
                                                    +------+
                                                       |
                    +------------------------------------------------------------------------+
                    |                                                                        |
            +----------------+                                                      +----------------+
            | abcd4cirrus-os |  _kairos Alpine_                                     |  abcd4desktop  |   _Alpine desktop_
            +----------------+ _k3s base system_                                    +----------------+   _sddm/kde/plasma_
                    |                                                                        |
           +-----------------+                     +---------------+------------------+--------------------------------------+
           |                 |                     |               |                  |                                      |
 +----------------+ +----------------+     +--------------+ +--------------+   +------------+                       +------------------+
 |abcdic4cirrus-os| |abcdik4cirrus-os|     |abcdic4desktop| |abcdik4desktop|   |abcd4viminal| _VIMINAL base_        |abcd4objectif-ipv6| _objectif IPv6 MOOC_
 +----------------+ +----------------+     +--------------+ +--------------+   +------------+                       +------------------+ _GNS3 lab platform_
         !                  !                     !                !                 |                                       |
    (container)           (iso)              (container)         (iso)      +-----------------+                  +-----------------------+
                                                                            |                 |                  |                       |
                                                                     +--------------+ +--------------+ +--------------------+ +--------------------+   
                                                                     |abcdic4viminal| |abcdik4viminal| |abcdic4objectif-ipv6| |abcdik4objectif-ipv6|
                                                                     +--------------+ +--------------+ +--------------------+ +--------------------+
                                                                            !                 !                  !                       !
                                                                       (container)          (iso)           (container)                (iso)

- abcd : ancestor of the line (kairos Alpine reference dockerfile);
- abcd4cirrus-os : k3s system for CIRRUS cluster nodes;
- abcd4desktop : Alpine kde/plasma desktop base image;
- abcd4viminal : base image for VIMINAL 3.x lab collection (katara network labs);
- abcd4objectifipv6 : Objectif IPv6 MOOC labs (GNS3 IPv6 network models).

Nota : On abcd4desktop line subtree, graphical display is managed by classical X11 X server on physical or paravirtualized (such spice/qxl) graphic card drivers for abcdik descendants and xpra/Xdummy for abcdic descendants. On that latter, remote desktop is displayed through xpra HTML5 web rest interface with dynamic resolution auto-adjustment to browser window size. See illustration below and SDDM/KDE/PLASMA remote desktop with xpra for config details.

About rootless container

In this context of running academic labs, without privileged rights on the lab workstation (sandbox), it is necessary to run the containers in unprivileged mode (and more generally, it is a good practice to run with less privileges). So the environment settings to run the container abcd in rootless mode is the good way, (Another argument was to get around a window manager startup issue inside container see : "unbearable delay at sddm login/logout starting kde Plasma desktop when running inside container" for more details).

TODO - What else next ?!

• cgroupv2 rootless pod to run virtual desktop on k3s ;
• native Alpine kernel and initrd in replacement of kairos-alpine-ubuntu to build iso - pure, Alpine flavor, which is now independent from systemd, stayed tuned ;
• cloud-init in the container runtime context to align container startup seamlessly with the equivalent baremetal/VM dynamic configuration ?
• for small image booting live iso to ram, but UKI will be even better !! Introducing Kairos 2.0: long live UKI! useful for cirrus diskless compute node with WOL wake-up ?
• help or guidelines from Kairos community for COS_PERSISTENT on f2fs block device (see COS_PERSISTENT on a F2FS partition howto ? #1470 ;
• abcd4desktop new descendant : aawwoo Alternative Appliance When Windows is Out of Order. A SDDM/KDE/Plasma desktop disguised windows11 to help in case of emergency such disaster recovery when under crypto locking attack. Start crypto locked host in live mode without its damaged disk to furnish partial "corprorate" workload capabilities (web browsing - web mail - Libreoffice - ...) or VDI appliance started on a k8s rescue cluster ?!
• Virtual Desktop Infrastructure : on demand remote desktop as a service (ODRDaaS) at scale on k8s cloud could be considered on several ways :
  • on k8s cluster run abcdic4desktop descendants on demand as one pod per user (personnal persistant data mounted at startup or login under /home eventually through keyed sshfs ?)
  • on k8s cluster run abcdik4desktop descendants on demand as one kubevirt pod per user (personnal persistant data mounted at startup or login under /home eventually keyed sshf ?) that latter heavier than former but more confined
• investigate SSLH to study ingress flow multiplexing on single secure 443 port, (goal : end2end encrypting flows, reducing filetring rules, single cert key per pod, ???) in replacement of actual ssh tunnel multiplexing ??
• ...

Illustration

Does it work ? 1st try

On a Gentoo host, 4 simultaneous abcd4desktop instances on 4 execution contexts : from left to right and top down
abcdik4desktop liveiso HVM (kvm/libvirt spice/qxl paravirtualized graphic) ; abcdic4desktop rootless docker container (html5 remote display on firefox) ; abcdic4desktop rootless podman container (html5 remote display on firefox) ; abcdic4desktop lxd rootless container (html5 remote display on firefox).

automatic resolution adjustment to browser tab size on KDE/Plasma destop

More details, howtos, dockerfiles, container runtime configs ...

abcd is published on an alternate associative sovereign gitlab portal, available at https://framagit.org/j-landru/abcd >>>

[mauromorales]

Thanks for sharing @j-landru this is super cool! I wonder if we could link it from https://github.com/kairos-io/community

[j-landru]

This is a free open-source systems integration/configuration work, so of course, if you think it's worthwhile, you can reference it by pointing to it.

[mauromorales]

perfect thanks, I added a PR kairos-io/community#2

[mauromorales]

Also, getting a preview at one of our meetings would be super nice

[j-landru]

I'm in an academic period and quite busy teaching in the next few weeks, but interested in chatting or doing a demo if you'd like.

[mauromorales]

@j-landru I can understand and there's no rush. We have weekly meetings on Mondays 17:30 CE(S)T, feel free to add yourself to the agenda whenever it works for you. If the time/day does not work in general, then we can find a different time.

[Itxaka]

2 things I get from this.

• This is awesome, thanks for sharing.
• If I need some naming done I will hit you up, because the names are pretty cool :D

[j-landru]

Ahah, I don't know much about marketing and I'm not a communications pro either, but I've always thought that choosing an neat name for a project could help it get noticed. But I have to admit that sometimes it takes a while to find the right acronym...