[Storage] talos support and dependencies on host OS

[HanssenKai]

Hitting a roadblock trying to implement nutanx csi on a talos cluster, where the nutanix csi node is unable to run either mkfs.ext4 or mkfs.xfs as they are not provided by talos. Would it be possible to bundle some of these tools in the container image?

sh-4.4# ls -l  /usr/sbin | grep chroot
lrwxrwxrwx 1 root root      23 Nov  4 14:53 free -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 lsscsi -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 mkfs.ext3 -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 mkfs.ext4 -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 mount -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 mount.nfs -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 multipath -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 multipathd -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 pgrep -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 resize2fs -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 umount -> /chroot-host-wrapper.sh
lrwxrwxrwx 1 root root      23 Nov  4 14:53 xfs_growfs -> /chroot-host-wrapper.sh

[HanssenKai]

Hey @tuxtof, we talked briefly in kubecon amsterdam about this issue, but i went with mayastor to quickly get something working. Got some time to look at it again and finally got it working with talos.

To give some more context around talos, here is a issue to get longhorn working on talos and documentation. The problem is due to talos having very few binaries and is immutable without package manager or shell capabilities. While it does make for a very smooth kubernetes experience, in particular CSI implementations tend to encounter issues because they make assumptions about host node capabilities.

Hope this can be a starting point to get the general idea if you ever want to support talos in the future, or if other lost souls ever find themselves on the same path

The rough implentation to get it to work was;

• Create a sidecar for nutanix-csi-node with missing utilities
• Setup talos with iscsi extension
• Create a wrapper script to replace:
  /chroot-host-wrapper.sh in nutanix-csi-node to run most commands through the utilities sidecar utiliteis container with nsenter, remembering to redirect paths like /dev and /var to the host node /dev and /var e.g mkfs.xfs /host/dev/disk/by-id...
• replace /usr/sbin/[mount/umount/xfs_growfs] with wrapper script to run the commands on kubelet with nsenter. Otherwise pods won't have the volume mounted correctly.
• replace /usr/sbin/[iscsiadm/iscsi] with wrapper script to run through the iscsi extension with nsenter

Utilities sidecar:

FROM alpine

# Update apk and install necessary packages
RUN apk update
RUN apk add --no-cache \
    procps \
    xfsprogs \
    xfsprogs-extra \
    util-linux \
    nfs-utils \
    multipath-tools \
    procps \
    && rm -rf /var/cache/apk/*

# Use host as workdir
WORKDIR /host

# Set a default command to run an infinite sleep loop
CMD while true; do \
        echo $(pgrep iscsid) > /etc/shared-data/iscsid; \
        echo $(pgrep kubelet) > /etc/shared-data/kubelet; \
        echo $$ > /etc/shared-data/utils; \
        sleep 30; \
    done;

wrapper scripts configmap:

data:
  iscsiadm: |
    #!/bin/sh
    iscsid_pid=$(cat /etc/shared-data/iscsid)

    nsenter --mount="/proc/${iscsid_pid}/ns/mnt" --net="/proc/${iscsid_pid}/ns/net" -- /usr/local/sbin/iscsiadm "$@"
  iscsi: |
    #!/bin/sh
    iscsid_pid=$(cat /etc/shared-data/iscsid)

    nsenter --mount="/proc/${iscsid_pid}/ns/mnt" --net="/proc/${iscsid_pid}/ns/net" -- /usr/local/sbin/iscsi "$@"

  mount: |
    #!/usr/bin/env bash
    ME=`basename "$0"`
    kubelet_pid=$(cat /etc/shared-data/kubelet)
    nsenter --target=${kubelet_pid} --mount -- "${ME}" "${@:1}"

  utils: |
    #!/usr/bin/env bash
    ME=`basename "$0"`
    utils_pid=$(cat /etc/shared-data/utils)
    #nsenter --mount="/proc/${utils_pid}/ns/mnt" --net="/proc/${utils_pid}/ns/net" -- "${ME}" "${@:1}"
    # Transform all occurrences of '/dev' in arguments to '/host/dev'
    # This processes all arguments, replacing each instance as needed
    args=("$@")
    for i in "${!args[@]}"; do
      args[$i]=$(echo "${args[$i]}" | sed 's|/dev|/host/dev|g')
      args[$i]=$(echo "${args[$i]}" | sed 's|/var|/host/var|g')
    done

    # Use nsenter to run the command in the target namespaces with the transformed arguments
    nsenter --target=${utils_pid} --mount -- "${ME}" "${args[@]}"