Provisioning a group of non-Kubernetes hosts using ClusterAPI and Virtual Kubelets

[mgazz]

Context:

Administrators of on-premise infrastructure often need to provision bare-metal hosts that are not part of a Kubernetes
cluster with custom data, such as SSH keys, packages, and configuration files.

Goal: Provision a set of Metal3 BareMetalHost with associated userData.

Challenge: Currently, manipulating BareMetalHosts (BMH) allows for provisioning and deprovisioning of individual hosts,
but there is no support for large-scale orchestration. At the same time, ClusterAPI (CAPI) and the Metal3 ClusterAPI provider (CAPM3) enable the provisioning of multiple hosts but such capabilities are dedicated to the management of Kubernetes nodes.

A previous proposal tackled this problem by introducing a new operator but this requires extensive development and the replication of host provisioning functionalities already implemented by CAPI/CAPM3.

In this post we tackle two main questions:

• Can we use the Metal3 ClusterAPI provider to manage a group of BareMetalHosts?
• If not, how can we overcome the identified challenges and adopt CAPI/CAPM3 for this use-case?

In the rest of this post I report our findings trying to tackle the two questions above. I would like to open a discussion on this approach and know if you any feedback.

TLDR:

We found that the CAPI Machine and MachineDeployment objects, while suitable in theory,
as part of their design they expect a Kubernetes Node in the target cluster for each Machine associated with a MachineDeployment.
Given that no Kubernetes Node object is created when provisioning non-Kubernetes hosts, the CAPI MachineDeployment remains stuck in the ScalingUp phase.

We found a solution to the challenge posed by the absence of a Kubernetes Node object: utilizing Virtual Kubelets.
By employing Virtual Kubelets, we can register hosts as Kubernetes Nodes without enabling Pod execution.
This allows each CAPI Machine to become Ready and the MachineDeployment to progress from the ScalingUp phase to the Running phase.

To streamline this process, a CAPI Bootstrap Provider for Virtual Kubelets automates the configuration
of virtual-kubelets, allowing users to provision a group of BareMetalHosts with ClusterAPI objects.

This approach allows us to reuse existing CAPI/CAPM3 resources and we avoid implementing an additional operator to manage a group of non-Kubernetes hosts. Also virtual kubelets could be used by administrators to implement custom management operations on the provisioned hosts. I have left additional considerations at the end of the post.

Following, an in-depht account of our investigation and how we ended up with the idea of a CAPI Bootstrap Provider for Virtual Kubelets.

We have 3 sections:

• Environment Setup: where we create a target cluster with CAPI/CAPM3 objects.
• Investigating the use of ClusterAPIs for non-Kubernetes nodes: where we investigate the use of CAPI objects to reach our goal and we identify the challenges to tackle.
• Enabling Cluster-APIs management for non-Kubernetes hosts using Virtual Kubelets: where we describe how we tackle the challenges identified in the previous section with a new ClusterAPI Bootstrap Provider using Virtual Kubelets.

Environment setup

As a first step, we use the metal3-dev-env project provided by Metal3 to
setup our testing environment.

For testing, we consider two additional virtual machines, supplementing the one dedicated to the Kubernetes control plane and workers nodes.
These extra machines will be used to test the management of non-Kubernetes hosts.

Following the variables defined in the config_<USER>.sh:

export NUM_NODES=7
export CONTROL_PLANE_MACHINE_COUNT=3
export WORKER_MACHINE_COUNT=2

We follow the metal3-dev-env documentation and we deploy the target cluster.

./tests/scripts/provision/cluster.sh

The objects created are reported here. (expand for the yaml definition)

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: test1
namespace: metal3
spec:
clusterNetwork:
  pods:
    cidrBlocks:
    - 192.168.0.0/18
  services:
    cidrBlocks:
    - 10.96.0.0/12
controlPlaneRef:
  apiVersion: controlplane.cluster.x-k8s.io/v1beta1
  kind: KubeadmControlPlane
  name: test1
infrastructureRef:
  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: Metal3Cluster
  name: test1
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3Cluster
metadata:
name: test1
namespace: metal3
spec:
controlPlaneEndpoint:
  host: 192.168.111.249
  port: 6443
noCloudProvider: true
---
apiVersion: ipam.metal3.io/v1alpha1
kind: IPPool
metadata:
name: provisioning-pool
namespace: metal3
spec:
clusterName: test1
namePrefix: test1-prov
pools:
- end: 172.22.0.200
  start: 172.22.0.100
prefix: 24
---
apiVersion: ipam.metal3.io/v1alpha1
kind: IPPool
metadata:
name: externalv4-pool
namespace: metal3
spec:
clusterName: test1
gateway: 192.168.111.1
namePrefix: test1-bmv4
pools:
- end: 192.168.111.200
  start: 192.168.111.100
prefix: 24

Following a visual representation of the objects created so far.

The CAPI Cluster refers a KubeadmControlPlane object (not defined yet), and a provisioning infrastructure for the cluster (Metal3Cluster).

Note: To simplify our explanation for the rest of the post we do not consider resources like IPPools, Metal3DataTemplate )

As a next step we install the Kubernetes control plane.

./tests/scripts/provision/controlplane.sh

The objects created are reported here. (expand for the yaml definition)

Note: We omit the ssh key and use the tag <KEY> instead

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: test1
  namespace: metal3
spec:
  kubeadmConfigSpec:
    clusterConfiguration: {}
    files:
    - content: |
        #!/bin/bash
        set -e
        url="$1"
        dst="$2"
        filename="$(basename $url)"
        tmpfile="/tmp/$filename"
        curl -sSL -w "%{http_code}" "$url" | sed "s:/usr/bin:/usr/local/bin:g" > /tmp/"$filename"
        http_status=$(cat "$tmpfile" | tail -n 1)
        if [ "$http_status" != "200" ]; then
          echo "Error: unable to retrieve $filename file";
          exit 1;
        else
          cat "$tmpfile"| sed '$d' > "$dst";
        fi
      owner: root:root
      path: /usr/local/bin/retrieve.configuration.files.sh
      permissions: "0755"
    - content: |
        ! Configuration File for keepalived

        script k8s_api_check {
            script "curl -sk https://127.0.0.1:6443/healthz"
            interval 5
            timeout 5
            rise 3
            fall 3
        }

        vrrp_instance VI_1 {
            state MASTER
            interface eth1
            virtual_router_id 1
            priority 101
            advert_int 1
            virtual_ipaddress {
                192.168.111.249
            }
            track_script {
                k8s_api_check
            }
        }
      path: /etc/keepalived/keepalived.conf
    - content: |
        [connection]
        id=eth0
        type=ethernet
        interface-name=eth0
        master=ironicendpoint
        slave-type=bridge
      owner: root:root
      path: /etc/NetworkManager/system-connections/eth0.nmconnection
      permissions: "0600"
    - content: |
        [connection]
        id=ironicendpoint
        interface-name=ironicendpoint
        type=bridge
        autoconnect=yes
        autoconnect-priority=1
        [bridge]
        interface-name=ironicendpoint
        stp=false
        [ipv4]
        address1={{ ds.meta_data.provisioningIP }}/{{ ds.meta_data.provisioningCIDR }}
        method=manual
        [ipv6]
        addr-gen-mode=eui64
        method=ignore
      owner: root:root
      path: /etc/NetworkManager/system-connections/ironicendpoint.nmconnection
      permissions: "0600"
    - content: |
        [kubernetes]
        name=Kubernetes
        baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
        enabled=1
        gpgcheck=1
        repo_gpgcheck=0
        gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
      owner: root:root
      path: /etc/yum.repos.d/kubernetes.repo
      permissions: "0644"
    - content: |
        [registries.search]
        registries = ['docker.io']

        [registries.insecure]
        registries = ['192.168.111.1:5000']
      path: /etc/containers/registries.conf
    initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          cgroup-driver: systemd
          container-runtime-endpoint: unix:///var/run/crio/crio.sock
          feature-gates: AllAlpha=false
          node-labels: metal3.io/uuid={{ ds.meta_data.uuid }}
          provider-id: metal3://{{ ds.meta_data.providerid }}
          runtime-request-timeout: 5m
        name: '{{ ds.meta_data.name }}'
    joinConfiguration:
      controlPlane: {}
      nodeRegistration:
        kubeletExtraArgs:
          cgroup-driver: systemd
          container-runtime-endpoint: unix:///var/run/crio/crio.sock
          feature-gates: AllAlpha=false
          node-labels: metal3.io/uuid={{ ds.meta_data.uuid }}
          provider-id: metal3://{{ ds.meta_data.providerid }}
          runtime-request-timeout: 5m
        name: '{{ ds.meta_data.name }}'
    postKubeadmCommands:
    - mkdir -p /home/metal3/.kube
    - chown metal3:metal3 /home/metal3/.kube
    - cp /etc/kubernetes/admin.conf /home/metal3/.kube/config
    - chown metal3:metal3 /home/metal3/.kube/config
    preKubeadmCommands:
    - systemctl restart NetworkManager.service
    - nmcli connection load /etc/NetworkManager/system-connections/ironicendpoint.nmconnection
    - nmcli connection up ironicendpoint
    - nmcli connection load /etc/NetworkManager/system-connections/eth0.nmconnection
    - nmcli connection up eth0
    - rm /etc/cni/net.d/*
    - systemctl enable --now keepalived
    - sleep 30
    - systemctl enable --now crio
    - sleep 30
    - systemctl enable --now kubelet
    - sleep 120
    users:
    - name: metal3
      sshAuthorizedKeys:
      - ssh-rsa <KEY>
        user@adcpu28
      sudo: ALL=(ALL) NOPASSWD:ALL
  machineTemplate:
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: Metal3MachineTemplate
      name: test1-controlplane
    nodeDrainTimeout: 0s
  replicas: 3
  rolloutStrategy:
    rollingUpdate:
      maxSurge: 1
  version: v1.31.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
  name: test1-controlplane
  namespace: metal3
spec:
  template:
    spec:
      dataTemplate:
        name: test1-controlplane-template
      image:
        checksum: http://172.22.0.1/images/CENTOS_9_NODE_IMAGE_K8S_v1.31.0-raw.img.sha256sum
        checksumType: sha256
        format: raw
        url: http://172.22.0.1/images/CENTOS_9_NODE_IMAGE_K8S_v1.31.0-raw.img
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
  name: test1-controlplane-template
  namespace: metal3
spec:
  clusterName: test1
  metaData:
    ipAddressesFromIPPool:
    - key: provisioningIP
      name: provisioning-pool
    objectNames:
    - key: name
      object: machine
    - key: local-hostname
      object: machine
    - key: local_hostname
      object: machine
    prefixesFromIPPool:
    - key: provisioningCIDR
      name: provisioning-pool
  networkData:
    links:
      ethernets:
      - id: enp1s0
        macAddress:
          fromHostInterface: enp1s0
        type: phy
      - id: enp2s0
        macAddress:
          fromHostInterface: enp2s0
        type: phy
      - id: enp3s0
        macAddress:
          fromHostInterface: enp3s0
        type: phy
      - id: enp4s0
        macAddress:
          fromHostInterface: enp4s0
        type: phy
    networks:
      ipv4:
      - id: externalv4
        ipAddressFromIPPool: externalv4-pool
        link: enp2s0
        routes:
        - gateway:
            fromIPPool: externalv4-pool
          network: 0.0.0.0
          prefix: 0
    services:
      dns:
      - 8.8.8.8

Here a visual representation:

The KubeadmControlPlane:

• refers a Metal3MachineTemplate, which allows customisation of key host provisioning elements, such as the image, network data, and metadata.
• specifies the number of replicas, determining how many CAPI Machines will be created for the target cluster control plane.

Each CAPI Machine is associated with:

• A KubeadmConfig, which provides a bootstrap configuration.
• A Metal3Machine, containing deployment information for the corresponding BareMetalHost

The KubeadmConfig is generated by the Cluster API Bootstrap Provider Kubeadm (CABPK), which creates a K8s Secret containing a cloud-config or ignition file to customize BareMetalHosts during boot.
By design, each KubeadmConfig provide a reference to the secret in status.dataSecretName

For each Metal3Machine, CAPM3 selects a BareMetalHost for provisioning and sets the secret generated by the KubeadmConfig
as userData.

As a final step we deploy the workers for the target cluster:

./tests/scripts/provision/worker.sh

The workers are deployed via a MachineDeployment (expand for the yaml definition)

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: test1
    nodepool: nodepool-0
  name: test1
  namespace: metal3
spec:
  clusterName: test1
  replicas: 2
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: test1
      nodepool: nodepool-0
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: test1
        nodepool: nodepool-0
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: test1-workers
      clusterName: test1
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: Metal3MachineTemplate
        name: test1-workers
      nodeDrainTimeout: 0s
      version: v1.31.0
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
  name: test1-workers
  namespace: metal3
spec:
  template:
    spec:
      dataTemplate:
        name: test1-workers-template
      image:
        checksum: http://172.22.0.1/images/CENTOS_9_NODE_IMAGE_K8S_v1.31.0-raw.img.sha256sum
        checksumType: sha256
        format: raw
        url: http://172.22.0.1/images/CENTOS_9_NODE_IMAGE_K8S_v1.31.0-raw.img
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
  name: test1-workers-template
  namespace: metal3
spec:
  clusterName: test1
  metaData:
    ipAddressesFromIPPool:
    - key: provisioningIP
      name: provisioning-pool
    objectNames:
    - key: name
      object: machine
    - key: local-hostname
      object: machine
    - key: local_hostname
      object: machine
    prefixesFromIPPool:
    - key: provisioningCIDR
      name: provisioning-pool
  networkData:
    links:
      ethernets:
      - id: enp1s0
        macAddress:
          fromHostInterface: enp1s0
        type: phy
      - id: enp2s0
        macAddress:
          fromHostInterface: enp2s0
        type: phy
      - id: enp3s0
        macAddress:
          fromHostInterface: enp3s0
        type: phy
      - id: enp4s0
        macAddress:
          fromHostInterface: enp4s0
        type: phy
    networks:
      ipv4:
      - id: externalv4
        ipAddressFromIPPool: externalv4-pool
        link: enp2s0
        routes:
        - gateway:
            fromIPPool: externalv4-pool
          network: 0.0.0.0
          prefix: 0
    services:
      dns:
      - 8.8.8.8
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: test1-workers
  namespace: metal3
spec:
  template:
    spec:
      files:
      - content: |
          #!/bin/bash
          set -e
          url="$1"
          dst="$2"
          filename="$(basename $url)"
          tmpfile="/tmp/$filename"
          curl -sSL -w "%{http_code}" "$url" | sed "s:/usr/bin:/usr/local/bin:g" > /tmp/"$filename"
          http_status=$(cat "$tmpfile" | tail -n 1)
          if [ "$http_status" != "200" ]; then
            echo "Error: unable to retrieve $filename file";
            exit 1;
          else
            cat "$tmpfile"| sed '$d' > "$dst";
          fi
        owner: root:root
        path: /usr/local/bin/retrieve.configuration.files.sh
        permissions: "0755"
      - content: |
          [connection]
          id=eth0
          type=ethernet
          interface-name=eth0
          master=ironicendpoint
          slave-type=bridge
          autoconnect=yes
          autoconnect-priority=999
        owner: root:root
        path: /etc/NetworkManager/system-connections/eth0.nmconnection
        permissions: "0600"
      - content: |
          [connection]
          id=ironicendpoint
          type=bridge
          interface-name=ironicendpoint

          [bridge]
          stp=false

          [ipv4]
          address1={{ ds.meta_data.provisioningIP }}/{{ ds.meta_data.provisioningCIDR }}
          method=manual

          [ipv6]
          addr-gen-mode=eui64
          method=ignore
        owner: root:root
        path: /etc/NetworkManager/system-connections/ironicendpoint.nmconnection
        permissions: "0600"
      - content: |
          [kubernetes]
          name=Kubernetes
          baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
          enabled=1
          gpgcheck=1
          repo_gpgcheck=0
          gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
        owner: root:root
        path: /etc/yum.repos.d/kubernetes.repo
        permissions: "0644"
      - content: |
          [registries.search]
          registries = ['docker.io']

          [registries.insecure]
          registries = ['192.168.111.1:5000']
        owner: root:root
        path: /etc/containers/registries.conf
        permissions: "0644"
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            cgroup-driver: systemd
            container-runtime-endpoint: unix:///var/run/crio/crio.sock
            feature-gates: AllAlpha=false
            node-labels: metal3.io/uuid={{ ds.meta_data.uuid }}
            provider-id: metal3://{{ ds.meta_data.providerid }}
            runtime-request-timeout: 5m
          name: '{{ ds.meta_data.name }}'
      preKubeadmCommands:
      - rm /etc/cni/net.d/*
      - systemctl restart NetworkManager.service
      - nmcli connection load /etc/NetworkManager/system-connections/ironicendpoint.nmconnection
      - nmcli connection up ironicendpoint
      - nmcli connection load /etc/NetworkManager/system-connections/eth0.nmconnection
      - nmcli connection up eth0
      - systemctl enable --now crio
      - sleep 30
      - systemctl enable --now kubelet
      - sleep 120
      users:
      - name: metal3
        sshAuthorizedKeys:
        - ssh-rsa <KEY>
          user@adcpu28
        sudo: ALL=(ALL) NOPASSWD:ALL

Following, a simplified representation of the relationship between the various objects created:

A MachineDeployment oversees a group of CAPI Machines that are part of a specific CAPI cluster. To tailor the installation process, it leverages two templates:

• A Metal3MachineTemplate: which customises the image provisioned on the hosts.
• A KubeadmConfigTemplate: which configures the data installed on the hosts.

Notably, such approach is consistent with the method used by the KubeadmControlPlane to provision the target cluster control plane.

The Cluster API Bootstrap Provider Kubeadm (CABPK) generates a customized KubeadmConfig for each CAPI Machine, based on the provided KubeadmConfigTemplate. This process also creates a Secret that contains a cloud-config or ignition configuration.
The Secret is then assigned as userData to the corresponding BareMetalHost, enabling the host to be initialized with a tailored configuration.

The current state of the system is as follows (snapshot below):

• The MachineDeployment has successfully reached its desired state, with 2 out of 2 replicas Ready and the deployment in a Running phase.
• Each Machine is associated with corresponding Node in the target cluster and BareMetalHost, identified by column NODENAME and PROVIDERID respectively.
• Each BareMetalHost is in a Provisioned state, indicating successful deployment and linked to a Machine listed in the CONSUMER column

user@adcpu28:~/workspace/metal3-dev-env$ kubectl get clusters,kubeadmcontrolplanes,kubeadmconfigtemplate,kubeadmconfigs,metal3clusters,metal3machinetemplate,machinedeployments,machinesets,machines,metal3machines,bmh
NAME                             CLUSTERCLASS   PHASE         AGE     VERSION
cluster.cluster.x-k8s.io/test1                  Provisioned   3h58m

NAME                                                      CLUSTER   INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE     VERSION
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/test1   test1     true          true                   3          3       3         0             3h57m   v1.31.0

NAME                                                             AGE
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/test1-workers   173m

NAME                                                         CLUSTER   AGE
kubeadmconfig.bootstrap.cluster.x-k8s.io/test1-6nvzx-bss44   test1     173m
kubeadmconfig.bootstrap.cluster.x-k8s.io/test1-6nvzx-ns9fl   test1     173m
kubeadmconfig.bootstrap.cluster.x-k8s.io/test1-bhcf4         test1     3h57m
kubeadmconfig.bootstrap.cluster.x-k8s.io/test1-cw695         test1     3h46m
kubeadmconfig.bootstrap.cluster.x-k8s.io/test1-djvs9         test1     3h51m

NAME                                                  AGE     READY   ERROR   CLUSTER   ENDPOINT
metal3cluster.infrastructure.cluster.x-k8s.io/test1   3h58m   true            test1     {"host":"192.168.111.249","port":6443}

NAME                                                                       AGE
metal3machinetemplate.infrastructure.cluster.x-k8s.io/test1-controlplane   3h57m
metal3machinetemplate.infrastructure.cluster.x-k8s.io/test1-workers        173m

NAME                                       CLUSTER   REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE    VERSION
machinedeployment.cluster.x-k8s.io/test1   test1     2          2       2         0             Running   173m   v1.31.0

NAME                                      CLUSTER   REPLICAS   READY   AVAILABLE   AGE    VERSION
machineset.cluster.x-k8s.io/test1-6nvzx   test1     2          2       2           173m   v1.31.0

NAME                                         CLUSTER   NODENAME            PROVIDERID                                 PHASE     AGE     VERSION
machine.cluster.x-k8s.io/test1-6nvzx-bss44   test1     test1-6nvzx-bss44   metal3://metal3/node-0/test1-6nvzx-bss44   Running   173m    v1.31.0
machine.cluster.x-k8s.io/test1-6nvzx-ns9fl   test1     test1-6nvzx-ns9fl   metal3://metal3/node-2/test1-6nvzx-ns9fl   Running   173m    v1.31.0
machine.cluster.x-k8s.io/test1-bhcf4         test1     test1-bhcf4         metal3://metal3/node-3/test1-bhcf4         Running   3h57m   v1.31.0
machine.cluster.x-k8s.io/test1-cw695         test1     test1-cw695         metal3://metal3/node-5/test1-cw695         Running   3h46m   v1.31.0
machine.cluster.x-k8s.io/test1-djvs9         test1     test1-djvs9         metal3://metal3/node-4/test1-djvs9         Running   3h51m   v1.31.0

NAME                                                              AGE     PROVIDERID                                 READY   CLUSTER   PHASE
metal3machine.infrastructure.cluster.x-k8s.io/test1-6nvzx-bss44   173m    metal3://metal3/node-0/test1-6nvzx-bss44   true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-6nvzx-ns9fl   173m    metal3://metal3/node-2/test1-6nvzx-ns9fl   true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-bhcf4         3h57m   metal3://metal3/node-3/test1-bhcf4         true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-cw695         3h46m   metal3://metal3/node-5/test1-cw695         true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-djvs9         3h51m   metal3://metal3/node-4/test1-djvs9         true    test1

NAME                             STATE         CONSUMER            ONLINE   ERROR   AGE
baremetalhost.metal3.io/node-0   provisioned   test1-6nvzx-bss44   true             4h45m
baremetalhost.metal3.io/node-1   available                         true             4h45m
baremetalhost.metal3.io/node-2   provisioned   test1-6nvzx-ns9fl   true             4h45m
baremetalhost.metal3.io/node-3   provisioned   test1-bhcf4         true             4h45m
baremetalhost.metal3.io/node-4   provisioned   test1-djvs9         true             4h45m
baremetalhost.metal3.io/node-5   provisioned   test1-cw695         true             4h45m
baremetalhost.metal3.io/node-6   available                         true             4h45m

It's interesting to see that at this stage we have many secrets tagged with the name of the provisioned cluster:

• a secret for each machine containing the cloud-configs or ignition file used as userData while provisioning the BareMetalHosts.
• a secret <CLUSTER_NAME>-kubeconfig that can be used to access the cluster.
• etc...

user@adcpu28:~/workspace/metal3-dev-env$ kubectl get secrets

NAME                                  TYPE                                     DATA   AGE
secret/node-0-bmc-secret              Opaque                                   2      4h45m
secret/node-1-bmc-secret              Opaque                                   2      4h45m
secret/node-2-bmc-secret              Opaque                                   2      4h45m
secret/node-3-bmc-secret              Opaque                                   2      4h45m
secret/node-4-bmc-secret              Opaque                                   2      4h45m
secret/node-5-bmc-secret              Opaque                                   2      4h45m
secret/node-6-bmc-secret              Opaque                                   2      4h45m
secret/test1-6nvzx-bss44              cluster.x-k8s.io/secret                  2      173m
secret/test1-6nvzx-bss44-metadata     infrastructure.cluster.x-k8s.io/secret   1      173m
secret/test1-6nvzx-bss44-networdata   infrastructure.cluster.x-k8s.io/secret   1      173m
secret/test1-6nvzx-ns9fl              cluster.x-k8s.io/secret                  2      173m
secret/test1-6nvzx-ns9fl-metadata     infrastructure.cluster.x-k8s.io/secret   1      173m
secret/test1-6nvzx-ns9fl-networdata   infrastructure.cluster.x-k8s.io/secret   1      173m
secret/test1-bhcf4                    cluster.x-k8s.io/secret                  2      3h57m
secret/test1-bhcf4-metadata           infrastructure.cluster.x-k8s.io/secret   1      3h57m
secret/test1-bhcf4-networdata         infrastructure.cluster.x-k8s.io/secret   1      3h57m
secret/test1-ca                       cluster.x-k8s.io/secret                  2      3h57m
secret/test1-cw695                    cluster.x-k8s.io/secret                  2      3h46m
secret/test1-cw695-metadata           infrastructure.cluster.x-k8s.io/secret   1      3h46m
secret/test1-cw695-networdata         infrastructure.cluster.x-k8s.io/secret   1      3h46m
secret/test1-djvs9                    cluster.x-k8s.io/secret                  2      3h51m
secret/test1-djvs9-metadata           infrastructure.cluster.x-k8s.io/secret   1      3h51m
secret/test1-djvs9-networdata         infrastructure.cluster.x-k8s.io/secret   1      3h51m
secret/test1-etcd                     cluster.x-k8s.io/secret                  2      3h57m
secret/test1-kubeconfig               cluster.x-k8s.io/secret                  1      3h57m
secret/test1-proxy                    cluster.x-k8s.io/secret                  2      3h57m
secret/test1-sa                       cluster.x-k8s.io/secret

Examining the target cluster, we observe that it comprises 5 nodes (3 for the control plane and 3 worker nodes), all of which are in a Ready state.

user@adcpu28:~/workspace/metal3-dev-env$ kubectl get secret test1-kubeconfig -o json | jq -r .data.value | base64 -d > test_kubeconfig
user@adcpu28:~/workspace/metal3-dev-env$ kubectl get nodes --kubeconfig test_kubeconfig -o wide
NAME                STATUS   ROLES           AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE          KERNEL-VERSION          CONTAINER-RUNTIME
test1-6nvzx-bss44   Ready    <none>          147m    v1.31.0   192.168.111.104   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4
test1-6nvzx-ns9fl   Ready    <none>          148m    v1.31.0   192.168.111.103   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4
test1-bhcf4         Ready    control-plane   3h30m   v1.31.0   192.168.111.100   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4
test1-cw695         Ready    control-plane   3h19m   v1.31.0   192.168.111.102   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4
test1-djvs9         Ready    control-plane   3h24m   v1.31.0   192.168.111.101   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4

Investigating the use of ClusterAPIs for non-Kubernetes nodes

Based on the current state we can draw several conclusions:

• The MachineDeployment object shows promise for provisioning multiple hosts, especially when combined with a user defined Metal3MachineTemplate.
• However, a MachineDeployment cannot be provisioned independently; it requires a CAPI Cluster.
• The secret generated by CABPK, which contains a cloud-config, is a crucial component for customising host.

Our ultimate goal is to provision a set of hosts with an identical configurations. While the MachineDeployment is a promising resource, using KubeadmConfigTemplates would result in hosts joining as worker nodes and allowing pod execution.
To avoid this, we explored an alternative solution using a simple bootstrap provider, dubbed MockBootstrap, which enables users to reference a secret containing the cloud-config or ignition file used for the provisioning of BareMetalHosts.

Note: MockBootstrap was created solely for the purpose of exploring ClusterAPI mechanisms and will be replaced by a solution leveraging virtual-kubelets. This section aims to demonstrate the necessity of virtual-kubelets.

MockBootstrap implements the bootstrap behaviour described by CAPI and has a straightforward implementation.
A bootstrap provider must create config resources that report a Ready status when the resource is prepared for use and specify a userDataSecret that will be injected as userData to the BareMetalHost during provisioning. MockBootstrap introduces two custom resources:

• MockConfigTemplate: allows users to specify a pre-existing Secret containing a cloud-config or ignition file.
• MockConfig: when created, reports the spec.userDataSecret as status.dataSecretName.

Examples of these resources are provided below.

apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
kind: MockConfigTemplate
metadata:
  name: mockconfigtemplate-sample
spec:
  template:
    spec:
      userDataSecret: testuserdata

apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
kind: MockConfig
metadata:
  name: mockconfig-sample
  namespace: default
spec:
  dataSecretName: testuserdata
status:
  dataSecretName: testuserdata
  ready: true

We can now create a new MachineDeployment named testmock1 that utilizes MockBootstrap by referencing a MockConfigTemplate.
This MockConfigTemplate, in turn, references a secret that will be used as userData during the provisioning of the BareMetalHosts. Following a visual representation.

Expand to see the definition of the cloud-init used for testing

For testing purposes, the secret referenced as dataSecretName contains a simple cloud-config consisting of two main actions: it adds an SSH key to the host; and it creates a file that serves as a quick validation check to confirm that the initialisation process was successful.

## template: jinja
#cloud-config
write_files:
-   path: /tmp/FOO.txt
    owner: root:root
    permissions: '0640'
    content: |
      Hello, this is a cloud-config file!
users:
  - name: metal3
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-rsa AAAAB...

Expand for the full definition of the MachineDeployment and related objects.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: test1
    nodepool: nodepool-0
  name: test1mock
  namespace: metal3
spec:
  clusterName: test1
  replicas: 2
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: test1
      nodepool: nodepool-0
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: test1
        nodepool: nodepool-0
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
          kind: MockConfigTemplate
          name: mockconfigtemplate-sample
      clusterName: test1
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: Metal3MachineTemplate
        name: test1mock-node
      nodeDrainTimeout: 0s
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
kind: MockConfigTemplate
metadata:
  name: mockconfigtemplate-sample
spec:
  template:
    spec:
      userDataSecret: testuserdata
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
  name: test1mock-node
  namespace: metal3
spec:
  template:
    spec:
      dataTemplate:
        name: test1mock-node-template
      image:
        checksum: http://172.22.0.1/images/CentOS-Stream-GenericCloud-9-20240101.0.x86_64.raw.sha256sum
        checksumType: sha256
        format: raw
        url: http://172.22.0.1/images/CentOS-Stream-GenericCloud-9-20240101.0.x86_64.raw
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
  name: test1mock-node-template
  namespace: metal3
spec:
  clusterName: test1
  metaData:
    ipAddressesFromIPPool:
    - key: provisioningIP
      name: provisioning-pool
    objectNames:
    - key: name
      object: machine
    - key: local-hostname
      object: machine
    - key: local_hostname
      object: machine
    prefixesFromIPPool:
    - key: provisioningCIDR
      name: provisioning-pool
  networkData:
    links:
      ethernets:
      - id: enp1s0
        macAddress:
          fromHostInterface: enp1s0
        type: phy
      - id: enp2s0
        macAddress:
          fromHostInterface: enp2s0
        type: phy
    networks:
      ipv4:
      - id: externalv4
        ipAddressFromIPPool: externalv4-pool
        link: enp2s0
        routes:
        - gateway:
            fromIPPool: externalv4-pool
          network: 0.0.0.0
          prefix: 0
    services:
      dns:
      - 8.8.8.8

After creating the new MachineDeployment, the system's state has been updated and we see the following:

• Two BareMetalHosts, node-1 and node-6, have been successfully provisioned. This is great because this was our main objective
• However, the test1mock MachineDeployment is stuck in the ScalingUP phase and it reports 2 machines as Unavailable.
• The related CAPI Machines are stuck in a Provisioning state.

user@adcpu28:~/workspace/metal3-dev-env$ kubectl get clusters,mockconfigtemplates,mockconfigs,metal3clusters,metal3machinetemplate,machinedeployments,machinesets,machines,metal3machines,bmh
NAME                             CLUSTERCLASS   PHASE         AGE   VERSION
cluster.cluster.x-k8s.io/test1                  Provisioned   27h

NAME                                                                      AGE
mockconfigtemplate.bootstrap.cluster.x-k8s.io/mockconfigtemplate-sample   158m

NAME                                                          AGE
mockconfig.bootstrap.cluster.x-k8s.io/test1mock-lwjkp-mdpg9   21m
mockconfig.bootstrap.cluster.x-k8s.io/test1mock-lwjkp-qxrnd   21m

NAME                                                  AGE   READY   ERROR   CLUSTER   ENDPOINT
metal3cluster.infrastructure.cluster.x-k8s.io/test1   27h   true            test1     {"host":"192.168.111.249","port":6443}

NAME                                                                       AGE
metal3machinetemplate.infrastructure.cluster.x-k8s.io/test1-controlplane   27h
metal3machinetemplate.infrastructure.cluster.x-k8s.io/test1-workers        25h
metal3machinetemplate.infrastructure.cluster.x-k8s.io/test1mock-node       21m

NAME                                           CLUSTER   REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE       AGE   VERSION
machinedeployment.cluster.x-k8s.io/test1       test1     2          2       2         0             Running     25h   v1.31.0
machinedeployment.cluster.x-k8s.io/test1mock   test1     2                  2         2             ScalingUp   21m

NAME                                          CLUSTER   REPLICAS   READY   AVAILABLE   AGE   VERSION
machineset.cluster.x-k8s.io/test1-6nvzx       test1     2          2       2           25h   v1.31.0
machineset.cluster.x-k8s.io/test1mock-lwjkp   test1     2                              21m

NAME                                             CLUSTER   NODENAME            PROVIDERID                                 PHASE          AGE   VERSION
machine.cluster.x-k8s.io/test1-6nvzx-bss44       test1     test1-6nvzx-bss44   metal3://metal3/node-0/test1-6nvzx-bss44   Running        25h   v1.31.0
machine.cluster.x-k8s.io/test1-6nvzx-ns9fl       test1     test1-6nvzx-ns9fl   metal3://metal3/node-2/test1-6nvzx-ns9fl   Running        25h   v1.31.0
machine.cluster.x-k8s.io/test1-bhcf4             test1     test1-bhcf4         metal3://metal3/node-3/test1-bhcf4         Running        27h   v1.31.0
machine.cluster.x-k8s.io/test1-cw695             test1     test1-cw695         metal3://metal3/node-5/test1-cw695         Running        26h   v1.31.0
machine.cluster.x-k8s.io/test1-djvs9             test1     test1-djvs9         metal3://metal3/node-4/test1-djvs9         Running        26h   v1.31.0
machine.cluster.x-k8s.io/test1mock-lwjkp-mdpg9   test1                                                                    Provisioning   21m
machine.cluster.x-k8s.io/test1mock-lwjkp-qxrnd   test1                                                                    Provisioning   21m

NAME                                                                  AGE   PROVIDERID                                 READY   CLUSTER   PHASE
metal3machine.infrastructure.cluster.x-k8s.io/test1-6nvzx-bss44       25h   metal3://metal3/node-0/test1-6nvzx-bss44   true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-6nvzx-ns9fl       25h   metal3://metal3/node-2/test1-6nvzx-ns9fl   true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-bhcf4             27h   metal3://metal3/node-3/test1-bhcf4         true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-cw695             26h   metal3://metal3/node-5/test1-cw695         true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-djvs9             26h   metal3://metal3/node-4/test1-djvs9         true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1mock-lwjkp-mdpg9   21m                                                      test1
metal3machine.infrastructure.cluster.x-k8s.io/test1mock-lwjkp-qxrnd   21m                                                      test1

NAME                             STATE         CONSUMER                ONLINE   ERROR   AGE
baremetalhost.metal3.io/node-0   provisioned   test1-6nvzx-bss44       true             27h
baremetalhost.metal3.io/node-1   provisioned   test1mock-lwjkp-mdpg9   true             27h
baremetalhost.metal3.io/node-2   provisioned   test1-6nvzx-ns9fl       true             27h
baremetalhost.metal3.io/node-3   provisioned   test1-bhcf4             true             27h
baremetalhost.metal3.io/node-4   provisioned   test1-djvs9             true             27h
baremetalhost.metal3.io/node-5   provisioned   test1-cw695             true             27h
baremetalhost.metal3.io/node-6   provisioned   test1mock-lwjkp-qxrnd   true             27h

We can look at the current conditions reported by the Metal3Machine and investigate why we do not have a ProviderID.

user@adcpu28:~/workspace/metal3-dev-env$ kubectl get metal3machine.infrastructure.cluster.x-k8s.io/test1mock-lwjkp-qxrnd -o json | jq -r .status.conditions
[
  {
    "lastTransitionTime": "2024-11-15T13:50:03Z",
    "message": "requeuing, could not find node with label: metal3.io/uuid=a93c1c55-1dc1-4c3a-882a-9e2219615313. Object will be requeued after 30s",
    "reason": "SettingProviderIDOnNodeFailed",
    "severity": "Error",
    "status": "False",
    "type": "Ready"
  },
  {
    "lastTransitionTime": "2024-11-15T13:48:42Z",
    "status": "True",
    "type": "AssociateBMH"
  },
  {
    "lastTransitionTime": "2024-11-15T13:50:03Z",
    "message": "requeuing, could not find node with label: metal3.io/uuid=a93c1c55-1dc1-4c3a-882a-9e2219615313. Object will be requeued after 30s",
    "reason": "SettingProviderIDOnNodeFailed",
    "severity": "Error",
    "status": "False",
    "type": "KubernetesNodeReady"
  },
  {
    "lastTransitionTime": "2024-11-15T13:48:42Z",
    "status": "True",
    "type": "Metal3DataReady"
  }
]

The controller responsible for managing the Metal3Machine is attempting to set a ProviderID to a Kubernetes Node
in the target cluster, but is unable to find it. This is expected, as the provisioned host is not a Kubernetes node.

Expand to see the validation of the provisioned BareMetalHosts

To confirm that the host was provisioned correctly, we test accessing the host.
We successfully connect to the host using the SSH key specified in the testuserdata secret.
Additionally, we verify that the file FOO.txt, as described in the cloud-config, has been created on the host.
This confirms that the provisioning process was successful.

user@adcpu28:~/workspace/metal3-dev-env$ kubectl get bmh node-1 -o json | jq -r .status.hardware.nics | grep 192.168
    "ip": "192.168.222.21",
    "ip": "192.168.221.21",
    "ip": "192.168.111.21",
user@adcpu28:~/workspace/metal3-dev-env$ ssh metal3@192.168.222.21
Last login: Fri Nov 15 08:58:14 2024 from 192.168.222.1
[metal3@test1mock-lwjkp-mdpg9 ~]$ sudo cat /tmp/FOO.txt
Hello, this is a cloud-config file!
[metal3@test1mock-lwjkp-mdpg9 ~]$

Also looking at the current spec section in the BareMetalHost node-1 that is one of the two hosts selected for the MachineDeployment test1mock we see the same image set by Metal3MachineTemplate and the testuserdata secret set in the MockConfig

As a double check, we examine the current spec section of the BareMetalHost node-1, one of the two hosts selected for the MachineDeployment test1mock, and we observe that:

• The image specified in the Metal3MachineTemplate has been applied.
• The testuserdata secret, set in the MockConfig, has been successfully applied to the host.

This confirms that the configuration defined in the Metal3MachineTemplate and MockConfig has been correctly propagated to the BareMetalHost.

user@adcpu28:~/workspace/metal3-dev-env$ kubectl get bmh node-1 -o yaml | yq '.spec'

architecture: x86_64
automatedCleaningMode: metadata
bmc:
  address: redfish+http://192.168.111.1:8000/redfish/v1/Systems/fcaaf8e4-2390-44cb-a32a-072db484f5ec
  credentialsName: node-1-bmc-secret
bootMACAddress: 00:75:ef:22:00:7e
bootMode: legacy
consumerRef:
  apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
  kind: Metal3Machine
  name: test1mock-lwjkp-mdpg9
  namespace: metal3
image:
  checksum: http://172.22.0.1/images/CentOS-Stream-GenericCloud-9-20240101.0.x86_64.raw.sha256sum
  checksumType: sha256
  format: raw
  url: http://172.22.0.1/images/CentOS-Stream-GenericCloud-9-20240101.0.x86_64.raw
metaData:
  name: test1mock-lwjkp-mdpg9-metadata
  namespace: metal3
networkData:
  name: test1mock-lwjkp-mdpg9-networdata
  namespace: metal3
online: true
userData:
  name: testuserdata
  namespace: metal3

Considerations: While we have successfully provisioned a group of hosts using a MachineDeployment, there are some important caveats to note:

• The CAPI and CAPM3 objects are not in a clean state, indicating that the provisioning process is not fully complete.
• The Metal3Machine will never reach the Running state because the related controller never find the respective Node object in the target cluster.

Enabling Cluster-APIs management for non-Kubernetes hosts using Virtual Kubelets

To utilise CAPI resources for provisioning a group of hosts, we need to register the hosts as Nodes without allowing any Pod execution.
To achieve this, we investigate the use of virtual-kubelets.

A virtual-kubelet is an open-source implementation of the Kubernetes kubelet that enables us to masquerade platforms, hosts, or components as a kubelet.
The project provides a pluggable interface for developers to implement custom providers (not to be confused with CAPI providers) that define how kubelet actions should be handled.
The project includes a mock provider that joins the cluster as a node with the role "agent" and accepts Pod definitions, but does not run any containers. (A more suitable implementation would return an error at the creation of a Pod but for the purpose of evaluating such method the mock provider works fine.

To leverage virtual-kubelets with CAPI, we prototyped a new bootstrap provider called ClusterAPI Bootstrap Provider Virtual Kubelets (CABPV).

This provider:

• Installs a virtual-kubelet that registers the host as an unschedulable Kubernetes Node
• Allows users to initialise hosts using sections like files, commands, users, etc.
• Injects certificates and kubeconfig necessary for the virtual kubelet to access the cluster
• Generates a Secret to be set as userData during BareMetalHost provisioning

CABPV being a bootstrap provider, would introduce a template (VirtualKubeletConfigTemplate) and a config object (VirtualKubeletConfig). These include:

• Format, files, users, and commands sections (with the ability to expand to other components in a similar fashion to the KubeadmConfigSpec)
• spec.template.spec.virtual-kubelet.url: allows dynamic installation of the virtual-kubelet during provisioning if it's not already available in the image
• spec.template.spec.virtual-kubelet.provider: for use-cases where users develop their own virtual-kubelet implementing custom operations (more on this in the final considerations)

Here an example:

apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
kind: VirtualKubeletConfigTemplate
metadata:
  labels:
  name: virtualkubeletconfigtemplate-sample
spec:
  template:
    spec:
      virtual-kubelet:
        url: http://172.22.0.1/virtual-kubelet
        provider: mock
      format: cloud-config
      files:
        - content: |
            Hello!!!!
          owner: root:root
          path: /tmp/HELLO
          permissions: "0755"
      users:
        - name: metal3
          sshAuthorizedKeys:
            - ssh-rsa <KEY>
          sudo: ALL=(ALL) NOPASSWD:ALL

The manifest used to deploy a MachineDeployment (named test1virtkube) leveraging CABPV is similar to the previous one,
but in this case it references a VirtualKubeletConfigTemplate instead of a MockConfigTemplate.

Expand to inspect the manifest used to test CABPV.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: test1
    nodepool: nodepool-0
  name: test1virtkube
  namespace: metal3
spec:
  clusterName: test1
  replicas: 2
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: test1
      nodepool: nodepool-0
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: test1
        nodepool: nodepool-0
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha1
          kind: VirtualKubeletConfigTemplate
          name: virtualkubeletconfigtemplate-sample
      clusterName: test1
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: Metal3MachineTemplate
        name: test1virtkube-node
      nodeDrainTimeout: 0s
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
metadata:
  name: test1virtkube-node
  namespace: metal3
spec:
  template:
    spec:
      dataTemplate:
        name: test1virtkube-node-template
      image:
        checksum: http://172.22.0.1/images/CentOS-Stream-GenericCloud-9-20240101.0.x86_64.raw.sha256sum
        checksumType: sha256
        format: raw
        url: http://172.22.0.1/images/CentOS-Stream-GenericCloud-9-20240101.0.x86_64.raw
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
  name: test1virtkube-node-template
  namespace: metal3
spec:
  clusterName: test1
  metaData:
    ipAddressesFromIPPool:
    - key: provisioningIP
      name: provisioning-pool
    objectNames:
    - key: name
      object: machine
    - key: local-hostname
      object: machine
    - key: local_hostname
      object: machine
    prefixesFromIPPool:
    - key: provisioningCIDR
      name: provisioning-pool
  networkData:
    links:
      ethernets:
      - id: enp1s0
        macAddress:
          fromHostInterface: enp1s0
        type: phy
      - id: enp2s0
        macAddress:
          fromHostInterface: enp2s0
        type: phy
    networks:
      ipv4:
      - id: externalv4
        ipAddressFromIPPool: externalv4-pool
        link: enp2s0
        routes:
        - gateway:
            fromIPPool: externalv4-pool
          network: 0.0.0.0
          prefix: 0
    services:
      dns:
      - 8.8.8.8

Below the state of the system after creating the new MachineDeployment.

• The test1virtkube MachineDeployment is in the Running phase and it has 2 replicas, both of which are in a Ready state.
• Each Machine associated with the MachineDeployment has a reference to a Kubernetes node object in the target cluster, (column NODENAME).

user@adcpu28:~/workspace/metal3-dev-env$ kubectl get clusters,metal3clusters,metal3machinetemplate,machinedeployments,machinesets,machines,metal3machines,bmh
NAME                             CLUSTERCLASS   PHASE         AGE   VERSION
cluster.cluster.x-k8s.io/test1                  Provisioned   5d

NAME                                                  AGE   READY   ERROR   CLUSTER   ENDPOINT
metal3cluster.infrastructure.cluster.x-k8s.io/test1   5d    true            test1     {"host":"192.168.111.249","port":6443}

NAME                                                                       AGE
metal3machinetemplate.infrastructure.cluster.x-k8s.io/test1-controlplane   5d
metal3machinetemplate.infrastructure.cluster.x-k8s.io/test1-workers        4d23h
metal3machinetemplate.infrastructure.cluster.x-k8s.io/test1virtkube-node   2m39s

NAME                                               CLUSTER   REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE     VERSION
machinedeployment.cluster.x-k8s.io/test1           test1     2          2       2         0             Running   4d23h   v1.31.0
machinedeployment.cluster.x-k8s.io/test1virtkube   test1     2          2       2         0             Running   2m39s

NAME                                              CLUSTER   REPLICAS   READY   AVAILABLE   AGE     VERSION
machineset.cluster.x-k8s.io/test1-6nvzx           test1     2          2       2           4d23h   v1.31.0
machineset.cluster.x-k8s.io/test1virtkube-ts524   test1     2          2       2           2m38s

NAME                                                 CLUSTER   NODENAME                    PROVIDERID                                         PHASE     AGE     VERSION
machine.cluster.x-k8s.io/test1-6nvzx-bss44           test1     test1-6nvzx-bss44           metal3://metal3/node-0/test1-6nvzx-bss44           Running   4d23h   v1.31.0
machine.cluster.x-k8s.io/test1-6nvzx-ns9fl           test1     test1-6nvzx-ns9fl           metal3://metal3/node-2/test1-6nvzx-ns9fl           Running   4d23h   v1.31.0
machine.cluster.x-k8s.io/test1-bhcf4                 test1     test1-bhcf4                 metal3://metal3/node-3/test1-bhcf4                 Running   5d      v1.31.0
machine.cluster.x-k8s.io/test1-cw695                 test1     test1-cw695                 metal3://metal3/node-5/test1-cw695                 Running   5d      v1.31.0
machine.cluster.x-k8s.io/test1-djvs9                 test1     test1-djvs9                 metal3://metal3/node-4/test1-djvs9                 Running   5d      v1.31.0
machine.cluster.x-k8s.io/test1virtkube-ts524-bf78j   test1     test1virtkube-ts524-bf78j   metal3://metal3/node-6/test1virtkube-ts524-bf78j   Running   2m38s
machine.cluster.x-k8s.io/test1virtkube-ts524-l9pzv   test1     test1virtkube-ts524-l9pzv   metal3://metal3/node-1/test1virtkube-ts524-l9pzv   Running   2m38s

NAME                                                                      AGE     PROVIDERID                                         READY   CLUSTER   PHASE
metal3machine.infrastructure.cluster.x-k8s.io/test1-6nvzx-bss44           4d23h   metal3://metal3/node-0/test1-6nvzx-bss44           true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-6nvzx-ns9fl           4d23h   metal3://metal3/node-2/test1-6nvzx-ns9fl           true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-bhcf4                 5d      metal3://metal3/node-3/test1-bhcf4                 true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-cw695                 5d      metal3://metal3/node-5/test1-cw695                 true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1-djvs9                 5d      metal3://metal3/node-4/test1-djvs9                 true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1virtkube-ts524-bf78j   2m38s   metal3://metal3/node-6/test1virtkube-ts524-bf78j   true    test1
metal3machine.infrastructure.cluster.x-k8s.io/test1virtkube-ts524-l9pzv   2m38s   metal3://metal3/node-1/test1virtkube-ts524-l9pzv   true    test1

NAME                             STATE         CONSUMER                    ONLINE   ERROR   AGE
baremetalhost.metal3.io/node-0   provisioned   test1-6nvzx-bss44           true             5d1h
baremetalhost.metal3.io/node-1   provisioned   test1virtkube-ts524-l9pzv   true             5d1h
baremetalhost.metal3.io/node-2   provisioned   test1-6nvzx-ns9fl           true             5d1h
baremetalhost.metal3.io/node-3   provisioned   test1-bhcf4                 true             5d1h
baremetalhost.metal3.io/node-4   provisioned   test1-djvs9                 true             5d1h
baremetalhost.metal3.io/node-5   provisioned   test1-cw695                 true             5d1h
baremetalhost.metal3.io/node-6   provisioned   test1virtkube-ts524-bf78j   true             5d1h

Upon inspecting the cluster, we observe that 2 new Node objects have been created, labeled with the role kubernetes.io/role: agent.
These Node objects were dynamically created when the virtual-kubelet successfully joined the cluster.

user@adcpu28:~/workspace/metal3-dev-env$ kubectl get nodes --kubeconfig test_kubeconfig -o wide
NAME                        STATUS   ROLES           AGE     VERSION                               INTERNAL-IP       EXTERNAL-IP   OS-IMAGE          KERNEL-VERSION          CONTAINER-RUNTIME
test1-6nvzx-bss44           Ready    <none>          4d23h   v1.31.0                               192.168.111.104   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4
test1-6nvzx-ns9fl           Ready    <none>          4d23h   v1.31.0                               192.168.111.103   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4
test1-bhcf4                 Ready    control-plane   5d      v1.31.0                               192.168.111.100   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4
test1-cw695                 Ready    control-plane   5d      v1.31.0                               192.168.111.102   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4
test1-djvs9                 Ready    control-plane   5d      v1.31.0                               192.168.111.101   <none>        CentOS Stream 9   5.14.0-480.el9.x86_64   cri-o://1.30.4
test1virtkube-ts524-bf78j   Ready    agent           106s    v1.15.2-vk-v1.11.0-18-g1f5e83b1-dev                     <none>        <unknown>         <unknown>               <unknown>
test1virtkube-ts524-l9pzv   Ready    agent           103s    v1.15.2-vk-v1.11.0-18-g1f5e83b1-dev                     <none>        <unknown>         <unknown>               <unknown>

Expand to see the state of a Node registered via the virtual-kubelet.

kubectl get node test1virtkube-ts524-bf78j --kubeconfig test_kubeconfig -o yaml

apiVersion: v1
kind: Node
metadata:
  annotations:
    cluster.x-k8s.io/cluster-name: test1
    cluster.x-k8s.io/cluster-namespace: metal3
    cluster.x-k8s.io/labels-from-machine: ""
    cluster.x-k8s.io/machine: test1virtkube-ts524-bf78j
    cluster.x-k8s.io/owner-kind: MachineSet
    cluster.x-k8s.io/owner-name: test1virtkube-ts524
    node.alpha.kubernetes.io/ttl: "0"
    virtual-kubelet.io/last-applied-node-status: '{"capacity":{"cpu":"2","memory":"32Gi","pods":"128"},"allocatable":{"cpu":"2","memory":"32Gi","pods":"128"},"phase":"Running","conditions":[{"type":"Ready","status":"True","lastHeartbeatTime":"2024-11-19T12:00:21Z","lastTransitionTime":"2024-11-19T11:58:21Z","reason":"KubeletReady","message":"Kubelet
      is ready"},{"type":"OutOfDisk","status":"False","lastHeartbeatTime":"2024-11-19T12:00:21Z","lastTransitionTime":"2024-11-19T11:58:21Z","reason":"KubeletHasSufficientDisk","message":"kubelet
      has sufficient disk space available"},{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2024-11-19T12:00:21Z","lastTransitionTime":"2024-11-19T11:58:21Z","reason":"KubeletHasSufficientMemory","message":"kubelet
      has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2024-11-19T12:00:21Z","lastTransitionTime":"2024-11-19T11:58:21Z","reason":"KubeletHasNoDiskPressure","message":"kubelet
      has no disk pressure"},{"type":"NetworkUnavailable","status":"False","lastHeartbeatTime":"2024-11-19T12:00:21Z","lastTransitionTime":"2024-11-19T11:58:21Z","reason":"RouteCreated","message":"RouteController
      created a route"}],"addresses":[{"type":"InternalIP","address":""}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"v1.15.2-vk-v1.11.0-18-g1f5e83b1-dev","kubeProxyVersion":"","operatingSystem":"linux","architecture":"amd64"}}'
    virtual-kubelet.io/last-applied-object-meta: '{"name":"test1virtkube-ts524-bf78j","uid":"83fe3875-a33a-4bbb-b471-8c7c0a1c422a","creationTimestamp":null,"labels":{"alpha.service-controller.kubernetes.io/exclude-balancer":"true","kubernetes.io/hostname":"test1virtkube-ts524-bf78j","kubernetes.io/role":"agent","metal3.io/uuid":"a93c1c55-1dc1-4c3a-882a-9e2219615313","node.kubernetes.io/exclude-from-external-load-balancers":"true","type":"virtual-kubelet"}}'
  creationTimestamp: "2024-11-19T11:58:21Z"
  labels:
    alpha.service-controller.kubernetes.io/exclude-balancer: "true"
    kubernetes.io/hostname: test1virtkube-ts524-bf78j
    kubernetes.io/role: agent
    metal3.io/uuid: a93c1c55-1dc1-4c3a-882a-9e2219615313
    node.kubernetes.io/exclude-from-external-load-balancers: "true"
    type: virtual-kubelet
  name: test1virtkube-ts524-bf78j
  resourceVersion: "960307"
  uid: 83fe3875-a33a-4bbb-b471-8c7c0a1c422a
spec:
  podCIDR: 192.168.9.0/24
  podCIDRs:
  - 192.168.9.0/24
  providerID: metal3://metal3/node-6/test1virtkube-ts524-bf78j
  taints:
  - effect: NoSchedule
    key: virtual-kubelet.io/provider
    value: mock
status:
  addresses:
  - address: ""
    type: InternalIP
  allocatable:
    cpu: "2"
    memory: 32Gi
    pods: "128"
  capacity:
    cpu: "2"
    memory: 32Gi
    pods: "128"
  conditions:
  - lastHeartbeatTime: "2024-11-19T12:00:21Z"
    lastTransitionTime: "2024-11-19T11:58:21Z"
    message: Kubelet is ready
    reason: KubeletReady
    status: "True"
    type: Ready
  - lastHeartbeatTime: "2024-11-19T12:00:21Z"
    lastTransitionTime: "2024-11-19T11:58:21Z"
    message: kubelet has sufficient disk space available
    reason: KubeletHasSufficientDisk
    status: "False"
    type: OutOfDisk
  - lastHeartbeatTime: "2024-11-19T12:00:21Z"
    lastTransitionTime: "2024-11-19T11:58:21Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2024-11-19T12:00:21Z"
    lastTransitionTime: "2024-11-19T11:58:21Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2024-11-19T12:00:21Z"
    lastTransitionTime: "2024-11-19T11:58:21Z"
    message: RouteController created a route
    reason: RouteCreated
    status: "False"
    type: NetworkUnavailable
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  nodeInfo:
    architecture: amd64
    bootID: ""
    containerRuntimeVersion: ""
    kernelVersion: ""
    kubeProxyVersion: ""
    kubeletVersion: v1.15.2-vk-v1.11.0-18-g1f5e83b1-dev
    machineID: ""
    operatingSystem: linux
    osImage: ""
    systemUUID: ""
  phase: Running

Following a visual representation of the objects created.

Compared to the MachineDeployment, now we reference a VirtualKubeletConfigTemplate as template.bootstrap.configRef
and on each host provisioned there is a virtual-kubelet that registers the host as a Node in the target cluster.

Expand to see more information about the userData secret generated by the bootstrap provider

We can look at one of the secrets used as userData during the provisioning of a BareMetalHost.
We truncate long files for ease of reading.

The bootstrap provider:

• adds the certificates related to the target cluster
• adds the kubeconfig for said cluster. This is used by the virtual-kubelet to register the node
• downloads the virtual-kubelet using the url defined in the template
• adds a service definition to run the virtual-kubelet with the selected provider
• sets the label "metal3.io/uuid":"{{ ds.meta_data.uuid }}" enabling the CAPM3 to tag the Kubernetes node object correctly

kubectl get secrets test1virtkube-ts524-bf78j -o json | jq -r .data.value | base64 -d

## template: jinja
#cloud-config

write_files:
-   path: /tmp/HELLO
    owner: root:root
    permissions: '0755'
    content: |
      Hello!!!!

-   path: /etc/kubernetes/kubeconfig
    owner: root:root
    permissions: '0644'
    content: |
      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority-data: LS0...
          server: https://192.168.111.249:6443
        name: test1
      contexts:
      - context:
          cluster: test1
          user: test1-admin
        name: test1-admin@test1
      current-context: test1-admin@test1
      kind: Config
      preferences: {}
      users:
      - name: test1-admin
        user:
          client-certificate-data: LS0...
          client-key-data: LS0...

-   path: /etc/kubernetes/pki/ca.crt
    owner: root:root
    permissions: '0644'
    content: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

-   path: /etc/kubernetes/pki/ca.key
    owner: root:root
    permissions: '0600'
    content: |
      -----BEGIN RSA PRIVATE KEY-----
      ...
      -----END RSA PRIVATE KEY-----

-   path: /etc/virtual-kubelet/vkubelet-cfg.json
    owner: root:root
    permissions: '0644'
    content: |

            {
              "{{ ds.meta_data.local_hostname }}": {
              "labels":{
                "metal3.io/uuid":"{{ ds.meta_data.uuid }}"
              }
              }
            }
-   path: /lib/systemd/system/virtual-kubelet.service
    owner: root:root
    permissions: '0640'
    content: |

      [Unit]
      Description=virtual-kubelet
      [Service]
      Environment=KUBECONFIG=/etc/kubernetes/kubeconfig
      Environment=APISERVER_CERT_LOCATION=/etc/kubernetes/pki/ca.crt
      Environment=APISERVER_KEY_LOCATION=/etc/kubernetes/pki/ca.key
      Environment=KUBERNETES_SERVICE_HOST=192.168.111.249
      Environment=KUBERNETES_SERVICE_PORT=6443
      ExecStart=/usr/local/bin/virtual-kubelet --provider mock  --nodename "{{ ds.meta_data.local_hostname }}" --provider-config /etc/virtual-kubelet/vkubelet-cfg.json  --kubeconfig /etc/kubernetes/kubeconfig
      [Install]
      WantedBy=multi-user.target

runcmd:
  - "curl http://172.22.0.1/virtual-kubelet  --output /usr/local/bin/virtual-kubelet"
  - "chmod a+x /usr/local/bin/virtual-kubelet"
  - "systemctl enable --now virtual-kubelet"
users:
  - name: metal3
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-rsa <KEY>

Key Considerations

Considerations:

• This approach allows us to reuse existing CAPI/CAPM3 resources and we avoid implementing an additional operator to manage non-Kubernetes hosts.
• While this approach eliminates the need for Kubernetes worker nodes, it still requires a Kubernetes control plane. However, using BareMetalHosts solely to run a control plane for registering virtual-kubelets is inefficient. Hosting the control plane in a namespace could mitigate this issue.
• Virtual-kubelets provide administrators with a hook into the host, enabling the creation of custom virtual-kubelets to tailor operations such as restarting or configuring services.

What to investigate next? Our primary objective was to provision BareMetalHosts, which led us to focus on Metal3. However, CABPK is used by multiple providers and this raises an interesting question: Can we leverage this method to provision hosts across various providers?

[lentzi90]

Hi!
Sorry for not commenting on this earlier. I have had it on my list for a while but never got around to it. I wanted to comment now so that you know it is not forgotten. Unfortunately I did not yet have time to go through it fully, and I go on vacation today. Rest assured that I will take a look when back in January!

[mgazz]

@lentzi90 thank you for the update.

[lentzi90]

Thank you for the thorough writeup! It is an interesting approach, or perhaps rather workaround for the missing functionality in Metal3. Hopefully people will find it useful.
Personally, I think it is a bit too much of a hack and with the obvious considerations you already mentioned I wouldn't be very comfortable doing this in production.

That said, I think this discussion also shows that the demand for non-Kubernetes hosts is still there. I know there are multiple people who want this and I think we may be able to move forward with it this year. Hopefully we can agree on a design already before the summer. 🙂

[mgazz]

Hello @lentzi90,

First of all, thank you for taking the time to reply.



Personally, I think it is a bit too much of a hack and with the obvious considerations you already mentioned I wouldn't be very comfortable doing this in production.

What I described above is a PoC so by no means it should be used in production as it is. Additional effort would be necessary in both design/implementation/testing to adopt such approach in production.
If possible, could you elaborate on which part you consider a hack and why? (Maybe different design choices would me improve this idea.)

I know there are multiple people who want this and I think we may be able to move forward with it this year. Hopefully we can agree on a design already before the summer.

That is great, I would like to participate to the conversation if possible. Is there a workgroup or issue on this already?

[lentzi90]

If possible, could you elaborate on which part you consider a hack and why? (Maybe different design choices would me improve this idea.)

Requiring a CAPI cluster and control plane just for getting the templating functionality feels like a hack and major inconvenience for me. It is especially bothering that there is then a CAPI Machine and a Kubernetes Node related to the provisioned BMH. It is confusing and adds potential for breaking things. For example, what if a user doesn't understand this setup fully and then discovers the "extra" machine and tries to get rid of it? Suddenly you risk nuking non-k8s resources that look unrelated, but are really tied to the k8s resources.

That is great, I would like to participate to the conversation if possible. Is there a workgroup or issue on this already?

Check this discussion: https://github.com/orgs/metal3-io/discussions/2029
We are planning a Metal3 Team Meetup soon where we would likely discuss some of these things. Check this email: https://groups.google.com/g/metal3-dev/c/Wl2DIdr_ZQ0