oneke_architecture

Architecture Overview

OneKE is available as a Virtual Appliance from the OpenNebula Public MarketPlace.

Let's take a closer look at the OneKE 1.27 MarketPlace app:

$ onemarketapp list -f NAME~'OneKE 1.27' -l NAME --no-header
OneKE 1.27 Storage
OneKE 1.27 OS disk
Service OneKE 1.27
OneKE 1.27
OneKE 1.27 VNF
OneKE 1.27 Storage disk

A specific version of OneKE consists of:

• A single OneFlow template Service OneKE .. used to instantiate a cluster.
• VM templates OneKE .. VNF, OneKE .., OneKE .. Storage used to instantiate VMs.
• Disk images OneKE .. OS disk, OneKE .. Storage disk (all Kubernetes nodes are cloned from the OneKE .. OS disk image).

📗 Note: A service template links to VM templates which link to disk images, in such a way that everything is recursively downloaded when importing the Virtual Appliance in the OpenNebula Cloud.

OneFlow Service

The OneKE virtual appliance is implemented as a OneFlow Service. OneFlow allows you to define, execute, and manage multi-tiered applications, known as Services, composed of interconnected Virtual Machines with deployment dependencies between them. Each group of Virtual Machines is deployed and managed as a single entity (called role).

📗 Note: For a full OneFlow API/template reference, please refer to the OneFlow Specification.

OneKE Service comprises four different Roles:

• VNF: Load Balancer for Control-Plane and Ingress Traffic
• Master: Control-Plane nodes
• Worker: Nodes to run application workloads
• Storage: Dedicated storage nodes for Persistent Volume replicas

To check the roles defined in the service template, use the following command:

$ oneflow-template show -j 'Service OneKE 1.27' | jq -r '.DOCUMENT.TEMPLATE.BODY.roles[].name'
vnf
master
worker
storage

VNF (Virtual Network Functions) Role

VNF is a multi-node service that provides Routing, NAT, and Load-Balancing to OneKE clusters. VNF has been implemented on top of Keepalived which allows for basic HA/Failover functionality via Virtual IPs (VIPs).

OneKE operates in a dual subnet environment: VNF facilitates NAT and Routing between public and private VNETs. When the public VNET acts as a gateway to the public Internet, it also enables Internet connectivity to all internal VMs.

Dedicated documentation for VNF can be found in the VNF documentation.

Master Role

The master role is responsible for running RKE2's Control Plane, managing the etcd database, API server, controller manager, scheduler, along with the worker nodes. It has been implemented according to principles defined in RKE2's High Availability section. Specifically, the fixed registration address is an HAProxy instance exposing TCP port 9345 on a VNF node.

Worker Role

The worker role deploys standard RKE2 nodes without any taints or labels and serves as the default destination for regular workloads.

Storage Role

The storage role deploys labeled and tainted nodes specifically designated to run only Longhorn replicas.

Selectors and tolerations can be applied to deploy pods into storage nodes. Here's an example in YAML format:

tolerations:
  - key: node.longhorn.io/create-default-disk
    value: "true"
    operator: Equal
    effect: NoSchedule
nodeSelector:
  node.longhorn.io/create-default-disk: "true"

OneKE includes a retain version of the default Longhorn storage class, defined as:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-retain
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Retain
volumeBindingMode: Immediate
parameters:
  fsType: "ext4"
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"
  fromBackup: ""

Further information about Kubernetes storage classes can be found in the storage classes documentation.

Warning: Each storage node requires a dedicated storage block device attached to the VM (/dev/vdb by default) to hold Longhorn's replicas (mounted at /var/lib/longhorn/). Deleting a cluster will also remove all its Longhorn replicas, so always ensure to back up your data!

Networking

OneKE's OneFlow Service necessitates two networks: a public and a private VNET. These can be simple bridged networks.

Consider the following network settings:

• The public VNET/subnet: 10.2.11.0/24 with the IPv4 range 10.2.11.200-10.2.11.249, providing public Internet access via NAT.
• The private VNET/subnet: 172.20.0.0/24 with the IPv4 range 172.20.0.100-172.20.0.199, DNS context value 1.1.1.1, and complete isolation from the public Internet.

Avoid including VIP addresses within VNET ranges to prevent possible conflicts. For example:

VIP                          IPv4
================================
ONEAPP_VROUTER_ETH0_VIP0      10.2.11.86
ONEAPP_VROUTER_ETH1_VIP0      172.20.0.86
================================

graph LR
    Internet[Internet] -- Public VNF -- VNF
    Public VNF -- Master -- Master
    Public VNF -- Worker -- Worker
    Public VNF -- Storage -- Storage

    subgraph VNF
        style VNF fill:#fff,stroke:#333,stroke-width:2px
        VNF[Load Balancer: 10.2.11.86]
        VNF -->|NAT ⇅| Master
        VNF --> Worker
        VNF -->|NAT ⇅| Storage
    end

    subgraph Master
        style Master fill:#fff,stroke:#333,stroke-width:2px
        Master[172.20.0.101<br>GW: 172.20.0.86<br>DNS: 1.1.1.1]
    end

    subgraph Worker
        style Worker fill:#fff,stroke:#333,stroke-width:2px
        Worker[172.20.0.102<br>GW: 172.20.0.86<br>DNS: 1.1.1.1]
    end

    subgraph Storage
        style Storage fill:#fff,stroke:#333,stroke-width:2px
        Storage[172.20.0.103<br>GW: 172.20.0.86<br>DNS: 1.1.1.1]
    end

Loading