oneke_deploy

Deploying OneKE

In this section we focus on some advanced topics when deploying your Kubernetes clusters:

High Availability (HA) deployments
Using affinity rules in HA deployments
Using Terraform to create your OneKE Kubernetes clusters
Customize the Kubernetes components deployed with the cluster

For those seeking a quick start, we recommend referring to the Quick Start guide here, which provides step-by-step instructions to get up and running with OneKE.

High Availability OneKE

Important

For the highly-available setup both VIPs need to be provided.

By default, OneKE Virtual Appliance is preconfigured to work as a non-Highly-Available K8s cluster, since OneFlow Service Templates deploys each service role as a single VM. Kubernetes High-Availability is about setting up a Kubernetes cluster, along with its components, in such a way that there is no single point of failure. To achieve high-availability, the following OneKE components should be scaled up: VNF (at least 2 VMs), master (at least 3 VMs) and storage (at least 2 VMs), see figure below:

Public Network
─────────────────────────────────────────┬───────────────────────────────────────────────────────────
                                         │ VIP0 / FLOATING_IP (3.27.6.5) (ep0.eth0.vr)
                       ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ┐
                       │ OneKE vnf       │     │
                       │ ┌──eth0──┐ ┌──eth0──┐ │
                       │ │        │ │        │ │
                       │ │  VM-1  │ │  VM-2  │ │
                       │ │        │ │        │ │
                       │ └──eth1──┘ └──eth1──┘ │
                       │     │           │     │
                       │  10.0.0.2    10.0.0.3 │
                       │                 │     │
                       └ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ┘
                                         │ VIP1 / FLOATING_IP (10.0.0.1) (ep0.eth1.vr)
───────┬───────────┬─────────┬───────────┴──┬───────────┬─────────────┬───────────┬─────────┬────────
Private Network    │         │              │           │             │           │         │
       │           │         │              │           │             │           │         │
 ┌ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ┼ ─ ─ ─┐ ┌ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ┐ ┌ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ┼ ─ ─ ─┐
 │ OneKE master    │         │      │ │ OneKE worker    │     │ │ OneKE storage   │         │      │
 │     │           │         │      │ │     │           │     │ │     │           │         │      │
 │  10.0.0.4    10.0.0.5   10.0.0.6 │ │  10.0.0.7    10.0.0.8 │ │  10.0.0.9  10.0.0.10   10.0.0.11 │
 │     │           │         │      │ │     │           │     │ │     │           │         │      │
 │ ┌──eth0──┐ ┌──eth0──┐ ┌──eth0──┐ │ │ ┌──eth0──┐ ┌──eth0──┐ │ │ ┌──eth0──┐ ┌──eth0──┐ ┌──eth0──┐ │
 │ │        │ │        │ │        │ │ │ │        │ │        │ │ │ │        │ │        │ │        │ │
 │ │  VM-3  │ │  VM-4  │ │  VM-5  │ │ │ │  VM-6  │ │  VM-7  │ │ │ │  VM-8  │ │  VM-9  │ │  VM-10 │ │
 │ │        │ │        │ │        │ │ │ │        │ │        │ │ │ │        │ │        │ │        │ │
 │ └────────┘ └────────┘ └────────┘ │ │ └────────┘ └────────┘ │ │ └────────┘ └────────┘ └────────┘ │
 └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘

Download the Service OneKE 1.27 (or the airgapped version Service OneKE 1.27a):

$ onemarketapp export 'Service OneKE 1.27' 'Service OneKE 1.27' --datastore 1
IMAGE
    ID: 202
    ID: 203
    ID: 204
VMTEMPLATE
    ID: 204
    ID: 205
    ID: 206
SERVICE_TEMPLATE
    ID: 104

Adjust Flow and VM templates to meet you needs. (You may want to increase CPU and MEMORY for workers, add HOT_RESIZE=[...] or setup anti-affined VM groups, etc.)
```
$ oneflow-template update 104
```
```
$ onetemplate update 205
```

Instantiate the Flow service:

$ cat >/tmp/OneKE-instantiate <<'EOF'
{
    "name": "OneKE/1",
    "networks_values": [
        {"Public": {"id": "0"}},
        {"Private": {"id": "1"}}
    ],
    "custom_attrs_values": {
        "ONEAPP_VROUTER_ETH0_VIP0": "3.27.6.5",
        "ONEAPP_VROUTER_ETH1_VIP0": "10.0.0.1",

        "ONEAPP_RKE2_SUPERVISOR_EP": "",
        "ONEAPP_K8S_CONTROL_PLANE_EP": "",
        "ONEAPP_K8S_EXTRA_SANS": "localhost,127.0.0.1,k8s.yourdomain.it",

        "ONEAPP_K8S_MULTUS_ENABLED": "NO",
        "ONEAPP_K8S_MULTUS_CONFIG": "",
        "ONEAPP_K8S_CNI_PLUGIN": "cilium",
        "ONEAPP_K8S_CNI_CONFIG": "",
        "ONEAPP_K8S_CILIUM_RANGE": "",

        "ONEAPP_K8S_METALLB_ENABLED": "NO",
        "ONEAPP_K8S_METALLB_CONFIG": "",
        "ONEAPP_K8S_METALLB_RANGE": "",

        "ONEAPP_K8S_LONGHORN_ENABLED": "YES",
        "ONEAPP_STORAGE_DEVICE": "/dev/vdb",
        "ONEAPP_STORAGE_FILESYSTEM": "xfs",

        "ONEAPP_K8S_TRAEFIK_ENABLED": "YES",
        "ONEAPP_VNF_HAPROXY_INTERFACES": "eth0",
        "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "30",
        "ONEAPP_VNF_HAPROXY_LB0_PORT": "9345",
        "ONEAPP_VNF_HAPROXY_LB1_PORT": "6443",
        "ONEAPP_VNF_HAPROXY_LB2_PORT": "443",
        "ONEAPP_VNF_HAPROXY_LB3_PORT": "80",

        "ONEAPP_VNF_DNS_ENABLED": "YES",
        "ONEAPP_VNF_DNS_INTERFACES": "eth1",
        "ONEAPP_VNF_DNS_NAMESERVERS": "1.1.1.1,8.8.8.8",
        "ONEAPP_VNF_NAT4_ENABLED": "YES",
        "ONEAPP_VNF_NAT4_INTERFACES_OUT": "eth0",
        "ONEAPP_VNF_ROUTER4_ENABLED": "YES",
        "ONEAPP_VNF_ROUTER4_INTERFACES": "eth0,eth1"
    }
}
EOF
$ oneflow-template instantiate 'Service OneKE 1.27' /tmp/OneKE-instantiate
ID: 105

After successful deployment the Flow service should be in the RUNNING state:

$ oneflow list
  ID USER     GROUP    NAME                           STARTTIME STAT
 105 oneadmin oneadmin Service OneKE 1.27        02/15 09:46:56 RUNNING

Scale Flow roles (if required):

$ oneflow scale 105 vnf 2

$ oneflow scale 105 master 3

$ oneflow scale 105 worker 2

$ oneflow scale 105 storage 3

Note

Cardinality of the storage role is set to 0 by default to allow deployments with Longhorn disabled.

Note

You can set the cardinilaty of each role in step 2 when you update the template.

Warning

You can scale the master role up to an odd number of masters, but be careful while scaling down as it may break your cluster. If you require multi-master HA, just start with a single master and then scale up to 3 and keep it that way.

After a while we can examine the service log:

$ oneflow show 'OneKE/1'
...
LOG MESSAGES
05/11/23 18:30 [I] New state: DEPLOYING_NETS
05/11/23 18:30 [I] New state: DEPLOYING
05/11/23 18:39 [I] New state: RUNNING
05/11/23 18:43 [I] Role master scaling up from 1 to 3 nodes
05/11/23 18:43 [I] New state: SCALING
05/11/23 18:52 [I] New state: COOLDOWN
05/11/23 18:55 [I] New state: RUNNING

And afterwards we can list cluster nodes using kubectl:

$ kubectl get nodes
NAME                    STATUS   ROLES                       AGE     VERSION
oneke-ip-172-20-0-101   Ready    control-plane,etcd,master   31m     v1.27.2+rke2r1
oneke-ip-172-20-0-102   Ready    <none>                      28m     v1.27.2+rke2r1
oneke-ip-172-20-0-103   Ready    <none>                      28m     v1.27.2+rke2r1
oneke-ip-172-20-0-104   Ready    control-plane,etcd,master   11m     v1.27.2+rke2r1
oneke-ip-172-20-0-105   Ready    control-plane,etcd,master   10m     v1.27.2+rke2r1

Warning

Please plan ahead and avoid scaling down master and storage roles as it may break ETCD's quorum or cause data loss. There is no obvious restriction for the worker role, however. It can be safely rescaled at will.

Anti-affinity

VMs related to the same role should be scheduled on different physical hosts in an HA setup to guarantee HA in case of a host failure. OpenNebula provides VM Group resources to achieve proper Host/VM affinity/anti-affinity.

In the following section, we provide an example of how to create VM Group resources and how to modify OneKE's OneFlow Service Template to include VM groups.

Let's assume that epsilon and omicron are hosts we want to use to deploy OneKE; a VM Group may be created in the following way:

$ cat >/tmp/OneKE-vmgroup <<'EOF'
NAME = "Service OneKE 1.27"
ROLE = [
    NAME         = "vnf",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "master",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "worker",
    HOST_AFFINED = "epsilon,omicron"
]
ROLE = [
    NAME         = "storage",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
EOF

$ onevmgroup create /tmp/OneKE-vmgroup
ID: 1

Important

The worker role does not have POLICY defined, this allows you to reuse hosts multiple times!

Now, let's modify the OneKE OneFlow Service Template:

$ oneflow-template show 'Service OneKE 1.27' --json | >/tmp/OneKE-update.json jq -r --arg vmgroup 'Service OneKE 1.27' -f /dev/fd/3 3<<'EOF'
.DOCUMENT.TEMPLATE.BODY | del(.registration_time) | . += {
  roles: .roles | map(
    .vm_template_contents = "VMGROUP=[VMGROUP_NAME=\"\($vmgroup)\",ROLE=\"\(.name)\"]\n" + .vm_template_contents
  )
}
EOF

Content of the update (/tmp/OneKE-update.json) will look like this:

{
  "name": "Service OneKE 1.27",
  "deployment": "straight",
  "description": "",
  "roles": [
    {
      "name": "vnf",
      "cardinality": 1,
      "min_vms": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"vnf\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Public\" ]\nNIC = [\n  NAME = \"NIC1\",\n  NETWORK_ID = \"$Private\" ]\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_VNF_HAPROXY_INTERFACES = \"$ONEAPP_VNF_HAPROXY_INTERFACES\"\nONEAPP_VNF_HAPROXY_REFRESH_RATE = \"$ONEAPP_VNF_HAPROXY_REFRESH_RATE\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\nONEAPP_VNF_DNS_INTERFACES = \"$ONEAPP_VNF_DNS_INTERFACES\"\nONEAPP_VNF_DNS_NAMESERVERS = \"$ONEAPP_VNF_DNS_NAMESERVERS\"\nONEAPP_VNF_NAT4_ENABLED = \"$ONEAPP_VNF_NAT4_ENABLED\"\nONEAPP_VNF_NAT4_INTERFACES_OUT = \"$ONEAPP_VNF_NAT4_INTERFACES_OUT\"\nONEAPP_VNF_ROUTER4_ENABLED = \"$ONEAPP_VNF_ROUTER4_ENABLED\"\nONEAPP_VNF_ROUTER4_INTERFACES = \"$ONEAPP_VNF_ROUTER4_INTERFACES\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 29
    },
    {
      "name": "master",
      "parents": [
        "vnf"
      ],
      "cardinality": 1,
      "min_vms": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"master\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_EXTRA_SANS = \"$ONEAPP_K8S_EXTRA_SANS\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_MULTUS_CONFIG = \"$ONEAPP_K8S_MULTUS_CONFIG\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_CNI_CONFIG = \"$ONEAPP_K8S_CNI_CONFIG\"\nONEAPP_K8S_CILIUM_RANGE = \"$ONEAPP_K8S_CILIUM_RANGE\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_METALLB_CONFIG = \"$ONEAPP_K8S_METALLB_CONFIG\"\nONEAPP_K8S_METALLB_RANGE = \"$ONEAPP_K8S_METALLB_RANGE\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 30
    },
    {
      "name": "worker",
      "parents": [
        "vnf"
      ],
      "cardinality": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"worker\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 30
    },
    {
      "name": "storage",
      "parents": [
        "vnf"
      ],
      "cardinality": 0,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"storage\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_STORAGE_DEVICE = \"$ONEAPP_STORAGE_DEVICE\"\nONEAPP_STORAGE_FILESYSTEM = \"$ONEAPP_STORAGE_FILESYSTEM\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 31
    }
  ],
  "networks": {
    "Public": "M|network|Public||id:",
    "Private": "M|network|Private||id:"
  },
  "custom_attrs": {
    "ONEAPP_VROUTER_ETH0_VIP0": "O|text|Control Plane Endpoint VIP (IPv4)||",
    "ONEAPP_VROUTER_ETH1_VIP0": "O|text|Default Gateway VIP (IPv4)||",
    "ONEAPP_RKE2_SUPERVISOR_EP": "O|text|RKE2 Supervisor endpoint||ep0.eth0.vr:9345",
    "ONEAPP_K8S_CONTROL_PLANE_EP": "O|text|Control Plane endpoint||ep0.eth0.vr:6443",
    "ONEAPP_K8S_EXTRA_SANS": "O|text|ApiServer extra certificate SANs||localhost,127.0.0.1,ep0.eth0.vr,${vnf.TEMPLATE.CONTEXT.ETH0_IP}",
    "ONEAPP_K8S_MULTUS_ENABLED": "O|boolean|Enable Multus||NO",
    "ONEAPP_K8S_MULTUS_CONFIG": "O|text64|Multus custom config (default none)||",
    "ONEAPP_K8S_CNI_PLUGIN": "O|list|CNI plugin supported by RKE2|canal,calico,cilium|cilium",
    "ONEAPP_K8S_CNI_CONFIG": "O|text64|CNI custom config (default none)||",
    "ONEAPP_K8S_CILIUM_RANGE": "O|text|Cilium LB IP CIDR (default none)||",
    "ONEAPP_K8S_METALLB_ENABLED": "O|boolean|Enable MetalLB||NO",
    "ONEAPP_K8S_METALLB_RANGE": "O|text|MetalLB IP range (default none)||",
    "ONEAPP_K8S_METALLB_CONFIG": "O|text64|MetalLB custom config (default none)||",
    "ONEAPP_K8S_LONGHORN_ENABLED": "O|boolean|Enable Longhorn||NO",
    "ONEAPP_STORAGE_DEVICE": "O|text|Storage device path||/dev/vdb",
    "ONEAPP_STORAGE_FILESYSTEM": "O|text|Storage device filesystem||xfs",
    "ONEAPP_K8S_TRAEFIK_ENABLED": "O|boolean|Enable Traefik||NO",
    "ONEAPP_VNF_HAPROXY_INTERFACES": "O|text|Interfaces to run Haproxy on||eth0",
    "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "O|number|Haproxy refresh rate||30",
    "ONEAPP_VNF_HAPROXY_LB0_PORT": "O|number|HTTPS RKE2 Supervisor port||9345",
    "ONEAPP_VNF_HAPROXY_LB1_PORT": "O|number|HTTPS API Server port||6443",
    "ONEAPP_VNF_HAPROXY_LB2_PORT": "O|number|HTTPS ingress port||443",
    "ONEAPP_VNF_HAPROXY_LB3_PORT": "O|number|HTTP ingress port||80",
    "ONEAPP_VNF_DNS_ENABLED": "O|boolean|Enable DNS recursor||YES",
    "ONEAPP_VNF_DNS_INTERFACES": "O|text|DNS - Interfaces||eth1",
    "ONEAPP_VNF_DNS_NAMESERVERS": "O|text|DNS - Nameservers||1.1.1.1,8.8.8.8",
    "ONEAPP_VNF_NAT4_ENABLED": "O|boolean|Enable NAT||YES",
    "ONEAPP_VNF_NAT4_INTERFACES_OUT": "O|text|NAT - Outgoing Interfaces||eth0",
    "ONEAPP_VNF_ROUTER4_ENABLED": "O|boolean|Enable Router||YES",
    "ONEAPP_VNF_ROUTER4_INTERFACES": "O|text|Router - Interfaces||eth0,eth1"
  },
  "ready_status_gate": true
}

Note

We removed the registration_time key from the document as it is immutable.

Next, let's update the template:

$ oneflow-template update 'Service OneKE 1.27' /tmp/OneKE-update.json

Deploying with Terraform

Note

You can learn more about the OpenNebula Terraform Provider here.

An example HCL2 single-file config to deploy a basic non-Airgapped / non-HA OneKE instance could look like this:

terraform {
  required_providers {
    opennebula = {
      source = "OpenNebula/opennebula"
      version = ">= 1.4.0"
    }
  }
}

variable "one" {
  type = object({
    host = string
    pswd = string
  })
}

data "http" "appliances" {
  url = "https://marketplace.opennebula.io/appliance"
  request_headers = {
    User-Agent = "OpenNebula 6.8.0"
    Accept     = "application/json"
  }
}

locals {
  appliances = {
    for a in jsondecode(data.http.appliances.response_body).appliances : a.name => a
  }

  name = "Service OneKE 1.27"

  service = local.appliances[local.name]

  roles = {
    for k, v in local.service.roles : k => merge(local.appliances[v], {
      opennebula_template = jsondecode(local.appliances[v].opennebula_template)
    })
  }

  md5_to_url = {
    for f in distinct(flatten([
      for r in values(local.roles) : concat(
        try(r.files, []),
        [for d in try(r.disks, []) : local.appliances[d].files]
      )
    ])) : f.md5 => f.url
  }

  role_to_md5 = {
    for k, v in local.roles : k => [
      for f in flatten(concat(
        try(v.files, []),
        [for d in try(v.disks, []) : local.appliances[d].files]
      )) : f.md5
    ]
  }
}

provider "opennebula" {
  endpoint      = "http://${var.one.host}:2633/RPC2"
  flow_endpoint = "http://${var.one.host}:2474"
  username      = "oneadmin"
  password      = var.one.pswd
}

resource "opennebula_image" "oneke" {
  for_each     = local.md5_to_url
  name         = "${local.name} ${each.key}"
  datastore_id = 1
  persistent   = false
  permissions  = 642
  dev_prefix   = "vd"
  driver       = "qcow2"
  path         = each.value
}

resource "opennebula_template" "oneke" {
  for_each = local.roles
  name     = "${local.name} ${each.key}"
  cpu      = try(each.value["opennebula_template"].CPU, null)
  vcpu     = try(each.value["opennebula_template"].VCPU, null)
  memory   = try(each.value["opennebula_template"].MEMORY, null)
  context  = try(each.value["opennebula_template"].CONTEXT, null)

  dynamic "graphics" {
    for_each = try([each.value["opennebula_template"].GRAPHICS], [])
    content {
      type   = try(graphics.value.TYPE, null)
      listen = try(graphics.value.LISTEN, null)
    }
  }

  dynamic "os" {
    for_each = try([each.value["opennebula_template"].OS], [])
    content {
      arch = try(os.value.ARCH, null)
      boot = "disk0"
    }
  }

  dynamic "disk" {
    for_each = local.role_to_md5[each.key]
    content {
      image_id = opennebula_image.oneke[disk.value].id
    }
  }
}

resource "opennebula_service_template" "oneke" {
  name        = local.name
  permissions = 642
  uname       = "oneadmin"
  gname       = "oneadmin"
  template = jsonencode({ "TEMPLATE" = { "BODY" = merge(
    jsondecode(local.service["opennebula_template"]),
    {
      "roles" : [
        for r in jsondecode(local.service["opennebula_template"]).roles : merge(
          r,
          { vm_template = tonumber(opennebula_template.oneke[r.name].id) }
        )
      ]
    }
  ) } })
}

resource "opennebula_service" "oneke" {
  name        = local.name
  template_id = opennebula_service_template.oneke.id
  extra_template = jsonencode({
    networks_values = [
      { Public = { id = "0" } },
      { Private = { id = "1" } },
    ]
    custom_attrs_values = {
      ONEAPP_VROUTER_ETH0_VIP0 = ""
      ONEAPP_VROUTER_ETH1_VIP0 = ""

      ONEAPP_RKE2_SUPERVISOR_EP   = "ep0.eth0.vr:9345"
      ONEAPP_K8S_CONTROL_PLANE_EP = "ep0.eth0.vr:6443"
      ONEAPP_K8S_EXTRA_SANS       = "localhost,127.0.0.1,ep0.eth0.vr,$${vnf.TEMPLATE.CONTEXT.ETH0_IP}"

      ONEAPP_K8S_MULTUS_ENABLED = "NO"
      ONEAPP_K8S_MULTUS_CONFIG  = ""

      ONEAPP_K8S_CNI_PLUGIN   = "cilium"
      ONEAPP_K8S_CNI_CONFIG   = ""
      ONEAPP_K8S_CILIUM_RANGE = ""

      ONEAPP_K8S_METALLB_ENABLED = "NO"
      ONEAPP_K8S_METALLB_CONFIG  = ""
      ONEAPP_K8S_METALLB_RANGE   = ""

      ONEAPP_K8S_LONGHORN_ENABLED = "YES"
      ONEAPP_STORAGE_DEVICE       = "/dev/vdb"
      ONEAPP_STORAGE_FILESYSTEM   = "xfs"

      ONEAPP_K8S_TRAEFIK_ENABLED      = "YES"
      ONEAPP_VNF_HAPROXY_INTERFACES   = "eth0"
      ONEAPP_VNF_HAPROXY_REFRESH_RATE = "30"
      ONEAPP_VNF_HAPROXY_LB0_PORT     = "9345"
      ONEAPP_VNF_HAPROXY_LB1_PORT     = "6443"
      ONEAPP_VNF_HAPROXY_LB2_PORT     = "443"
      ONEAPP_VNF_HAPROXY_LB3_PORT     = "80"

      ONEAPP_VNF_DNS_ENABLED         = "YES"
      ONEAPP_VNF_DNS_INTERFACES      = "eth1"
      ONEAPP_VNF_DNS_NAMESERVERS     = "1.1.1.1,8.8.8.8"
      ONEAPP_VNF_NAT4_ENABLED        = "YES"
      ONEAPP_VNF_NAT4_INTERFACES_OUT = "eth0"
      ONEAPP_VNF_ROUTER4_ENABLED     = "YES"
      ONEAPP_VNF_ROUTER4_INTERFACES  = "eth0,eth1"
    }
  })
  timeouts {
    create = "10m"
    delete = "5m"
  }
}

Important

The script above takes care of downloading templates and images, no preliminary oneflow export operation is required.

Deployment Customization

It is possible to modify VM templates related to the OneKE Virtual Appliance in order to customize the deployment, for example by adding more VM memory, VCPU cores to the workers, and resizing the Disk for the storage nodes. This should be done before the creation of the K8s cluster, i.e. before instantiating the OneKE OneFlow Service Template.

When instantiating OneKE's OneFlow Service Template, you can further customize the deployment using the following custom attributes:

Parameter	Default	Roles	Description
`ONEAPP_VROUTER_ETH0_VIP0`		all	Control Plane Endpoint VIP (IPv4)
`ONEAPP_VROUTER_ETH1_VIP0`		all	Default Gateway VIP (IPv4)
`ONEAPP_RKE2_SUPERVISOR_EP`	`ep0.eth0.vr:9345`	master	RKE2 Supervisor endpoint
`ONEAPP_K8S_CONTROL_PLANE_EP`	`ep0.eth0.vr:6443`	!vnf	Control Plane endpoint
`ONEAPP_K8S_EXTRA_SANS`	`localhost,127.0.0.1,ep0.eth0.vr, ${vnf.TEMPLATE.CONTEXT.ETH0_IP}`	master	ApiServer extra certificate SANs
`ONEAPP_K8S_MULTUS_ENABLED`	`NO`	all	Enable Multus
`ONEAPP_K8S_MULTUS_CONFIG`		master	Multus custom config
`ONEAPP_K8S_CNI_PLUGIN`	`cilium`	all	CNI plugin (canal, calico or cilium)
`ONEAPP_K8S_CNI_CONFIG`		master	CNI custom config
`ONEAPP_K8S_CILIUM_RANGE`		master	Cilium LB IP/CIDR
`ONEAPP_K8S_METALLB_ENABLED`	`NO`	master,worker	Enable MetalLB
`ONEAPP_K8S_METALLB_RANGE`		master	MetalLB IP range
`ONEAPP_K8S_METALLB_CONFIG`		master	MetalLB custom config
`ONEAPP_K8S_LONGHORN_ENABLED`	`NO`	all	Enable Longhorn
`ONEAPP_STORAGE_DEVICE`	`/dev/vdb`	storage	Dedicated storage device for Longhorn
`ONEAPP_STORAGE_FILESYSTEM`	`xfs`	storage	Filesystem type to init dedicated storage device
`ONEAPP_K8S_TRAEFIK_ENABLED`	`NO`	master,worker	Enable Traefik
`ONEAPP_VNF_HAPROXY_INTERFACES`	`eth0`	vnf	Interfaces to run HAProxy on
`ONEAPP_VNF_HAPROXY_REFRESH_RATE`	`30`	vnf	HAProxy / OneGate refresh rate
`ONEAPP_VNF_HAPROXY_LB0_PORT`	`9345`	vnf,master	HTTPS RKE2 Supervisor port
`ONEAPP_VNF_HAPROXY_LB1_PORT`	`6443`	vnf,master	HTTPS API Server port
`ONEAPP_VNF_HAPROXY_LB2_PORT`	`443`	vnf,worker	HTTPS ingress port
`ONEAPP_VNF_HAPROXY_LB3_PORT`	`80`	vnf,worker	HTTP ingress port
`ONEAPP_VNF_DNS_ENABLED`	`YES`	all	Enable DNS recursor
`ONEAPP_VNF_DNS_INTERFACES`	`eth1`	vnf	DNS - Interfaces
`ONEAPP_VNF_DNS_NAMESERVERS`	`1.1.1.1,8.8.8.8`	vnf	DNS - Nameservers
`ONEAPP_VNF_NAT4_ENABLED`	`YES`	vnf	Enable NAT for the whole cluster
`ONEAPP_VNF_NAT4_INTERFACES_OUT`	`eth0`	vnf	NAT - Outgoing (public) interfaces
`ONEAPP_VNF_ROUTER4_ENABLED`	`YES`	vnf	Enable IPv4 forwarding for selected NICs
`ONEAPP_VNF_ROUTER4_INTERFACES`	`eth0,eth1`	vnf	IPv4 Router - NICs selected for IPv4 forwarding

Important

ONEAPP_VROUTER_ETH0_VIP0 - VNF cluster uses this VIP to bind and expose Kubernetes API port 6443 and RKE2's management port 9345. The eth0 NIC should be connected to the public subnet (Routed or NATed). You can leave this parameter empty, then OneKE will assume it's a non-HA setup and it will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

Important

ONEAPP_VROUTER_ETH1_VIP0 - VNF cluster uses this VIP to act as a NAT gateway for every other VM deployed inside the private subnet. The eth1 NIC should be connected to the private subnet. You can leave this parameter empty, then OneKE will assume it's a non-HA setup and it will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

Important

If you enable Longhorn (via ONEAPP_K8S_LONGHORN_ENABLED), then you also need to scale up the storage role or modify the cardinality parameter in the service template.

Home

OpenNebula Apps Overview
OS Appliances Update Policy
OneApps Quick Intro
Build Instructions
Linux Contextualization Packages
Windows Contextualization Packages
OneKE (OpenNebula Kubernetes Edition)
Virtual Router
- Release Notes
- Overview
- Quick Start
- OpenRC Services
- Virtual Router Modules
  - Router4
  - NAT4
  - SDNAT4
  - Load Balancing
  - DNS
  - DHCP4
  - Keepalived: Failover
  - Wireguard VPN
- Glossary
WordPress
Harbor Container Registry
MinIO
Development
- Virtual Router

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

oneke_deploy

Deploying OneKE

High Availability OneKE

Anti-affinity

Deploying with Terraform

Deployment Customization

Clone this wiki locally