Skip to content

oneke_deploy

Jump to bottom
Ruben S. Montero edited this page Feb 21, 2024 · 13 revisions

Deploying OneKE

In this section we focus on some advanced topics when deploying your Kubernetes clusters:

  • High Availability (HA) deployments
  • Using affinity rules in HA deployments
  • Using Terraform to create your OneKE Kubernetes clusters
  • Customize the Kubernetes components deployed with the cluster

For those seeking a quick start, we recommend referring to the Quick Start guide here, which provides step-by-step instructions to get up and running with OneKE.

High Availability OneKE

Important

For the highly-available setup both VIPs need to be provided.

By default, OneKE Virtual Appliance is preconfigured to work as a non-Highly-Available K8s cluster, since OneFlow Service Templates deploys each service role as a single VM. Kubernetes High-Availability is about setting up a Kubernetes cluster, along with its components, in such a way that there is no single point of failure. To achieve high-availability, the following OneKE components should be scaled up: VNF (at least 2 VMs), master (at least 3 VMs) and storage (at least 2 VMs), see figure below:

Public Network
─────────────────────────────────────────┬───────────────────────────────────────────────────────────
                                         │ VIP0 / FLOATING_IP (3.27.6.5) (ep0.eth0.vr)
                       ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ┐
                       │ OneKE vnf       │     │
                       │ ┌──eth0──┐ ┌──eth0──┐ │
                       │ │        │ │        │ │
                       │ │  VM-1  │ │  VM-2  │ │
                       │ │        │ │        │ │
                       │ └──eth1──┘ └──eth1──┘ │
                       │     │           │     │
                       │  10.0.0.2    10.0.0.3 │
                       │                 │     │
                       └ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ┘
                                         │ VIP1 / FLOATING_IP (10.0.0.1) (ep0.eth1.vr)
───────┬───────────┬─────────┬───────────┴──┬───────────┬─────────────┬───────────┬─────────┬────────
Private Network    │         │              │           │             │           │         │
       │           │         │              │           │             │           │         │
 ┌ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ┼ ─ ─ ─┐ ┌ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ┐ ┌ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ┼ ─ ─ ─┐
 │ OneKE master    │         │      │ │ OneKE worker    │     │ │ OneKE storage   │         │      │
 │     │           │         │      │ │     │           │     │ │     │           │         │      │
 │  10.0.0.4    10.0.0.5   10.0.0.6 │ │  10.0.0.7    10.0.0.8 │ │  10.0.0.9  10.0.0.10   10.0.0.11 │
 │     │           │         │      │ │     │           │     │ │     │           │         │      │
 │ ┌──eth0──┐ ┌──eth0──┐ ┌──eth0──┐ │ │ ┌──eth0──┐ ┌──eth0──┐ │ │ ┌──eth0──┐ ┌──eth0──┐ ┌──eth0──┐ │
 │ │        │ │        │ │        │ │ │ │        │ │        │ │ │ │        │ │        │ │        │ │
 │ │  VM-3  │ │  VM-4  │ │  VM-5  │ │ │ │  VM-6  │ │  VM-7  │ │ │ │  VM-8  │ │  VM-9  │ │  VM-10 │ │
 │ │        │ │        │ │        │ │ │ │        │ │        │ │ │ │        │ │        │ │        │ │
 │ └────────┘ └────────┘ └────────┘ │ │ └────────┘ └────────┘ │ │ └────────┘ └────────┘ └────────┘ │
 └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘

  1. Download the Service OneKE 1.27 (or the airgapped version Service OneKE 1.27a):

    $ onemarketapp export 'Service OneKE 1.27' 'Service OneKE 1.27' --datastore 1
IMAGE
    ID: 202
    ID: 203
    ID: 204
VMTEMPLATE
    ID: 204
    ID: 205
    ID: 206
SERVICE_TEMPLATE
    ID: 104

  2. Adjust Flow and VM templates to meet you needs. (You may want to increase CPU and MEMORY for workers, add HOT_RESIZE=[...] or setup anti-affined VM groups, etc.)

    $ oneflow-template update 104
    $ onetemplate update 205

  3. Instantiate the Flow service:

    $ cat >/tmp/OneKE-instantiate <<'EOF'
{
    "name": "OneKE/1",
    "networks_values": [
        {"Public": {"id": "0"}},
        {"Private": {"id": "1"}}
    ],
    "custom_attrs_values": {
        "ONEAPP_VROUTER_ETH0_VIP0": "3.27.6.5",
        "ONEAPP_VROUTER_ETH1_VIP0": "10.0.0.1",

        "ONEAPP_RKE2_SUPERVISOR_EP": "",
        "ONEAPP_K8S_CONTROL_PLANE_EP": "",
        "ONEAPP_K8S_EXTRA_SANS": "localhost,127.0.0.1,k8s.yourdomain.it",

        "ONEAPP_K8S_MULTUS_ENABLED": "NO",
        "ONEAPP_K8S_MULTUS_CONFIG": "",
        "ONEAPP_K8S_CNI_PLUGIN": "cilium",
        "ONEAPP_K8S_CNI_CONFIG": "",
        "ONEAPP_K8S_CILIUM_RANGE": "",

        "ONEAPP_K8S_METALLB_ENABLED": "NO",
        "ONEAPP_K8S_METALLB_CONFIG": "",
        "ONEAPP_K8S_METALLB_RANGE": "",

        "ONEAPP_K8S_LONGHORN_ENABLED": "YES",
        "ONEAPP_STORAGE_DEVICE": "/dev/vdb",
        "ONEAPP_STORAGE_FILESYSTEM": "xfs",

        "ONEAPP_K8S_TRAEFIK_ENABLED": "YES",
        "ONEAPP_VNF_HAPROXY_INTERFACES": "eth0",
        "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "30",
        "ONEAPP_VNF_HAPROXY_LB0_PORT": "9345",
        "ONEAPP_VNF_HAPROXY_LB1_PORT": "6443",
        "ONEAPP_VNF_HAPROXY_LB2_PORT": "443",
        "ONEAPP_VNF_HAPROXY_LB3_PORT": "80",

        "ONEAPP_VNF_DNS_ENABLED": "YES",
        "ONEAPP_VNF_DNS_INTERFACES": "eth1",
        "ONEAPP_VNF_DNS_NAMESERVERS": "1.1.1.1,8.8.8.8",
        "ONEAPP_VNF_NAT4_ENABLED": "YES",
        "ONEAPP_VNF_NAT4_INTERFACES_OUT": "eth0",
        "ONEAPP_VNF_ROUTER4_ENABLED": "YES",
        "ONEAPP_VNF_ROUTER4_INTERFACES": "eth0,eth1"
    }
}
EOF
$ oneflow-template instantiate 'Service OneKE 1.27' /tmp/OneKE-instantiate
ID: 105

    After successful deployment the Flow service should be in the RUNNING state:

    $ oneflow list
  ID USER     GROUP    NAME                           STARTTIME STAT
 105 oneadmin oneadmin Service OneKE 1.27        02/15 09:46:56 RUNNING

  4. Scale Flow roles (if required):

    $ oneflow scale 105 vnf 2
    $ oneflow scale 105 master 3
    $ oneflow scale 105 worker 2
    $ oneflow scale 105 storage 3

Note

Cardinality of the storage role is set to 0 by default to allow deployments with Longhorn disabled.

Note

You can set the cardinilaty of each role in step 2 when you update the template.

Warning

You can scale the master role up to an odd number of masters, but be careful while scaling down as it may break your cluster. If you require multi-master HA, just start with a single master and then scale up to 3 and keep it that way.

After a while we can examine the service log:

$ oneflow show 'OneKE/1'
...
LOG MESSAGES
05/11/23 18:30 [I] New state: DEPLOYING_NETS
05/11/23 18:30 [I] New state: DEPLOYING
05/11/23 18:39 [I] New state: RUNNING
05/11/23 18:43 [I] Role master scaling up from 1 to 3 nodes
05/11/23 18:43 [I] New state: SCALING
05/11/23 18:52 [I] New state: COOLDOWN
05/11/23 18:55 [I] New state: RUNNING

And afterwards we can list cluster nodes using kubectl:

$ kubectl get nodes
NAME                    STATUS   ROLES                       AGE     VERSION
oneke-ip-172-20-0-101   Ready    control-plane,etcd,master   31m     v1.27.2+rke2r1
oneke-ip-172-20-0-102   Ready    <none>                      28m     v1.27.2+rke2r1
oneke-ip-172-20-0-103   Ready    <none>                      28m     v1.27.2+rke2r1
oneke-ip-172-20-0-104   Ready    control-plane,etcd,master   11m     v1.27.2+rke2r1
oneke-ip-172-20-0-105   Ready    control-plane,etcd,master   10m     v1.27.2+rke2r1

Warning

Please plan ahead and avoid scaling down master and storage roles as it may break ETCD's quorum or cause data loss. There is no obvious restriction for the worker role, however. It can be safely rescaled at will.

Anti-affinity

VMs related to the same role should be scheduled on different physical hosts in an HA setup to guarantee HA in case of a host failure. OpenNebula provides VM Group resources to achieve proper Host/VM affinity/anti-affinity.

In the following section, we provide an example of how to create VM Group resources and how to modify OneKE's OneFlow Service Template to include VM groups.

Let's assume that epsilon and omicron are hosts we want to use to deploy OneKE; a VM Group may be created in the following way:

$ cat >/tmp/OneKE-vmgroup <<'EOF'
NAME = "Service OneKE 1.27"
ROLE = [
    NAME         = "vnf",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "master",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "worker",
    HOST_AFFINED = "epsilon,omicron"
]
ROLE = [
    NAME         = "storage",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
EOF

$ onevmgroup create /tmp/OneKE-vmgroup
ID: 1

Important

The worker role does not have POLICY defined, this allows you to reuse hosts multiple times!

Now, let's modify the OneKE OneFlow Service Template:

$ oneflow-template show 'Service OneKE 1.27' --json | >/tmp/OneKE-update.json jq -r --arg vmgroup 'Service OneKE 1.27' -f /dev/fd/3 3<<'EOF'
.DOCUMENT.TEMPLATE.BODY | del(.registration_time) | . += {
  roles: .roles | map(
    .vm_template_contents = "VMGROUP=[VMGROUP_NAME=\"\($vmgroup)\",ROLE=\"\(.name)\"]\n" + .vm_template_contents
  )
}
EOF

Content of the update (/tmp/OneKE-update.json) will look like this:

{
  "name": "Service OneKE 1.27",
  "deployment": "straight",
  "description": "",
  "roles": [
    {
      "name": "vnf",
      "cardinality": 1,
      "min_vms": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"vnf\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Public\" ]\nNIC = [\n  NAME = \"NIC1\",\n  NETWORK_ID = \"$Private\" ]\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_VNF_HAPROXY_INTERFACES = \"$ONEAPP_VNF_HAPROXY_INTERFACES\"\nONEAPP_VNF_HAPROXY_REFRESH_RATE = \"$ONEAPP_VNF_HAPROXY_REFRESH_RATE\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\nONEAPP_VNF_DNS_INTERFACES = \"$ONEAPP_VNF_DNS_INTERFACES\"\nONEAPP_VNF_DNS_NAMESERVERS = \"$ONEAPP_VNF_DNS_NAMESERVERS\"\nONEAPP_VNF_NAT4_ENABLED = \"$ONEAPP_VNF_NAT4_ENABLED\"\nONEAPP_VNF_NAT4_INTERFACES_OUT = \"$ONEAPP_VNF_NAT4_INTERFACES_OUT\"\nONEAPP_VNF_ROUTER4_ENABLED = \"$ONEAPP_VNF_ROUTER4_ENABLED\"\nONEAPP_VNF_ROUTER4_INTERFACES = \"$ONEAPP_VNF_ROUTER4_INTERFACES\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 29
    },
    {
      "name": "master",
      "parents": [
        "vnf"
      ],
      "cardinality": 1,
      "min_vms": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"master\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_EXTRA_SANS = \"$ONEAPP_K8S_EXTRA_SANS\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_MULTUS_CONFIG = \"$ONEAPP_K8S_MULTUS_CONFIG\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_CNI_CONFIG = \"$ONEAPP_K8S_CNI_CONFIG\"\nONEAPP_K8S_CILIUM_RANGE = \"$ONEAPP_K8S_CILIUM_RANGE\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_METALLB_CONFIG = \"$ONEAPP_K8S_METALLB_CONFIG\"\nONEAPP_K8S_METALLB_RANGE = \"$ONEAPP_K8S_METALLB_RANGE\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 30
    },
    {
      "name": "worker",
      "parents": [
        "vnf"
      ],
      "cardinality": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"worker\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 30
    },
    {
      "name": "storage",
      "parents": [
        "vnf"
      ],
      "cardinality": 0,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"storage\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_STORAGE_DEVICE = \"$ONEAPP_STORAGE_DEVICE\"\nONEAPP_STORAGE_FILESYSTEM = \"$ONEAPP_STORAGE_FILESYSTEM\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 31
    }
  ],
  "networks": {
    "Public": "M|network|Public||id:",
    "Private": "M|network|Private||id:"
  },
  "custom_attrs": {
    "ONEAPP_VROUTER_ETH0_VIP0": "O|text|Control Plane Endpoint VIP (IPv4)||",
    "ONEAPP_VROUTER_ETH1_VIP0": "O|text|Default Gateway VIP (IPv4)||",
    "ONEAPP_RKE2_SUPERVISOR_EP": "O|text|RKE2 Supervisor endpoint||ep0.eth0.vr:9345",
    "ONEAPP_K8S_CONTROL_PLANE_EP": "O|text|Control Plane endpoint||ep0.eth0.vr:6443",
    "ONEAPP_K8S_EXTRA_SANS": "O|text|ApiServer extra certificate SANs||localhost,127.0.0.1,ep0.eth0.vr,${vnf.TEMPLATE.CONTEXT.ETH0_IP}",
    "ONEAPP_K8S_MULTUS_ENABLED": "O|boolean|Enable Multus||NO",
    "ONEAPP_K8S_MULTUS_CONFIG": "O|text64|Multus custom config (default none)||",
    "ONEAPP_K8S_CNI_PLUGIN": "O|list|CNI plugin supported by RKE2|canal,calico,cilium|cilium",
    "ONEAPP_K8S_CNI_CONFIG": "O|text64|CNI custom config (default none)||",
    "ONEAPP_K8S_CILIUM_RANGE": "O|text|Cilium LB IP CIDR (default none)||",
    "ONEAPP_K8S_METALLB_ENABLED": "O|boolean|Enable MetalLB||NO",
    "ONEAPP_K8S_METALLB_RANGE": "O|text|MetalLB IP range (default none)||",
    "ONEAPP_K8S_METALLB_CONFIG": "O|text64|MetalLB custom config (default none)||",
    "ONEAPP_K8S_LONGHORN_ENABLED": "O|boolean|Enable Longhorn||NO",
    "ONEAPP_STORAGE_DEVICE": "O|text|Storage device path||/dev/vdb",
    "ONEAPP_STORAGE_FILESYSTEM": "O|text|Storage device filesystem||xfs",
    "ONEAPP_K8S_TRAEFIK_ENABLED": "O|boolean|Enable Traefik||NO",
    "ONEAPP_VNF_HAPROXY_INTERFACES": "O|text|Interfaces to run Haproxy on||eth0",
    "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "O|number|Haproxy refresh rate||30",
    "ONEAPP_VNF_HAPROXY_LB0_PORT": "O|number|HTTPS RKE2 Supervisor port||9345",
    "ONEAPP_VNF_HAPROXY_LB1_PORT": "O|number|HTTPS API Server port||6443",
    "ONEAPP_VNF_HAPROXY_LB2_PORT": "O|number|HTTPS ingress port||443",
    "ONEAPP_VNF_HAPROXY_LB3_PORT": "O|number|HTTP ingress port||80",
    "ONEAPP_VNF_DNS_ENABLED": "O|boolean|Enable DNS recursor||YES",
    "ONEAPP_VNF_DNS_INTERFACES": "O|text|DNS - Interfaces||eth1",
    "ONEAPP_VNF_DNS_NAMESERVERS": "O|text|DNS - Nameservers||1.1.1.1,8.8.8.8",
    "ONEAPP_VNF_NAT4_ENABLED": "O|boolean|Enable NAT||YES",
    "ONEAPP_VNF_NAT4_INTERFACES_OUT": "O|text|NAT - Outgoing Interfaces||eth0",
    "ONEAPP_VNF_ROUTER4_ENABLED": "O|boolean|Enable Router||YES",
    "ONEAPP_VNF_ROUTER4_INTERFACES": "O|text|Router - Interfaces||eth0,eth1"
  },
  "ready_status_gate": true
}

Note

We removed the registration_time key from the document as it is immutable.

Next, let's update the template:

$ oneflow-template update 'Service OneKE 1.27' /tmp/OneKE-update.json

Deploying with Terraform

An example HCL2 file could look like this:

terraform {
  required_providers {
    opennebula = {
      source = "OpenNebula/opennebula"
      version = ">= 1.4.0"
    }
  }
}

variable "one" {
  type = object({
    host = string
    pswd = string
  })
}

data "http" "appliances" {
  url = "https://marketplace.opennebula.io/appliance"
  request_headers = {
    User-Agent = "OpenNebula 6.8.0"
    Accept     = "application/json"
  }
}

locals {
  appliances = {
    for a in jsondecode(data.http.appliances.response_body).appliances : a.name => a
  }

  name = "Service OneKE 1.27"

  service = local.appliances[local.name]

  roles = {
    for k, v in local.service.roles : k => merge(local.appliances[v], {
      opennebula_template = jsondecode(local.appliances[v].opennebula_template)
    })
  }

  md5_to_url = {
    for f in distinct(flatten([
      for r in values(local.roles) : concat(
        try(r.files, []),
        [for d in try(r.disks, []) : local.appliances[d].files]
      )
    ])) : f.md5 => f.url
  }

  role_to_md5 = {
    for k, v in local.roles : k => [
      for f in flatten(concat(
        try(v.files, []),
        [for d in try(v.disks, []) : local.appliances[d].files]
      )) : f.md5
    ]
  }
}

provider "opennebula" {
  endpoint      = "http://${var.one.host}:2633/RPC2"
  flow_endpoint = "http://${var.one.host}:2474"
  username      = "oneadmin"
  password      = var.one.pswd
}

resource "opennebula_image" "oneke" {
  for_each     = local.md5_to_url
  name         = "${local.name} ${each.key}"
  datastore_id = 1
  persistent   = false
  permissions  = 642
  dev_prefix   = "vd"
  driver       = "qcow2"
  path         = each.value
}

resource "opennebula_template" "oneke" {
  for_each = local.roles
  name     = "${local.name} ${each.key}"
  cpu      = try(each.value["opennebula_template"].CPU, null)
  vcpu     = try(each.value["opennebula_template"].VCPU, null)
  memory   = try(each.value["opennebula_template"].MEMORY, null)
  context  = try(each.value["opennebula_template"].CONTEXT, null)

  dynamic "graphics" {
    for_each = try([each.value["opennebula_template"].GRAPHICS], [])
    content {
      type   = try(graphics.value.TYPE, null)
      listen = try(graphics.value.LISTEN, null)
    }
  }

  dynamic "os" {
    for_each = try([each.value["opennebula_template"].OS], [])
    content {
      arch = try(os.value.ARCH, null)
      boot = "disk0"
    }
  }

  dynamic "disk" {
    for_each = local.role_to_md5[each.key]
    content {
      image_id = opennebula_image.oneke[disk.value].id
    }
  }
}

resource "opennebula_service_template" "oneke" {
  name        = local.name
  permissions = 642
  uname       = "oneadmin"
  gname       = "oneadmin"
  template = jsonencode({ "TEMPLATE" = { "BODY" = merge(
    jsondecode(local.service["opennebula_template"]),
    {
      "roles" : [
        for r in jsondecode(local.service["opennebula_template"]).roles : merge(
          r,
          { vm_template = tonumber(opennebula_template.oneke[r.name].id) }
        )
      ]
    }
  ) } })
}

resource "opennebula_service" "oneke" {
  name        = local.name
  template_id = opennebula_service_template.oneke.id
  extra_template = jsonencode({
    networks_values = [
      { Public = { id = "0" } },
      { Private = { id = "1" } },
    ]
    custom_attrs_values = {
      ONEAPP_VROUTER_ETH0_VIP0 = ""
      ONEAPP_VROUTER_ETH1_VIP0 = ""

      ONEAPP_RKE2_SUPERVISOR_EP   = "ep0.eth0.vr:9345"
      ONEAPP_K8S_CONTROL_PLANE_EP = "ep0.eth0.vr:6443"
      ONEAPP_K8S_EXTRA_SANS       = "localhost,127.0.0.1,ep0.eth0.vr,$${vnf.TEMPLATE.CONTEXT.ETH0_IP}"

      ONEAPP_K8S_MULTUS_ENABLED = "NO"
      ONEAPP_K8S_MULTUS_CONFIG  = ""

      ONEAPP_K8S_CNI_PLUGIN   = "cilium"
      ONEAPP_K8S_CNI_CONFIG   = ""
      ONEAPP_K8S_CILIUM_RANGE = ""

      ONEAPP_K8S_METALLB_ENABLED = "NO"
      ONEAPP_K8S_METALLB_CONFIG  = ""
      ONEAPP_K8S_METALLB_RANGE   = ""

      ONEAPP_K8S_LONGHORN_ENABLED = "YES"
      ONEAPP_STORAGE_DEVICE       = "/dev/vdb"
      ONEAPP_STORAGE_FILESYSTEM   = "xfs"

      ONEAPP_K8S_TRAEFIK_ENABLED      = "YES"
      ONEAPP_VNF_HAPROXY_INTERFACES   = "eth0"
      ONEAPP_VNF_HAPROXY_REFRESH_RATE = "30"
      ONEAPP_VNF_HAPROXY_LB0_PORT     = "9345"
      ONEAPP_VNF_HAPROXY_LB1_PORT     = "6443"
      ONEAPP_VNF_HAPROXY_LB2_PORT     = "443"
      ONEAPP_VNF_HAPROXY_LB3_PORT     = "80"

      ONEAPP_VNF_DNS_ENABLED         = "YES"
      ONEAPP_VNF_DNS_INTERFACES      = "eth1"
      ONEAPP_VNF_DNS_NAMESERVERS     = "1.1.1.1,8.8.8.8"
      ONEAPP_VNF_NAT4_ENABLED        = "YES"
      ONEAPP_VNF_NAT4_INTERFACES_OUT = "eth0"
      ONEAPP_VNF_ROUTER4_ENABLED     = "YES"
      ONEAPP_VNF_ROUTER4_INTERFACES  = "eth0,eth1"
    }
  })
  timeouts {
    create = "10m"
    delete = "5m"
  }
}

Note

The script above takes care of downloading templates and images, no preliminary oneflow export operation is required.

Deployment Customization

It is possible to modify VM templates related to the OneKE Virtual Appliance in order to customize the deployment, for example by adding more VM memory, VCPU cores to the workers, and resizing the Disk for the storage nodes. This should be done before the creation of the K8s cluster, i.e. before instantiating the OneKE OneFlow Service Template.

When instantiating OneKE's OneFlow Service Template, you can further customize the deployment using the following custom attributes:

Parameter Default Roles Description
ONEAPP_VROUTER_ETH0_VIP0 all Control Plane Endpoint VIP (IPv4)
ONEAPP_VROUTER_ETH1_VIP0 all Default Gateway VIP (IPv4)
ONEAPP_RKE2_SUPERVISOR_EP ep0.eth0.vr:9345 master RKE2 Supervisor endpoint
ONEAPP_K8S_CONTROL_PLANE_EP ep0.eth0.vr:6443 !vnf Control Plane endpoint
ONEAPP_K8S_EXTRA_SANS localhost,127.0.0.1,ep0.eth0.vr, ${vnf.TEMPLATE.CONTEXT.ETH0_IP} master ApiServer extra certificate SANs
ONEAPP_K8S_MULTUS_ENABLED NO all Enable Multus
ONEAPP_K8S_MULTUS_CONFIG master Multus custom config
ONEAPP_K8S_CNI_PLUGIN cilium all CNI plugin (canal, calico or cilium)
ONEAPP_K8S_CNI_CONFIG master CNI custom config
ONEAPP_K8S_CILIUM_RANGE master Cilium LB IP/CIDR
ONEAPP_K8S_METALLB_ENABLED NO master,worker Enable MetalLB
ONEAPP_K8S_METALLB_RANGE master MetalLB IP range
ONEAPP_K8S_METALLB_CONFIG master MetalLB custom config
ONEAPP_K8S_LONGHORN_ENABLED NO all Enable Longhorn
ONEAPP_STORAGE_DEVICE /dev/vdb storage Dedicated storage device for Longhorn
ONEAPP_STORAGE_FILESYSTEM xfs storage Filesystem type to init dedicated storage device
ONEAPP_K8S_TRAEFIK_ENABLED NO master,worker Enable Traefik
ONEAPP_VNF_HAPROXY_INTERFACES eth0 vnf Interfaces to run HAProxy on
ONEAPP_VNF_HAPROXY_REFRESH_RATE 30 vnf HAProxy / OneGate refresh rate
ONEAPP_VNF_HAPROXY_LB0_PORT 9345 vnf,master HTTPS RKE2 Supervisor port
ONEAPP_VNF_HAPROXY_LB1_PORT 6443 vnf,master HTTPS API Server port
ONEAPP_VNF_HAPROXY_LB2_PORT 443 vnf,worker HTTPS ingress port
ONEAPP_VNF_HAPROXY_LB3_PORT 80 vnf,worker HTTP ingress port
ONEAPP_VNF_DNS_ENABLED YES all Enable DNS recursor
ONEAPP_VNF_DNS_INTERFACES eth1 vnf DNS - Interfaces
ONEAPP_VNF_DNS_NAMESERVERS 1.1.1.1,8.8.8.8 vnf DNS - Nameservers
ONEAPP_VNF_NAT4_ENABLED YES vnf Enable NAT for the whole cluster
ONEAPP_VNF_NAT4_INTERFACES_OUT eth0 vnf NAT - Outgoing (public) interfaces
ONEAPP_VNF_ROUTER4_ENABLED YES vnf Enable IPv4 forwarding for selected NICs
ONEAPP_VNF_ROUTER4_INTERFACES eth0,eth1 vnf IPv4 Router - NICs selected for IPv4 forwarding

Important

ONEAPP_VROUTER_ETH0_VIP0 - VNF cluster uses this VIP to bind and expose Kubernetes API port 6443 and RKE2's management port 9345. The eth0 NIC should be connected to the public subnet (Routed or NATed). You can leave this parameter empty, then OneKE will assume it's a non-HA setup and it will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

Important

ONEAPP_VROUTER_ETH1_VIP0 - VNF cluster uses this VIP to act as a NAT gateway for every other VM deployed inside the private subnet. The eth1 NIC should be connected to the private subnet. You can leave this parameter empty, then OneKE will assume it's a non-HA setup and it will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

Important

If you enable Longhorn (via ONEAPP_K8S_LONGHORN_ENABLED), then you also need to scale up the storage role or modify the cardinality parameter in the service template.

Home

Clone this wiki locally