Skip to content

oneke_deploy

Jump to bottom
Pedro Ielpi edited this page Oct 3, 2024 · 13 revisions

Deploying OneKE

In this section we'll focus on advanced topics when deploying your Kubernetes clusters:

  • High Availability (HA) deployments
  • Using affinity rules in HA deployments
  • Using Terraform to create your OneKE Kubernetes clusters
  • Customizing the Kubernetes components deployed with the cluster

For those seeking a quick start, we recommend referring to the Quick Start Guide, which provides step-by-step instructions to get up and running with OneKE. Additionally, for simple deployments using the Sunstone web UI, you can check Running Kubernetes Clusters in the OpenNebula documentation.

High Availability OneKE

Important

For high availability both Virtual IPs (VIPs) must be provided.

By default, the OneKE Virtual Appliance is preconfigured to work as a non-Highly-Available K8s cluster, since OneFlow Service Templates deploy each service role as a single VM. Kubernetes High-Availability involves setting up a Kubernetes cluster, along with its components, in such a way that there is no single point of failure. To achieve high-availability, the following OneKE components should be scaled up: VNF (at least 2 VMs), master (at least 3 VMs) and storage (at least 2 VMs). See figure below:

Public Network
─────────────────────────────────────────┬───────────────────────────────────────────────────────────
                                         │ VIP0 / FLOATING_IP (3.27.6.5) (ep0.eth0.vr)
                       ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ┐
                       │ OneKE vnf       │     │
                       │ ┌──eth0──┐ ┌──eth0──┐ │
                       │ │        │ │        │ │
                       │ │  VM-1  │ │  VM-2  │ │
                       │ │        │ │        │ │
                       │ └──eth1──┘ └──eth1──┘ │
                       │     │           │     │
                       │  10.0.0.2    10.0.0.3 │
                       │                 │     │
                       └ ─ ─ ─ ─ ─ ─ ─ ─ ┼ ─ ─ ┘
                                         │ VIP1 / FLOATING_IP (10.0.0.1) (ep0.eth1.vr)
───────┬───────────┬─────────┬───────────┴──┬───────────┬─────────────┬───────────┬─────────┬────────
Private Network    │         │              │           │             │           │         │
       │           │         │              │           │             │           │         │
 ┌ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ┼ ─ ─ ─┐ ┌ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ┐ ┌ ─ ─ ┼ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ┼ ─ ─ ─┐
 │ OneKE master    │         │      │ │ OneKE worker    │     │ │ OneKE storage   │         │      │
 │     │           │         │      │ │     │           │     │ │     │           │         │      │
 │  10.0.0.4    10.0.0.5   10.0.0.6 │ │  10.0.0.7    10.0.0.8 │ │  10.0.0.9  10.0.0.10   10.0.0.11 │
 │     │           │         │      │ │     │           │     │ │     │           │         │      │
 │ ┌──eth0──┐ ┌──eth0──┐ ┌──eth0──┐ │ │ ┌──eth0──┐ ┌──eth0──┐ │ │ ┌──eth0──┐ ┌──eth0──┐ ┌──eth0──┐ │
 │ │        │ │        │ │        │ │ │ │        │ │        │ │ │ │        │ │        │ │        │ │
 │ │  VM-3  │ │  VM-4  │ │  VM-5  │ │ │ │  VM-6  │ │  VM-7  │ │ │ │  VM-8  │ │  VM-9  │ │  VM-10 │ │
 │ │        │ │        │ │        │ │ │ │        │ │        │ │ │ │        │ │        │ │        │ │
 │ └────────┘ └────────┘ └────────┘ │ │ └────────┘ └────────┘ │ │ └────────┘ └────────┘ └────────┘ │
 └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘

Downloading and Deploying

  1. Download the Service OneKE 1.29 (or the airgapped version, Service OneKE 1.29 Airgapped) from the OpenNebula Marketplace:

    onemarketapp export 'Service OneKE 1.29' 'Service OneKE 1.29' --datastore 1

    Output should be similar to the following:

    $ onemarketapp export 'Service OneKE 1.29' 'Service OneKE 1.29' --datastore 1
IMAGE
    ID: 202
    ID: 203
    ID: 204
VMTEMPLATE
    ID: 204
    ID: 205
    ID: 206
SERVICE_TEMPLATE
    ID: 104

  2. Adjust Flow and VM templates to meet you needs. (You may want to increase CPU and MEMORY for workers, add HOT_RESIZE=[...] or setup anti-affined VM groups, etc.)

    $ oneflow-template update 104
    $ onetemplate update 205

  3. Instantiate the Flow service:

    $ cat >/tmp/OneKE-instantiate <<'EOF'
{
    "name": "OneKE/1",
    "networks_values": [
        {"Public": {"id": "0"}},
        {"Private": {"id": "1"}}
    ],
    "custom_attrs_values": {
        "ONEAPP_VROUTER_ETH0_VIP0": "3.27.6.5",
        "ONEAPP_VROUTER_ETH1_VIP0": "10.0.0.1",

        "ONEAPP_RKE2_SUPERVISOR_EP": "",
        "ONEAPP_K8S_CONTROL_PLANE_EP": "",
        "ONEAPP_K8S_EXTRA_SANS": "localhost,127.0.0.1,k8s.yourdomain.it",

        "ONEAPP_K8S_MULTUS_ENABLED": "NO",
        "ONEAPP_K8S_MULTUS_CONFIG": "",
        "ONEAPP_K8S_CNI_PLUGIN": "cilium",
        "ONEAPP_K8S_CNI_CONFIG": "",
        "ONEAPP_K8S_CILIUM_RANGE": "",

        "ONEAPP_K8S_METALLB_ENABLED": "NO",
        "ONEAPP_K8S_METALLB_CONFIG": "",
        "ONEAPP_K8S_METALLB_RANGE": "",

        "ONEAPP_K8S_LONGHORN_ENABLED": "YES",
        "ONEAPP_STORAGE_DEVICE": "/dev/vdb",
        "ONEAPP_STORAGE_FILESYSTEM": "xfs",

        "ONEAPP_K8S_TRAEFIK_ENABLED": "YES",
        "ONEAPP_VNF_HAPROXY_INTERFACES": "eth0",
        "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "30",
        "ONEAPP_VNF_HAPROXY_LB0_PORT": "9345",
        "ONEAPP_VNF_HAPROXY_LB1_PORT": "6443",
        "ONEAPP_VNF_HAPROXY_LB2_PORT": "443",
        "ONEAPP_VNF_HAPROXY_LB3_PORT": "80",

        "ONEAPP_VNF_DNS_ENABLED": "YES",
        "ONEAPP_VNF_DNS_INTERFACES": "eth1",
        "ONEAPP_VNF_DNS_NAMESERVERS": "1.1.1.1,8.8.8.8",
        "ONEAPP_VNF_NAT4_ENABLED": "YES",
        "ONEAPP_VNF_NAT4_INTERFACES_OUT": "eth0",
        "ONEAPP_VNF_ROUTER4_ENABLED": "YES",
        "ONEAPP_VNF_ROUTER4_INTERFACES": "eth0,eth1"
    }
}
EOF
$ oneflow-template instantiate 'Service OneKE 1.29' /tmp/OneKE-instantiate
ID: 105

    After successful deployment the Flow service should display RUNNING state:

    $ oneflow list
  ID USER     GROUP    NAME                           STARTTIME STAT
 105 oneadmin oneadmin Service OneKE 1.27        02/15 09:46:56 RUNNING

  4. Scale Flow roles (if required):

    $ oneflow scale 105 vnf 2
    $ oneflow scale 105 master 3
    $ oneflow scale 105 worker 2
    $ oneflow scale 105 storage 3

Note

By default, the cardinality of the storage role is set to 0 to allow deployments with Longhorn disabled.

Note

You can set the cardinality of each role in step 2 when you update the template.

Warning

You can scale the master role up to an odd number of masters, but be careful if scaling down as it may break your cluster. If you require multi-master HA, start with a single master, then scale up to 3 and keep it that way.

After a while we can examine the service log:

$ oneflow show 'OneKE/1'
...
LOG MESSAGES
05/13/24 12:30 [I] New state: DEPLOYING_NETS
05/13/24 12:30 [I] New state: DEPLOYING
05/13/24 12:39 [I] New state: RUNNING
05/13/24 12:43 [I] Role master scaling up from 1 to 3 nodes
05/13/24 12:43 [I] New state: SCALING
05/13/24 12:52 [I] New state: COOLDOWN
05/13/24 12:55 [I] New state: RUNNING

And we can list the cluster nodes using kubectl:

$ kubectl get nodes
NAME                    STATUS   ROLES                       AGE     VERSION
oneke-ip-172-20-0-101   Ready    control-plane,etcd,master   31m     v1.29.4+rke2r1
oneke-ip-172-20-0-102   Ready    <none>                      28m     v1.29.4+rke2r1
oneke-ip-172-20-0-103   Ready    <none>                      28m     v1.29.4+rke2r1
oneke-ip-172-20-0-104   Ready    control-plane,etcd,master   11m     v1.29.4+rke2r1
oneke-ip-172-20-0-105   Ready    control-plane,etcd,master   10m     v1.29.4+rke2r1

Warning

Please plan ahead and avoid scaling down master and storage roles as it may break ETCD's quorum or cause data loss. There is no obvious restriction for the worker role, however; it can be safely rescaled at will.

Anti-affinity

In an HA configuration, VMs related to the same role should be scheduled on different physical hosts to guarantee HA in case of a host failure. OpenNebula provides VM Group resources to achieve proper Host/VM affinity/anti-affinity.

In the following section, we provide an example of how to create VM Group resources and how to modify OneKE's OneFlow Service Template to include VM groups.

Let's assume that epsilon and omicron are hosts we want to use to deploy OneKE; a VM Group may be created as shown below:

$ cat >/tmp/OneKE-vmgroup <<'EOF'
NAME = "Service OneKE 1.29"
ROLE = [
    NAME         = "vnf",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "master",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "worker",
    HOST_AFFINED = "epsilon,omicron"
]
ROLE = [
    NAME         = "storage",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
EOF

$ onevmgroup create /tmp/OneKE-vmgroup
ID: 1

Important

For the worker role, POLICY is not defined, which allows you to reuse hosts multiple times.

Now we can modify the OneKE OneFlow Service Template. In the example below we'll pipe the template in JSON format and modify it with the jq command-line JSON processor.

$ oneflow-template show 'Service OneKE 1.29' --json | >/tmp/OneKE-update.json jq -r --arg vmgroup 'Service OneKE 1.29' -f /dev/fd/3 3<<'EOF'
.DOCUMENT.TEMPLATE.BODY | del(.registration_time) | . += {
  roles: .roles | map(
    .vm_template_contents = "VMGROUP=[VMGROUP_NAME=\"\($vmgroup)\",ROLE=\"\(.name)\"]\n" + .vm_template_contents
  )
}
EOF

The above command adds the template contents for the VMs in the OneKE service. The contents of the updated file, /tmp/OneKE-update.json, are shown below:

{
  "name": "Service OneKE 1.29",
  "deployment": "straight",
  "description": "",
  "roles": [
    {
      "name": "vnf",
      "cardinality": 1,
      "min_vms": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.29\",ROLE=\"vnf\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Public\" ]\nNIC = [\n  NAME = \"NIC1\",\n  NETWORK_ID = \"$Private\" ]\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_VNF_HAPROXY_INTERFACES = \"$ONEAPP_VNF_HAPROXY_INTERFACES\"\nONEAPP_VNF_HAPROXY_REFRESH_RATE = \"$ONEAPP_VNF_HAPROXY_REFRESH_RATE\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\nONEAPP_VNF_DNS_INTERFACES = \"$ONEAPP_VNF_DNS_INTERFACES\"\nONEAPP_VNF_DNS_NAMESERVERS = \"$ONEAPP_VNF_DNS_NAMESERVERS\"\nONEAPP_VNF_NAT4_ENABLED = \"$ONEAPP_VNF_NAT4_ENABLED\"\nONEAPP_VNF_NAT4_INTERFACES_OUT = \"$ONEAPP_VNF_NAT4_INTERFACES_OUT\"\nONEAPP_VNF_ROUTER4_ENABLED = \"$ONEAPP_VNF_ROUTER4_ENABLED\"\nONEAPP_VNF_ROUTER4_INTERFACES = \"$ONEAPP_VNF_ROUTER4_INTERFACES\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 29
    },
    {
      "name": "master",
      "parents": [
        "vnf"
      ],
      "cardinality": 1,
      "min_vms": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.29\",ROLE=\"master\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_EXTRA_SANS = \"$ONEAPP_K8S_EXTRA_SANS\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_MULTUS_CONFIG = \"$ONEAPP_K8S_MULTUS_CONFIG\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_CNI_CONFIG = \"$ONEAPP_K8S_CNI_CONFIG\"\nONEAPP_K8S_CILIUM_RANGE = \"$ONEAPP_K8S_CILIUM_RANGE\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_METALLB_CONFIG = \"$ONEAPP_K8S_METALLB_CONFIG\"\nONEAPP_K8S_METALLB_RANGE = \"$ONEAPP_K8S_METALLB_RANGE\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 30
    },
    {
      "name": "worker",
      "parents": [
        "vnf"
      ],
      "cardinality": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.29\",ROLE=\"worker\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 30
    },
    {
      "name": "storage",
      "parents": [
        "vnf"
      ],
      "cardinality": 0,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.29\",ROLE=\"storage\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_STORAGE_DEVICE = \"$ONEAPP_STORAGE_DEVICE\"\nONEAPP_STORAGE_FILESYSTEM = \"$ONEAPP_STORAGE_FILESYSTEM\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cooldown": 120,
      "elasticity_policies": [],
      "scheduled_policies": [],
      "vm_template": 31
    }
  ],
  "networks": {
    "Public": "M|network|Public||id:",
    "Private": "M|network|Private||id:"
  },
  "custom_attrs": {
    "ONEAPP_VROUTER_ETH0_VIP0": "O|text|Control Plane Endpoint VIP (IPv4)||",
    "ONEAPP_VROUTER_ETH1_VIP0": "O|text|Default Gateway VIP (IPv4)||",
    "ONEAPP_RKE2_SUPERVISOR_EP": "O|text|RKE2 Supervisor endpoint||ep0.eth0.vr:9345",
    "ONEAPP_K8S_CONTROL_PLANE_EP": "O|text|Control Plane endpoint||ep0.eth0.vr:6443",
    "ONEAPP_K8S_EXTRA_SANS": "O|text|ApiServer extra certificate SANs||localhost,127.0.0.1,ep0.eth0.vr,${vnf.TEMPLATE.CONTEXT.ETH0_IP}",
    "ONEAPP_K8S_MULTUS_ENABLED": "O|boolean|Enable Multus||NO",
    "ONEAPP_K8S_MULTUS_CONFIG": "O|text64|Multus custom config (default none)||",
    "ONEAPP_K8S_CNI_PLUGIN": "O|list|CNI plugin supported by RKE2|canal,calico,cilium|cilium",
    "ONEAPP_K8S_CNI_CONFIG": "O|text64|CNI custom config (default none)||",
    "ONEAPP_K8S_CILIUM_RANGE": "O|text|Cilium LB IP CIDR (default none)||",
    "ONEAPP_K8S_METALLB_ENABLED": "O|boolean|Enable MetalLB||NO",
    "ONEAPP_K8S_METALLB_RANGE": "O|text|MetalLB IP range (default none)||",
    "ONEAPP_K8S_METALLB_CONFIG": "O|text64|MetalLB custom config (default none)||",
    "ONEAPP_K8S_LONGHORN_ENABLED": "O|boolean|Enable Longhorn||NO",
    "ONEAPP_STORAGE_DEVICE": "O|text|Storage device path||/dev/vdb",
    "ONEAPP_STORAGE_FILESYSTEM": "O|text|Storage device filesystem||xfs",
    "ONEAPP_K8S_TRAEFIK_ENABLED": "O|boolean|Enable Traefik||NO",
    "ONEAPP_VNF_HAPROXY_INTERFACES": "O|text|Interfaces to run Haproxy on||eth0",
    "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "O|number|Haproxy refresh rate||30",
    "ONEAPP_VNF_HAPROXY_LB0_PORT": "O|number|HTTPS RKE2 Supervisor port||9345",
    "ONEAPP_VNF_HAPROXY_LB1_PORT": "O|number|HTTPS API Server port||6443",
    "ONEAPP_VNF_HAPROXY_LB2_PORT": "O|number|HTTPS ingress port||443",
    "ONEAPP_VNF_HAPROXY_LB3_PORT": "O|number|HTTP ingress port||80",
    "ONEAPP_VNF_DNS_ENABLED": "O|boolean|Enable DNS recursor||YES",
    "ONEAPP_VNF_DNS_INTERFACES": "O|text|DNS - Interfaces||eth1",
    "ONEAPP_VNF_DNS_NAMESERVERS": "O|text|DNS - Nameservers||1.1.1.1,8.8.8.8",
    "ONEAPP_VNF_NAT4_ENABLED": "O|boolean|Enable NAT||YES",
    "ONEAPP_VNF_NAT4_INTERFACES_OUT": "O|text|NAT - Outgoing Interfaces||eth0",
    "ONEAPP_VNF_ROUTER4_ENABLED": "O|boolean|Enable Router||YES",
    "ONEAPP_VNF_ROUTER4_INTERFACES": "O|text|Router - Interfaces||eth0,eth1"
  },
  "ready_status_gate": true
}

Note

In the jq command we removed the registration_time key from the document as it is immutable.

Next, let's update the template:

$ oneflow-template update 'Service OneKE 1.29' /tmp/OneKE-update.json

Deploying with Terraform

Note

For information about the OpenNebula Terraform Provider please see the documentation.

Below is an example HCL2 single-file config to deploy a basic non-Airgapped, non-HA OneKE instance:

terraform {
  required_providers {
    opennebula = {
      source = "OpenNebula/opennebula"
      version = ">= 1.4.0"
    }
  }
}

variable "one_user" {
  type = string
  description = "User account name to authenticate with OpenNebula"
#  default = "oneadmin"
}

variable "one_password" {
  type = string
  description = "Password for the user account to authenticate with OpenNebula"
}

variable "one_host" {
  type = string
  description = "Hostname or IP of the host where OpenNebula is running"
#  default = "localhost"
}

data "http" "appliances" {
  url = "https://marketplace.opennebula.io/appliance"
  request_headers = {
    User-Agent = "OpenNebula 6.8.0"
    Accept     = "application/json"
  }
}

locals {
  appliances = {
    for a in jsondecode(data.http.appliances.response_body).appliances : a.name => a
  }

  name = "Service OneKE 1.29"

  service = local.appliances[local.name]

  roles = {
    for k, v in local.service.roles : k => merge(local.appliances[v], {
      opennebula_template = jsondecode(local.appliances[v].opennebula_template)
    })
  }

  md5_to_url = {
    for f in distinct(flatten([
      for r in values(local.roles) : concat(
        try(r.files, []),
        [for d in try(r.disks, []) : local.appliances[d].files]
      )
    ])) : f.md5 => f.url
  }

  role_to_md5 = {
    for k, v in local.roles : k => [
      for f in flatten(concat(
        try(v.files, []),
        [for d in try(v.disks, []) : local.appliances[d].files]
      )) : f.md5
    ]
  }
}

provider "opennebula" {
  endpoint      = "http://${var.one_host}:2633/RPC2"
  flow_endpoint = "http://${var.one_host}:2474"
  username      = var.one_user
  password      = var.one_password
}

resource "random_id" "oneke" {
  byte_length = 4
}

resource "opennebula_image" "oneke" {
  for_each     = local.md5_to_url
  name         = "${local.name} ${each.key} ${random_id.oneke.id}"
  datastore_id = 1
  persistent   = false
  permissions  = 642
  dev_prefix   = "vd"
  driver       = "qcow2"
  path         = each.value
}

resource "opennebula_template" "oneke" {
  for_each = local.roles
  name     = "${local.name} ${each.key} ${random_id.oneke.id}"
  cpu      = try(each.value["opennebula_template"].CPU, null)
  vcpu     = try(each.value["opennebula_template"].VCPU, null)
  memory   = try(each.value["opennebula_template"].MEMORY, null)
  context  = try(each.value["opennebula_template"].CONTEXT, null)

  dynamic "graphics" {
    for_each = try([each.value["opennebula_template"].GRAPHICS], [])
    content {
      type   = try(graphics.value.TYPE, null)
      listen = try(graphics.value.LISTEN, null)
    }
  }

  dynamic "os" {
    for_each = try([each.value["opennebula_template"].OS], [])
    content {
      arch = try(os.value.ARCH, null)
      boot = "disk0"
    }
  }

  dynamic "disk" {
    for_each = local.role_to_md5[each.key]
    content {
      image_id = opennebula_image.oneke[disk.value].id
    }
  }
}

resource "opennebula_service_template" "oneke" {
  name        = "${local.name} ${random_id.oneke.id}"
  permissions = 642
  uname       = "oneadmin"
  gname       = "oneadmin"
  template = jsonencode({ "TEMPLATE" = { "BODY" = merge(
    jsondecode(local.service["opennebula_template"]),
    {
      "roles" : [
        for r in jsondecode(local.service["opennebula_template"]).roles : merge(
          r,
          { vm_template = tonumber(opennebula_template.oneke[r.name].id) }
        )
      ]
    }
  ) } })
}

resource "opennebula_service" "oneke" {
  name        = "${local.name} ${random_id.oneke.id}"
  template_id = opennebula_service_template.oneke.id
  extra_template = jsonencode({
    networks_values = [
      { Public = { id = "0" } },
      { Private = { id = "1" } },
    ]
    custom_attrs_values = {
      ONEAPP_VROUTER_ETH0_VIP0 = ""
      ONEAPP_VROUTER_ETH1_VIP0 = ""

      ONEAPP_RKE2_SUPERVISOR_EP   = "ep0.eth0.vr:9345"
      ONEAPP_K8S_CONTROL_PLANE_EP = "ep0.eth0.vr:6443"
      ONEAPP_K8S_EXTRA_SANS       = "localhost,127.0.0.1,ep0.eth0.vr,$${vnf.TEMPLATE.CONTEXT.ETH0_IP}"

      ONEAPP_K8S_MULTUS_ENABLED = "NO"
      ONEAPP_K8S_MULTUS_CONFIG  = ""

      ONEAPP_K8S_CNI_PLUGIN   = "cilium"
      ONEAPP_K8S_CNI_CONFIG   = ""
      ONEAPP_K8S_CILIUM_RANGE = ""

      ONEAPP_K8S_METALLB_ENABLED = "NO"
      ONEAPP_K8S_METALLB_CONFIG  = ""
      ONEAPP_K8S_METALLB_RANGE   = ""

      ONEAPP_K8S_LONGHORN_ENABLED = "YES"
      ONEAPP_STORAGE_DEVICE       = "/dev/vdb"
      ONEAPP_STORAGE_FILESYSTEM   = "xfs"

      ONEAPP_K8S_TRAEFIK_ENABLED      = "YES"
      ONEAPP_VNF_HAPROXY_INTERFACES   = "eth0"
      ONEAPP_VNF_HAPROXY_REFRESH_RATE = "30"
      ONEAPP_VNF_HAPROXY_LB0_PORT     = "9345"
      ONEAPP_VNF_HAPROXY_LB1_PORT     = "6443"
      ONEAPP_VNF_HAPROXY_LB2_PORT     = "443"
      ONEAPP_VNF_HAPROXY_LB3_PORT     = "80"

      ONEAPP_VNF_DNS_ENABLED         = "YES"
      ONEAPP_VNF_DNS_INTERFACES      = "eth1"
      ONEAPP_VNF_DNS_NAMESERVERS     = "1.1.1.1,8.8.8.8"
      ONEAPP_VNF_NAT4_ENABLED        = "YES"
      ONEAPP_VNF_NAT4_INTERFACES_OUT = "eth0"
      ONEAPP_VNF_ROUTER4_ENABLED     = "YES"
      ONEAPP_VNF_ROUTER4_INTERFACES  = "eth0,eth1"
    }
  })
  timeouts {
    create = "10m"
    delete = "5m"
  }
}

Important

The above script takes care of downloading templates and images; no preliminary oneflow export operation is required.

Customizing the Deployment

It is possible to modify VM templates related to the OneKE Virtual Appliance in order to customize the deployment, for example by adding more VM memory or VCPU cores to the workers, and resizing the disks for the storage nodes. This should be done before the creation of the K8s cluster, i.e. before instantiating the OneKE OneFlow Service Template.

When instantiating OneKE's OneFlow Service Template, you can further customize the deployment using the following custom attributes:

Parameter Default Roles Description
ONEAPP_VROUTER_ETH0_VIP0 all Control Plane Endpoint VIP (IPv4)
ONEAPP_VROUTER_ETH1_VIP0 all Default Gateway VIP (IPv4)
ONEAPP_RKE2_SUPERVISOR_EP ep0.eth0.vr:9345 master RKE2 Supervisor endpoint
ONEAPP_K8S_CONTROL_PLANE_EP ep0.eth0.vr:6443 !vnf Control Plane endpoint
ONEAPP_K8S_EXTRA_SANS localhost,127.0.0.1,ep0.eth0.vr, ${vnf.TEMPLATE.CONTEXT.ETH0_IP} master ApiServer extra certificate SANs
ONEAPP_K8S_MULTUS_ENABLED NO all Enable Multus
ONEAPP_K8S_MULTUS_CONFIG master Multus custom config
ONEAPP_K8S_CNI_PLUGIN cilium all CNI plugin (canal, calico or cilium)
ONEAPP_K8S_CNI_CONFIG master CNI custom config
ONEAPP_K8S_CILIUM_RANGE master Cilium LB IP/CIDR
ONEAPP_K8S_METALLB_ENABLED NO master,worker Enable MetalLB
ONEAPP_K8S_METALLB_RANGE master MetalLB IP range
ONEAPP_K8S_METALLB_CONFIG master MetalLB custom config
ONEAPP_K8S_LONGHORN_ENABLED NO all Enable Longhorn
ONEAPP_STORAGE_DEVICE /dev/vdb storage Dedicated storage device for Longhorn
ONEAPP_STORAGE_FILESYSTEM xfs storage Filesystem type to init dedicated storage device
ONEAPP_K8S_TRAEFIK_ENABLED NO master,worker Enable Traefik
ONEAPP_VNF_HAPROXY_INTERFACES eth0 vnf Interfaces to run HAProxy on
ONEAPP_VNF_HAPROXY_REFRESH_RATE 30 vnf HAProxy / OneGate refresh rate
ONEAPP_VNF_HAPROXY_LB0_PORT 9345 vnf,master HTTPS RKE2 Supervisor port
ONEAPP_VNF_HAPROXY_LB1_PORT 6443 vnf,master HTTPS API Server port
ONEAPP_VNF_HAPROXY_LB2_PORT 443 vnf,worker HTTPS ingress port
ONEAPP_VNF_HAPROXY_LB3_PORT 80 vnf,worker HTTP ingress port
ONEAPP_VNF_DNS_ENABLED YES all Enable DNS recursor
ONEAPP_VNF_DNS_INTERFACES eth1 vnf DNS - Interfaces
ONEAPP_VNF_DNS_NAMESERVERS 1.1.1.1,8.8.8.8 vnf DNS - Nameservers
ONEAPP_VNF_NAT4_ENABLED YES vnf Enable NAT for the whole cluster
ONEAPP_VNF_NAT4_INTERFACES_OUT eth0 vnf NAT - Outgoing (public) interfaces
ONEAPP_VNF_ROUTER4_ENABLED YES vnf Enable IPv4 forwarding for selected NICs
ONEAPP_VNF_ROUTER4_INTERFACES eth0,eth1 vnf IPv4 Router - NICs selected for IPv4 forwarding

Important

Virtual IP Parameters

ONEAPP_VROUTER_ETH0_VIP0: The VNF cluster uses this VIP to bind and expose the Kubernetes API port 6443 and RKE2's management port 9345. The eth0 NIC should be connected (routed or NATed) to the public subnet. Note that you can leave this parameter empty; in this case OneKE will assume this is a non-HA configuration, and will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

ONEAPP_VROUTER_ETH0_VIP0: The VNF cluster uses this VIP to act for every VM deployed in the private subnet. The eth1 NIC should be connected to the private subnet. Note that you can leave this parameter empty; in this case OneKE will assume this is a non-HA configuration, and will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

Important

If you enable Longhorn (via ONEAPP_K8S_LONGHORN_ENABLED), you will also need to scale up the storage role or modify the cardinality parameter in the service template.

Home

Clone this wiki locally