oneke_deploy

Deploying OneKE

In this section we focus on a deployment of OneKE using CLI commands. For an easier Sunstone UI guide (with screenshots) please refer to the Running Kubernetes Clusters quick-start document.

Importing the OneKE Appliance

Let's run the following command to import in the OpenNebula Cloud the whole set of resources corresponding to the OneKE Virtual Appliance. An image datastore must be specified for storing the Virtual Appliance images.

$ onemarketapp export 'Service OneKE 1.27' 'Service OneKE 1.27' --datastore 1
IMAGE
    ID: 202
    ID: 203
    ID: 204
VMTEMPLATE
    ID: 204
    ID: 205
    ID: 206
SERVICE_TEMPLATE
    ID: 104

Note

IDs are automatically assigned and their actual values depend on the state of the OpenNebula cluster at hand.

Important

You can also use the Service OneKE 1.27a, the airgapped version that includes all the docker images required for initial deployment (without the Internet connection).

Create a K8s Cluster

Once the OneKE Virtual Appliance has been imported, a new cluster can be created by instantiating the OneKE OneFlow Service as shown here:

$ cat >/tmp/OneKE-instantiate <<'EOF'
{
    "name": "OneKE/1",
    "networks_values": [
        {"Public": {"id": "0"}},
        {"Private": {"id": "1"}}
    ],
    "custom_attrs_values": {
        "ONEAPP_VROUTER_ETH0_VIP0": "",
        "ONEAPP_VROUTER_ETH1_VIP0": "",

        "ONEAPP_RKE2_SUPERVISOR_EP": "ep0.eth0.vr:9345",
        "ONEAPP_K8S_CONTROL_PLANE_EP": "ep0.eth0.vr:6443",
        "ONEAPP_K8S_EXTRA_SANS": "localhost,127.0.0.1,ep0.eth0.vr,k8s.yourdomain.it",

        "ONEAPP_K8S_MULTUS_ENABLED": "YES",
        "ONEAPP_K8S_MULTUS_CONFIG": "",
        "ONEAPP_K8S_CNI_PLUGIN": "cilium",
        "ONEAPP_K8S_CNI_CONFIG": "",
        "ONEAPP_K8S_CILIUM_RANGE": "",

        "ONEAPP_K8S_METALLB_ENABLED": "YES",
        "ONEAPP_K8S_METALLB_CONFIG": "",
        "ONEAPP_K8S_METALLB_RANGE": "172.20.0.87-172.20.0.88",

        "ONEAPP_K8S_LONGHORN_ENABLED": "YES",
        "ONEAPP_STORAGE_DEVICE": "/dev/vdb",
        "ONEAPP_STORAGE_FILESYSTEM": "xfs",

        "ONEAPP_K8S_TRAEFIK_ENABLED": "YES",
        "ONEAPP_VNF_HAPROXY_INTERFACES": "eth0",
        "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "30",
        "ONEAPP_VNF_HAPROXY_LB0_PORT": "9345",
        "ONEAPP_VNF_HAPROXY_LB1_PORT": "6443",
        "ONEAPP_VNF_HAPROXY_LB2_PORT": "443",
        "ONEAPP_VNF_HAPROXY_LB3_PORT": "80",

        "ONEAPP_VNF_DNS_ENABLED": "YES",
        "ONEAPP_VNF_DNS_INTERFACES": "eth1",
        "ONEAPP_VNF_NAT4_ENABLED": "YES",
        "ONEAPP_VNF_NAT4_INTERFACES_OUT": "eth0",
        "ONEAPP_VNF_ROUTER4_ENABLED": "YES",
        "ONEAPP_VNF_ROUTER4_INTERFACES": "eth0,eth1"
    }
}
EOF
$ oneflow-template instantiate 'Service OneKE 1.27' /tmp/OneKE-instantiate
ID: 105

K8s cluster creation can take some minutes. The cluster is available once the OneFlow service is in RUNNING state

$ oneflow show 'OneKE/1'
SERVICE 105 INFORMATION
ID                  : 105
NAME                : OneKE/1
USER                : oneadmin
GROUP               : oneadmin
STRATEGY            : straight
SERVICE STATE       : RUNNING

and all VMs are also in RUNNING state

$ onevm list -f NAME~'service_105' -l NAME,STAT
NAME                    ... STAT
storage_0_(service_105) ... runn
worker_0_(service_105)  ... runn
master_0_(service_105)  ... runn
vnf_0_(service_105)     ... runn

Deployment Customization

It is possible to modify VM templates related to the OneKE Virtual Appliance in order to customize the deployment, for example by adding more VM memory, VCPU cores to the workers, and resizing the Disk for the storage nodes. This should be done before the creation of the K8s cluster, i.e. before instantiating the OneKE OneFlow Service Template.

When instantiating OneKE's OneFlow Service Template, you can further customize the deployment using the following custom attributes:

Parameter	Default	Roles	Description
`ONEAPP_VROUTER_ETH0_VIP0`		all	Control Plane Endpoint VIP (IPv4)
`ONEAPP_VROUTER_ETH1_VIP0`		all	Default Gateway VIP (IPv4)
`ONEAPP_RKE2_SUPERVISOR_EP`	`ep0.eth0.vr:9345`	master	RKE2 Supervisor endpoint
`ONEAPP_K8S_CONTROL_PLANE_EP`	`ep0.eth0.vr:6443`	!vnf	Control Plane endpoint
`ONEAPP_K8S_EXTRA_SANS`	`localhost,127.0.0.1,ep0.eth0.vr`	master	ApiServer extra certificate SANs
`ONEAPP_K8S_MULTUS_ENABLED`	`NO`	all	Enable Multus
`ONEAPP_K8S_MULTUS_CONFIG`		master	Multus custom config
`ONEAPP_K8S_CNI_PLUGIN`	`cilium`	all	CNI plugin (canal, calico or cilium)
`ONEAPP_K8S_CNI_CONFIG`		master	CNI custom config
`ONEAPP_K8S_CILIUM_RANGE`		master	Cilium LB IP/CIDR
`ONEAPP_K8S_METALLB_ENABLED`	`NO`	master,worker	Enable MetalLB
`ONEAPP_K8S_METALLB_RANGE`		master	MetalLB IP range
`ONEAPP_K8S_METALLB_CONFIG`		master	MetalLB custom config
`ONEAPP_K8S_LONGHORN_ENABLED`	`NO`	all	Enable Longhorn
`ONEAPP_STORAGE_DEVICE`	`/dev/vdb`	storage	Dedicated storage device for Longhorn
`ONEAPP_STORAGE_FILESYSTEM`	`xfs`	storage	Filesystem type to init dedicated storage device
`ONEAPP_K8S_TRAEFIK_ENABLED`	`NO`	master,worker	Enable Traefik
`ONEAPP_VNF_HAPROXY_INTERFACES`	`eth0`	vnf	Interfaces to run HAProxy on
`ONEAPP_VNF_HAPROXY_REFRESH_RATE`	`30`	vnf	HAProxy / OneGate refresh rate
`ONEAPP_VNF_HAPROXY_LB0_PORT`	`9345`	vnf,master	HTTPS RKE2 Supervisor port
`ONEAPP_VNF_HAPROXY_LB1_PORT`	`6443`	vnf,master	HTTPS API Server port
`ONEAPP_VNF_HAPROXY_LB2_PORT`	`443`	vnf,worker	HTTPS ingress port
`ONEAPP_VNF_HAPROXY_LB3_PORT`	`80`	vnf,worker	HTTP ingress port
`ONEAPP_VNF_DNS_ENABLED`	`YES`	all	Enable DNS recursor
`ONEAPP_VNF_DNS_INTERFACES`	`eth1`	vnf	DNS - Interfaces
`ONEAPP_VNF_NAT4_ENABLED`	`YES`	vnf	Enable NAT for the whole cluster
`ONEAPP_VNF_NAT4_INTERFACES_OUT`	`eth0`	vnf	NAT - Outgoing (public) interfaces
`ONEAPP_VNF_ROUTER4_ENABLED`	`YES`	vnf	Enable IPv4 forwarding for selected NICs
`ONEAPP_VNF_ROUTER4_INTERFACES`	`eth0,eth1`	vnf	IPv4 Router - NICs selected for IPv4 forwarding

Important

ONEAPP_VROUTER_ETH0_VIP0 - VNF cluster uses this VIP to bind and expose Kubernetes API port 6443 and RKE2's management port 9345. The eth0 NIC should be connected to the public subnet (Routed or NATed). You can leave this parameter empty, then OneKE will assume it's a non-HA setup and it will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

Important

ONEAPP_VROUTER_ETH1_VIP0 - VNF cluster uses this VIP to act as a NAT gateway for every other VM deployed inside the private subnet. The eth1 NIC should be connected to the private subnet. You can leave this parameter empty, then OneKE will assume it's a non-HA setup and it will try to use a regular IP (non-VIP) address of the VNF instance (single VM).

Important

If you enable Longhorn (via ONEAPP_K8S_LONGHORN_ENABLED), then you also need to scale up the storage role or modify the cardinality parameter in the service template.

High-Availability

By default, OneKE Virtual Appliance is preconfigured to work as a non-Highly-Available K8s cluster, since OneFlow Service Templates deploys each service role as a single VM. Kubernetes High-Availability is about setting up a Kubernetes cluster, along with its components, in such a way that there is no single point of failure. To achieve high-availability, the following OneKE components should be scaled up: VNF (at least 2 VMs), master (at least 3 VMs) and storage (at least 2 VMs).

OneKE HA setup can be achieved by modifying the OneFlow Service Template before creating the cluster or by scaling up each role after the cluster creation.

For example, to scale the master role from a single node to 3, you can use the following command:

$ oneflow scale 'OneKE/1' master 3

Warning

You can scale the master role up to an odd number of masters, but be careful while scaling down as it may break your cluster. If you require multi-master HA, just start with a single master and then scale up to 3 and keep it that way.

After a while we can examine the service log:

$ oneflow show 'OneKE/1'
...
LOG MESSAGES
05/11/23 18:30 [I] New state: DEPLOYING_NETS
05/11/23 18:30 [I] New state: DEPLOYING
05/11/23 18:39 [I] New state: RUNNING
05/11/23 18:43 [I] Role master scaling up from 1 to 3 nodes
05/11/23 18:43 [I] New state: SCALING
05/11/23 18:52 [I] New state: COOLDOWN
05/11/23 18:55 [I] New state: RUNNING

And afterwards we can list cluster nodes using kubectl:

$ kubectl get nodes
NAME                    STATUS   ROLES                       AGE     VERSION
oneke-ip-172-20-0-101   Ready    control-plane,etcd,master   31m     v1.27.2+rke2r1
oneke-ip-172-20-0-102   Ready    <none>                      28m     v1.27.2+rke2r1
oneke-ip-172-20-0-103   Ready    <none>                      28m     v1.27.2+rke2r1
oneke-ip-172-20-0-104   Ready    control-plane,etcd,master   11m     v1.27.2+rke2r1
oneke-ip-172-20-0-105   Ready    control-plane,etcd,master   10m     v1.27.2+rke2r1

Warning

Please plan ahead and avoid scaling down master and storage roles as it may break ETCD's quorum or cause data loss. There is no obvious restriction for the worker role, however. It can be safely rescaled at will.

Anti-affinity

VMs related to the same role should be scheduled on different physical hosts in an HA setup to guarantee HA in case of a host failure. OpenNebula provides VM Group resources to achieve proper Host/VM affinity/anti-affinity.

In the following section, we provide an example of how to create VM Group resources and how to modify OneKE's OneFlow Service Template to include VM groups.

Let's assume that epsilon and omicron are hosts we want to use to deploy OneKE; a VM Group may be created in the following way:

$ cat >/tmp/OneKE-vmgroup <<'EOF'
NAME = "Service OneKE 1.27"
ROLE = [
    NAME         = "vnf",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "master",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
ROLE = [
    NAME         = "worker",
    HOST_AFFINED = "epsilon,omicron"
]
ROLE = [
    NAME         = "storage",
    HOST_AFFINED = "epsilon,omicron",
    POLICY       = "ANTI_AFFINED"
]
EOF

$ onevmgroup create /tmp/OneKE-vmgroup
ID: 1

Important

The worker role does not have POLICY defined, this allows you to reuse hosts multiple times!

Now, let's modify the OneKE OneFlow Service Template:

$ oneflow-template show 'Service OneKE 1.27' --json | >/tmp/OneKE-update.json jq -r --arg vmgroup 'Service OneKE 1.27' -f /dev/fd/3 3<<'EOF'
.DOCUMENT.TEMPLATE.BODY | del(.registration_time) | . += {
  roles: .roles | map(
    .vm_template_contents = "VMGROUP=[VMGROUP_NAME=\"\($vmgroup)\",ROLE=\"\(.name)\"]\n" + .vm_template_contents
  )
}
EOF

Content of the update (/tmp/OneKE-update.json) will look like this:

{
  "custom_attrs": {
    "ONEAPP_K8S_CILIUM_RANGE": "O|text|Cilium LB IP CIDR (default none)||",
    "ONEAPP_K8S_CNI_CONFIG": "O|text64|CNI custom config (default none)||",
    "ONEAPP_K8S_CNI_PLUGIN": "O|list|CNI plugin supported by RKE2|canal,calico,cilium|cilium",
    "ONEAPP_K8S_CONTROL_PLANE_EP": "O|text|Control Plane endpoint||ep0.eth0.vr:6443",
    "ONEAPP_K8S_EXTRA_SANS": "O|text|ApiServer extra certificate SANs||localhost,127.0.0.1,ep0.eth0.vr",
    "ONEAPP_K8S_LONGHORN_ENABLED": "O|boolean|Enable Longhorn||NO",
    "ONEAPP_K8S_METALLB_CONFIG": "O|text64|MetalLB custom config (default none)||",
    "ONEAPP_K8S_METALLB_ENABLED": "O|boolean|Enable MetalLB||NO",
    "ONEAPP_K8S_METALLB_RANGE": "O|text|MetalLB IP range (default none)||",
    "ONEAPP_K8S_MULTUS_CONFIG": "O|text64|Multus custom config (default none)||",
    "ONEAPP_K8S_MULTUS_ENABLED": "O|boolean|Enable Multus||NO",
    "ONEAPP_K8S_TRAEFIK_ENABLED": "O|boolean|Enable Traefik||NO",
    "ONEAPP_RKE2_SUPERVISOR_EP": "O|text|RKE2 Supervisor endpoint||ep0.eth0.vr:9345",
    "ONEAPP_STORAGE_DEVICE": "O|text|Storage device path||/dev/vdb",
    "ONEAPP_STORAGE_FILESYSTEM": "O|text|Storage device filesystem||xfs",
    "ONEAPP_VNF_DNS_ENABLED": "O|boolean|Enable DNS recursor||YES",
    "ONEAPP_VNF_DNS_INTERFACES": "O|text|DNS - Interfaces||eth1",
    "ONEAPP_VNF_HAPROXY_INTERFACES": "O|text|Interfaces to run Haproxy on||eth0",
    "ONEAPP_VNF_HAPROXY_LB0_PORT": "O|number|HTTPS RKE2 Supervisor port||9345",
    "ONEAPP_VNF_HAPROXY_LB1_PORT": "O|number|HTTPS API Server port||6443",
    "ONEAPP_VNF_HAPROXY_LB2_PORT": "O|number|HTTPS ingress port||443",
    "ONEAPP_VNF_HAPROXY_LB3_PORT": "O|number|HTTP ingress port||80",
    "ONEAPP_VNF_HAPROXY_REFRESH_RATE": "O|number|Haproxy refresh rate||30",
    "ONEAPP_VNF_NAT4_ENABLED": "O|boolean|Enable NAT||YES",
    "ONEAPP_VNF_NAT4_INTERFACES_OUT": "O|text|NAT - Outgoing Interfaces||eth0",
    "ONEAPP_VNF_ROUTER4_ENABLED": "O|boolean|Enable Router||YES",
    "ONEAPP_VNF_ROUTER4_INTERFACES": "O|text|Router - Interfaces||eth0,eth1",
    "ONEAPP_VROUTER_ETH0_VIP0": "O|text|Control Plane Endpoint VIP (IPv4)||",
    "ONEAPP_VROUTER_ETH1_VIP0": "O|text|Default Gateway VIP (IPv4)||"
  },
  "deployment": "straight",
  "name": "OneKE 1.27",
  "networks": {
    "Private": "M|network|Private||id:",
    "Public": "M|network|Public||id:"
  },
  "ready_status_gate": true,
  "roles": [
    {
      "cardinality": 1,
      "cooldown": 120,
      "min_vms": 1,
      "name": "vnf",
      "vm_template": 2,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"vnf\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Public\" ]\nNIC = [\n  NAME = \"NIC1\",\n  NETWORK_ID = \"$Private\" ]\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_VNF_HAPROXY_INTERFACES = \"$ONEAPP_VNF_HAPROXY_INTERFACES\"\nONEAPP_VNF_HAPROXY_REFRESH_RATE = \"$ONEAPP_VNF_HAPROXY_REFRESH_RATE\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\nONEAPP_VNF_DNS_INTERFACES = \"$ONEAPP_VNF_DNS_INTERFACES\"\nONEAPP_VNF_NAT4_ENABLED = \"$ONEAPP_VNF_NAT4_ENABLED\"\nONEAPP_VNF_NAT4_INTERFACES_OUT = \"$ONEAPP_VNF_NAT4_INTERFACES_OUT\"\nONEAPP_VNF_ROUTER4_ENABLED = \"$ONEAPP_VNF_ROUTER4_ENABLED\"\nONEAPP_VNF_ROUTER4_INTERFACES = \"$ONEAPP_VNF_ROUTER4_INTERFACES\"\n"
    },
    {
      "cardinality": 1,
      "cooldown": 120,
      "min_vms": 1,
      "name": "master",
      "parents": [
        "vnf"
      ],
      "vm_template": 0,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"master\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_EXTRA_SANS = \"$ONEAPP_K8S_EXTRA_SANS\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_MULTUS_CONFIG = \"$ONEAPP_K8S_MULTUS_CONFIG\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_CNI_CONFIG = \"$ONEAPP_K8S_CNI_CONFIG\"\nONEAPP_K8S_CILIUM_RANGE = \"$ONEAPP_K8S_CILIUM_RANGE\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_METALLB_CONFIG = \"$ONEAPP_K8S_METALLB_CONFIG\"\nONEAPP_K8S_METALLB_RANGE = \"$ONEAPP_K8S_METALLB_RANGE\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n"
    },
    {
      "cardinality": 1,
      "cooldown": 120,
      "name": "worker",
      "parents": [
        "vnf"
      ],
      "vm_template": 3,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"worker\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_K8S_METALLB_ENABLED = \"$ONEAPP_K8S_METALLB_ENABLED\"\nONEAPP_K8S_TRAEFIK_ENABLED = \"$ONEAPP_K8S_TRAEFIK_ENABLED\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_HAPROXY_LB2_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB2_PORT = \"$ONEAPP_VNF_HAPROXY_LB2_PORT\"\nONEAPP_VNF_HAPROXY_LB3_IP = \"<ETH0_EP0>\"\nONEAPP_VNF_HAPROXY_LB3_PORT = \"$ONEAPP_VNF_HAPROXY_LB3_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n"
    },
    {
      "cooldown": 120,
      "name": "storage",
      "parents": [
        "vnf"
      ],
      "vm_template": 1,
      "vm_template_contents": "VMGROUP=[VMGROUP_NAME=\"Service OneKE 1.27\",ROLE=\"storage\"]\nNIC = [\n  NAME = \"NIC0\",\n  NETWORK_ID = \"$Private\" ]\nFALLBACK_GW = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nFALLBACK_DNS = \"${vnf.TEMPLATE.CONTEXT.ETH1_IP}\"\nONEAPP_VROUTER_ETH0_VIP0 = \"$ONEAPP_VROUTER_ETH0_VIP0\"\nONEAPP_VROUTER_ETH1_VIP0 = \"$ONEAPP_VROUTER_ETH1_VIP0\"\nONEAPP_RKE2_SUPERVISOR_EP = \"$ONEAPP_RKE2_SUPERVISOR_EP\"\nONEAPP_K8S_CONTROL_PLANE_EP = \"$ONEAPP_K8S_CONTROL_PLANE_EP\"\nONEAPP_K8S_MULTUS_ENABLED = \"$ONEAPP_K8S_MULTUS_ENABLED\"\nONEAPP_K8S_CNI_PLUGIN = \"$ONEAPP_K8S_CNI_PLUGIN\"\nONEAPP_K8S_LONGHORN_ENABLED = \"$ONEAPP_K8S_LONGHORN_ENABLED\"\nONEAPP_STORAGE_DEVICE = \"$ONEAPP_STORAGE_DEVICE\"\nONEAPP_STORAGE_FILESYSTEM = \"$ONEAPP_STORAGE_FILESYSTEM\"\nONEAPP_VNF_HAPROXY_LB0_PORT = \"$ONEAPP_VNF_HAPROXY_LB0_PORT\"\nONEAPP_VNF_HAPROXY_LB1_PORT = \"$ONEAPP_VNF_HAPROXY_LB1_PORT\"\nONEAPP_VNF_DNS_ENABLED = \"$ONEAPP_VNF_DNS_ENABLED\"\n",
      "cardinality": 1
    }
  ],
  "description": ""
}

Note

We removed the registration_time key from the document as it is immutable.

Next, let's update the template:

$ oneflow-template update 'Service OneKE 1.27' /tmp/OneKE-update.json

Home

OpenNebula Apps Overview
OS Appliances Update Policy
OneApps Quick Intro
Build Instructions
Linux Contextualization Packages
Windows Contextualization Packages
OneKE (OpenNebula Kubernetes Edition)
Virtual Router
- Release Notes
- Overview
- Quick Start
- OpenRC Services
- Virtual Router Modules
  - Router4
  - NAT4
  - SDNAT4
  - Load Balancing
  - DNS
  - DHCP4
  - Keepalived: Failover
  - Wireguard VPN
- Glossary
WordPress
Harbor Container Registry
MinIO
Development
- Virtual Router

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

oneke_deploy

Deploying OneKE

Importing the OneKE Appliance

Create a K8s Cluster

Deployment Customization

High-Availability

Anti-affinity

Clone this wiki locally