Skip to content

Latest commit

 

History

History
502 lines (292 loc) · 24.3 KB

APP_K8S_K3S_STEP_BY_STEP_DEPLOYMENT.MD

File metadata and controls

502 lines (292 loc) · 24.3 KB

Step-by-step application components deployment into AKS-EE-Windows or k3s-Ubuntu-Linux

Supported environment infrastructure at the edge

This version of MEC-App-Accelerator supports the following Kubernetes distributions and host operating system.

  • AKS Edge Essentials on Windows
  • K3s on Ubuntu Linux

Kubernetes from Docker-Desktop is not supported by Azure IoT Operations, however, if using the below OSS/Mosquito alternative, it should also work for you, but we recommend the above environments since those are the tested environments by our team.

Supported edge components on top of the Kubernetes/k3s cluster:

  • Azure IoT Operations alternative

    • Azure IoT MQ as messaging broker
    • AKRI for dynamic cameras provissioning (OSS version)
  • Plain OSS alternative

    • Eclipse Mosquitto as messaging broker
    • AKRI for dynamic cameras provissioning (OSS version)

Install the Kubernetes/K3s cluster at the edge

Install the AKS-EE-Windows cluster or the K3s-Ubuntu cluster by following the instructions in the below Microsoft official doc.

Even when the below documentation link is the supported baseline environment for an Azure IoT Operations cluster, you can also follow this AKS-EE/K3s cluster installation procedure documentation even if you won't use the Azure IoT Operations alternative but the plain OSS/Mosquitto alternative:

Install AKS-EE-Windows alternative:

Create your Azure Arc-enabled Kubernetes cluster on Windows

IMPORTANT: AKS-EE has no persistent storage solution out of the box which is needed by accelerator's mongodb. You can either configure a storage binding manually or use Rancher Labs Local-Path-Provisioner. Persistent storage solution must be configured before deploying the accelerator.

Install K3s-Ubuntu-Linux alternative:

Create your Azure Arc-enabled K3s cluster on Ubuntu-Linux

IMPORTANT If you are NOT going to use Azure IoT Operations, when following the previous setup doc, stop before the section named "Deploy Azure IoT Operations". Hence, the last secion to follow should be "Verify cluster".

At this point, and after verifying the cluster from the command-line, you should also be able to check it out from the Azure portal because even when it's deployed on a machine on-premises or at the edge, it's connected to Azure through Azure Arc, so you can monitor the cluster from there, like in the following screenshot:

image

However, in order to check out your cluster's Workloads (pods, deployments, etc.), you need to generate and provide a "Service account bearer token", by following this procedure Service account token authentication option and finally providing the token like in this other screenshot:

image

So then, you can monitor your AKS EE workloads in your cluster on-premises/edge from Azure cloud portal, thanks to Azure Arc connection to your cluster:

image

Azure Arc support: For most cases, it's interesting to follow the above setup instructions instead of a plain K8s/K3s setup because then you will also have your cluster as an Azure Arc-enabled cluster, so you can monitor the cluster at the edge from anywhere through Azure portal in the cloud.

Eventual removal of AKS Edge Essentials: Eventually, if you need to install the AKS-EE, follow this procedure:

Uninstall AKS Edge Essentials

Install Azure IoT Operations into your cluster

Note: You only need to install Azure IoT operations if you want to use Azure IoT MQ (aka. E4K) instead of Mosquitto, as your selected selected MQTT broker. For using AKRI, since we're curently using the OSS version of AKRI, it can be installed on any Kubernetes cluster without needing to have Azure IoT Operations installed.

Prerequisites for Azure IoT Operations

It's important to check and install the prerequisites for Azure IoT Operations explained here:

Prerequisites for Azure IoT Operations

Deploy Azure IoT Operations

Install Azure IoT Operations by following the official "simplified/automated approach" documentation here:

Deploy Azure IoT Operations Preview

When deploying Azure IoT Operations by following the above documentation link, you should see a console app similar to the following execution:

image

(OPTIONAL LEARNING) Detailed Azure IoT Operations setup

Note that there's an alternative "Detailed step-by-step AKS Edge-Essentials plus Azure IoT Operations deployment" available in Microsoft's docs, but we initially recommend to go the easy path explained above instead of the following docs which require significantly more time due to the many step-by-step instructions, especially on what's related to the AKS-EE cluster setup:

However, in case you want to know what's going on under the covers, it's interesting to take a look to those alternative and detailed setup docs for Azure IoT Operations.

Deploy the MEC-Application into the cluster (Step by step components deployment)

Check your kubectl context

  1. If you previosly had already installed kubectl and it is pointing to some other environment, such as MiniKube, make sure the default kubectl context is pointing to the right cluster.

    Open a new command-shell and run the following command to check your current contexts:

    kubectl config get-contexts

    To set the context to point to the right Kuberentes cluster, use kubectl and provide the name of the context to use:

    kubectl config use-context [my-cluster-context]

Deploy Dapr into the cluster

This example application is based on Dapr framework which is an OSS Microsoft framework specialized on microservices architecture, so you need to install it and enable in the Kubernetes cluster, first.

Install and Initialize DAPR

Install or make sure you have installed Dapr CLI on the machine that you use/run KUBECTL with the following instructions:

Install Dapr CLI

Then, from the same machine/environment you use KUBECTL against your Kubernetes cluster, install/initialize DAPR in the Kubernetes cluster by running this command:

dapr init -k

image

For further details on Dapr deployment, check this doc.

Dapr deployment details.

IMPORTANT: This DAPR installation is okay for a dev machine or edge machine with Kubernetes in AKS Edge Essentials or a single node k3s, but when installing DAPR on a larger "production" AKS cluster, for example in an ASE (Azure Stack Edge) or HCI server on in Azure AKS cluster in the cloud, you need to install DAPR via AKS extension following the Doc: (https://learn.microsoft.com/en-us/azure/aks/dapr) which is how DAPR should be installed in production environments and it doesn’t require cluster admin access.

You can test DAPR status with:

dapr status -k

If DAPR is initialized, you should get this list of Dapr pods running:

image

Otherwise, if it's not initialized it'd be like this:

image

**IMPORTANT:** This DAPR installation is okay for a dev machine with Kubernetes in AKS Edge Essentials or a single node k3s, but when installing DAPR on a "production" AKS cluster, for example in an ASE (Azure Stack Edge) server on in Azure AKS cluster in the cloud, you need to install DAPR via AKS extension following the Doc: (https://learn.microsoft.com/en-us/azure/aks/dapr) which is how DAPR should be installed in production environments and it doesn’t require cluster admin access.

Install HELM in your client machine

If you don't have HELM installed in your client system, you need to have it since AKRI-OSS is deployed through HELM.

Make sure you have install helm in the machine you are using with KUBECTL accessing the Kubernetes cluster. By default, that machine can be a Windows machine with KUBECTL installed and pointing to the right Kubernetes cluster context.

Full installation instructions of Helm can be found here.

(Optional) Instructions for building your own Docker Images to deploy to Kubernetes

This step is not needed if you directly deploy to Kubernetes and test the images we have already uploaded into Docker Hub, so creating your own Docker images and uploading to Docker Hub is optional, initially.

However, if you make changes in the .NET or Python code or you want to use your own Docker Images for any other reason, you'd need to create your own Docker images and upload the images first to Docker Hub before deploying to Kubernetes.

Use these instructions to Buil your own Docker Images to upload to Docker Registry and deploy to Kubernetes

Clone the MEC-Accelerator GitHub repo

You need to clone the MEC-Accelerator GitHub repo in the client machine from where you will deploy the application against your Kubernetes/k3s cluster.

Basically, you need to have the cloned repo in the same machine from where you also use KUBECTL against your K8s/K3s cluster.

Logically, you need to have installed Git in the first place.

Then, using a command-line tool (PowerShell in Windows or Bash in Linux) clone the repo:

git clone https://github.com/Azure/mec-app-solution-accelerator.git

Make sure you are using the MAIN branch by typing the following:

git status

Move to the deployment folder

From the console, move to the deployment folder within the cloned repo:

cd <YOUR PATH>/deploy/k8s

Deploy the application namespace (custom K8s namespace)

  1. Open a new command-shell and move into the deploy/k8s folder of this repo, as current folder of the command-shell:

    cd <your_path>/deploy/k8s
  2. Deploy the mec-accelerator namespace in Kuberentes by running this command:

    kubectl apply -f ./00-namespace.yaml

image

Update the application control plane configuration

  1. (OPTIONAL for Azure Private 5G network) Update the config map that contains essential information for connecting to our Azure Mobile Network. You'll find the configuration in the file deploy/k8s/13-control-plane-api-config-map.yaml.

Replace the following values in the file:

  MobileNetwork__SubscriptionId: "your_subscription_id"
  MobileNetwork__ResourceGroup: "your_resource_group"
  MobileNetwork__Name: "your_network_name"
  MobileNetwork__AttachedDataNetwork: "your_attachedDataNetwork_fullID"
  MobileNetwork__Slice: "your_slice_fullID"
  ClientCredentials__TenantId: "your_tenant_id"
  ClientCredentials__ClientId: "your_client_id"
  ClientCredentials__ClientSecret: "your_client_secret"

Note: This step is required if you want to manage SIMs provisioning from this app against AP5GC (Azure Private 5G Core) in Azure. For just getting alerts and camera management/provisioning (i.e. Wi-Fi or Ethernet cameras provisioning), you can skip this part if you don't have an AP5GC Mobile Network and packet core deployed into an ASE, RAN, 5G UEs, etc.

Install AKRI and related application discovery handler

Install HELM in your client machine

If you don't have HELM installed in your client system, you need to have it since AKRI-OSS is deployed through HELM.

Make sure you have install helm in the machine you are using with KUBECTL accessing the Kubernetes cluster. By default, that machine can be a Windows machine with KUBECTL installed and pointing to the right Kubernetes cluster context.

Full installation instructions of Helm can be found here.

Install AKRI

Install akri repository.

helm repo add akri-helm-charts https://project-akri.github.io/akri/

Update the HELM repo with this command:

helm repo update

Install AKRI with the application's "camera discovery handler" AKRI Helm chart and related configuration:

IMPORTANT: If you are installing agains a K3s or K8s (Kubernetes on Docker Desktop should be "k8s"), you need to specify that distro in the kubernetesDistro parameter below as:

--set kubernetesDistro=k8s

or

--set kubernetesDistro=k3s

In the "by default" cluster installed by Azure IoT Operations script, it installs AKS Edge Essentials with K3s under the covers, so if using that AKS-EE cluster, you should use k3s.

This is the whole command to run, but make sure of the kubernetesDistro parameter first depending on your target cluster:

 helm install akri akri-helm-charts/akri `
 -n mec-accelerator `
 --set kubernetesDistro=k3s `
 --set custom.discovery.enabled=true `
 --set custom.discovery.image.repository=mecsolutionaccelerator/akri-camera-discovery-handler `
 --set custom.discovery.image.tag=2.0 `
 --set custom.discovery.name=akri-camera-discovery `
 --set custom.configuration.enabled=true `
 --set custom.configuration.name=akri-camera `
 --set custom.configuration.discoveryHandlerName=camera `
 --set custom.configuration.discoveryDetails.connectionString="mongodb://control-plane-mongodb.mec-accelerator:27017" `
 --set custom.configuration.discoveryDetails.database="ControlPlane" `
 --set custom.configuration.discoveryDetails.collection="Cameras" `
 --set custom.configuration.brokerPod.image.repository=mecsolutionaccelerator/framesplitter `
 --set custom.configuration.brokerPod.image.tag=2.0

Here's an example execution when targeting k3s:

image

But if you are deploying to a different type of Kubernetes cluster (i.e. Docker-Desktop-Kuberentes or plain Kubernetes) then you should use k8s.

WARNING: If you get the following error, try runnning helm repo update and re-run the previous AKRI installation command:

Error: INSTALLATION FAILED: no cached repo found. (try 'helm repo update'): open C:\Users\cesardl\AppData\Local\Temp\helm\repository\akri-helm-charts-index.yaml: The system cannot find the file specified.

Deploy MQTT Broker resources to Kubernetes

Before continuing with the deployment, decide wether to use Azure IoT Operations E4K or Mosquitto MQTT broker.

You only need one or the other. Either Azure IoT MQ or Mosquitto. Of course, if you deployed Azure IoT Operations into your cluster, we recommend to use Azure IoT MQ.

Alternative A: Deploy Azure IoT MQ broker listener pod (aka. E4K)

  • Open a new command-shell and move into the deploy/k8s/E4K folder of this repo, as current folder of the command-shell:

    cd <your_path>/deploy/k8s/E4K
  • Deploy E4K broker listener and the required configurations by running this command:

    kubectl apply -f ./

Alternative B: Deploy Mosquitto pod

  • Open a new command-shell and move into the deploy/k8s/mosquitto folder of this repo, as current folder of the command-shell:

    cd <your_path>/deploy/k8s/mosquitto
  • Deploy mosquitto and the required configurations by running this command:

    kubectl apply -f ./

Deploy the rest of the application's services into the Kubernetes cluster

  • Open a new command-shell and move into the deploy/k8s folder of this repo, as current folder of the command-shell:

    cd <your_path>/deploy/k8s
  • Deploy the application in Kuberentes by running this command:

    kubectl apply -f ./

    All services will be created in the specified Kubernetes namespace "mec-accelerator" for this application, like in the following command execution:

    image

Deploy AKRI configuration

  • Open a new command-shell and move into the deploy/k8s folder of this repo, as current folder of the command-shell:
./deploy-akri-secrets.ps1

image

Check the application status with Kubernetes Dashboard

image

However, at this point you need to provide a security token to athenticate and enter the dashboard. Let's do that in the following steps.

Configure Bearer Tokens from Kuberentes to access the Dashboard

  • You need to do this step only one time: Change the current folder to the deploy/k8s/dashboard_auth folder of this repo and run the instructions in commands.txt to configure the tokens. Basically, the following commands need to be run just once from that folder:

    kubectl apply -f dashboard-adminuser.yaml 
    
    kubectl apply -f adminuser-cluster-role-binding.yaml

    image

Generate and copy the token to provide to Kubernetes Dashboard

  • Whenever you need a token, run this command:

    kubectl -n kubernetes-dashboard create token admin-user --duration=48h --output yaml

    Select and copy the token to the clipboard.

    image

Provide the token to Kubernetes dashboard

image

In the Kubernetes dashboard you should be able to explore the application's pods, services, etc.

image

Try the 'Control Plain app' to provision cameras

In order to know the URL for the 'Control Plain web app' (IP and port to use), type the following command:

kubectl get services --namespace mec-accelerator

image

Then search for the service with name "control-plane-ui-service" and related IP and external port, so you write a URL similar to the following in a browser:

http://<your-IP>:90/

image

You could also find out the IP and port by using the Kubernetes dashboard, select the Kubernetes namespace where the application is deployed ("mec-accelerator"), go to the "Services" menu in the left tab and click on the url to the right on the "control-plane-ui-service" service row.

image

Access the Alerts dashboard UI with Alerts originated from AI model detections

To access the front-end, In order to know the URL for the 'Alerts dashboard UI' front-end (IP and port to use), type the following command:

kubectl get services --namespace mec-accelerator

image

Then search for the service with name "alerts-ui" and related IP and external port, so you write a URL similar to the following in a browser:

http://<your-IP>:88/

Alternatively, you can also click on the link "Alerts Dashboard" from the Control-Plane app here:

image

As a third alternative and again using the K8s dashboard, you can also find out the real URL from the K8s dashboard, select the Kubernetes namespace where the application is deployed (MEC-Accelerator), go to the services menu in the left tab and click on the url to the right on the Alerts-UI service row:

image

Following any of those ways you should be able to run the 'Alerts dashboard UI' and check out the Alerts originated from the AI models when analyzing the video:

image

Expose Kubernetes services to external devices (Important if using AKS EE)

If you use K3s on an Ubuntu Linux VM, it's straightforward since the IP used for your Kubernetes services is usually the same IP used by the VM in the real IP network, so you should be able to access the application services (UI apps in this case) from remote client machines.

However, if you deployed the application into AKS edge Essentials, you might need to make Kubernetes services accessible to external devices so they can interact with the web applications you've deployed. Follow the next article that explains how to expose Kubernetes services running on an AKS Edge Essentials cluster to external devices:

AKS Edge Essentials - Expose Kubernetes services to external devices

Remove the MEC-Application from Kubernetes

  • From command-line (PowerShell or Bash) change the current folder to the deploy/k8s folder of this repo.

  • If Azure IoT Operations E4K broker has been used, run the following commands to remove kubernetes resouces:

    kubectl delete -f ./E4K/
  • If Mosquitto broker has been used, run the following commands to remove kubernetes resouces:

    kubectl delete -f ./mosquitto/
  • Run the following command to remove the rest of the kubernetes resources for this application, since there is no stop action on kubectl.

    kubectl delete -f ./