From a9d23974389c532f5a1c9288719a1517a30d6b61 Mon Sep 17 00:00:00 2001 From: guofei Date: Wed, 22 Mar 2023 14:59:32 -0700 Subject: [PATCH] Revise README.md --- README.md | 634 ++++-------------------------------------------------- 1 file changed, 36 insertions(+), 598 deletions(-) diff --git a/README.md b/README.md index 6ac6555e..f8c0de6e 100644 --- a/README.md +++ b/README.md @@ -5,40 +5,25 @@ [![Go Report Card](https://goreportcard.com/badge/github.com/virtual-kubelet/azure-aci)](https://goreportcard.com/report/github.com/virtual-kubelet/azure-aci) [![codecov](https://codecov.io/gh/virtual-kubelet/azure-aci/branch/master/graph/badge.svg?token=XHb1xbrki0)](https://codecov.io/gh/virtual-kubelet/azure-aci) -Azure Container Instances (ACI) provide a hosted environment for running containers in Azure. When using ACI, there is no need to manage the underlying compute infrastructure, Azure handles this management for you. When running containers in ACI, you are charged by the second for each running container. +Azure Container Instances (ACI) provides a hosted environment for running containers in Azure. When using ACI, there is no need to manage the underlying compute infrastructure since Azure handles this management overhead. When running containers in ACI, users are charged based on the container lifecycle in seconds. -The Azure Container Instances provider for the Virtual Kubelet configures an ACI instance as a node in any Kubernetes cluster. When using the Virtual Kubelet ACI provider, pods can be scheduled on an ACI instance as if the ACI instance is a standard Kubernetes node. This configuration allows you to take advantage of both the capabilities of Kubernetes and the management value and cost benefit of ACI. +The ACI provider for the Virtual Kubelet configures ACI service as a virtual node in a Kubernetes cluster. Hence, pods scheduled on the virtual node can be run on ACI instances. This configuration allows users to take advantage of both the capabilities of Kubernetes and the management value and cost benefit of ACI. -This document details configuring the Virtual Kubelet ACI provider. +## Features -## Table of Contents +Virtual Kubelet's ACI provider relies heavily on the feature set that ACI service provides. Please check the Azure documentations for region availability, pricing and new features. The list here presents a sample reference for the features that ACI provider supports. **Note**: Users should **NOT** expect feature parities between Virutal Kubelet and real Kubernetes Kubelet. -* [Feature set](#current-feature-set) -* [Prerequisites](#prerequisites) -* [Set-up virtual node in AKS](#cluster-and-azure-account-setup) -* [Helm set-up](#helm-set-up) -* [Create AKS cluster with a Virtual Network](#create-an-aks-cluster-with-vnet) -* [Validate the Virtual Kubelet ACI provider](#validate-the-virtual-kubelet-aci-provider) -* [Schedule a pod in ACI](#schedule-a-pod-in-aci) -* [Work around for the virtual kubelet pod](#Work-around-for-the-virtual-kubelet-pod) -* [Remove the Virtual Kubelet](#remove-the-virtual-kubelet) - -## Current feature set - -Virtual Kubelet's ACI provider relies heavily on the feature set that Azure Container Instances provide. Please check the Azure documentation accurate details on region availability, pricing and new features. The list here attempts to give an accurate reference for the features we support in ACI and the ACI provider within Virtual Kubelet. - -### Features +### Supported * Volumes: empty dir, github repo, projection, Azure Files, Azure Files CSI drivers * Secure env variables, config maps -* Bring your own virtual network (VNet) +* Virtual network integration (VNet) * Network security group support -* Basic Azure Networking support within AKS virtual node * [Exec support](https://docs.microsoft.com/azure/container-instances/container-instances-exec) for container instances * Azure Monitor integration ( aka OMS) * Support for init-containers ([use init containers](#Create-pod-with-init-containers)) -### Limitations +### Limitations (Not supported) * Using service principal credentials to pull ACR images ([see workaround](#Private-registry)) * Liveness and readiness probes @@ -46,408 +31,39 @@ Virtual Kubelet's ACI provider relies heavily on the feature set that Azure Cont * VNet peering * Argument support for exec * [Host aliases](https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/) support -* downward APIs (i.e podIP) - -## Prerequisites - -- [Microsoft Azure account](https://azure.microsoft.com/free/). -- [Helm](https://helm.sh/docs/intro/quickstart/#install-helm) -- [AKS](https://docs.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-cli) -- [Kubernetes CLI](#install-the-kubernetes-cli). - -## Cluster and Azure Account Setup - -Now that we have all the tools, we will set up your Azure account to work with ACI. - -### Configure your Azure account - -First let's identify your Azure subscription and save it for use later on in the quickstart. - -1. Run `az login` and follow the instructions in the command output to authorize `az` to use your account -2. List your Azure subscriptions: - - ```bash - az account list -o table - ``` - -3. Copy your subscription ID and save it in an environment variable: - - **Bash** - - ```bash - export AZURE_SUBSCRIPTION_ID="" - ``` - - **PowerShell** - - ```powershell - $env:AZURE_SUBSCRIPTION_ID = "" - ``` - -4. Enable ACI in your subscription: - - ```bash - az provider register -n Microsoft.ContainerInstance - ``` - -## Quick set up with AKS (Virtual Nodes) - -Azure Kubernetes Service has an efficient way of setting up virtual kubelet with the ACI provider with a feature called virtual node. You can easily install a virtual node that will deploy Linux workloads to ACI. The pods that spin out will automatically get private IPs and will be within a subnet that is within the AKS cluster's Virtual Network. **Virtual Nodes is the recommended path for using the ACI provider on Linux AKS clusters.** - -To install virtual node in the Azure portal go [here](https://docs.microsoft.com/azure/aks/virtual-nodes-portal). To install virtual node in the Azure CLI go [here](https://docs.microsoft.com/azure/aks/virtual-nodes-cli). - -## Helm set-up - -### Create a Resource Group for ACI - -To use Azure Container Instances, you must provide a resource group. Create one with the az cli using the following command. - -```bash -export ACI_REGION=eastus -az group create --name aci-group --location "$ACI_REGION" -export AZURE_RG=aci-group -``` - -### Create Azure identity - -You may choose to create either [MSI](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview-for-developers?tabs=portal%2Cdotnet#creating-a-managed-identity) or a service principal as following: - -This creates an identity for the Virtual Kubelet ACI provider to use when provisioning -resources on your account on behalf of Kubernetes. If you are provisioning Virtual Kubelet on AKS, please note: 1) This step is optional 2) The service principal will expire within a year unless [updated](https://docs.microsoft.com/azure/aks/update-credentials). - -1. Create a service principal with RBAC enabled for the quickstart: - - ```bash - az ad sp create-for-rbac --name virtual-kubelet-quickstart -o table - ``` - -2. Save the values from the command output in environment variables: - - **Bash** - - ```bash - export AZURE_TENANT_ID= - export AZURE_CLIENT_ID= - export AZURE_CLIENT_SECRET= - ``` - - **PowerShell** - - ```powershell - $env:AZURE_TENANT_ID = "" - $env:AZURE_CLIENT_ID = "" - $env:AZURE_CLIENT_SECRET = "" - ``` - -## Deployment of the ACI provider in your cluster - -1. Run these commands to deploy the virtual kubelet which connects your Kubernetes cluster to Azure Container Instances. - - ```bash - $ export RELEASE_TAG=1.5.1 - $ export RELEASE_NAME=virtual-kubelet-azure-aci - $ export VK_RELEASE=$RELEASE_NAME-$RELEASE_TAG - $ export NODE_NAME=virtual-kubelet - $ export CHART_URL=https://github.com/virtual-kubelet/azure-aci/raw/gh-pages/charts/$VK_RELEASE.tgz - ``` - -2. Grab the public master URI for your Kubernetes cluster and save the value. - - ```bash - export MASTER_URI="$(kubectl cluster-info | awk '/Kubernetes control plane/{print $7}' | sed "s,\x1B\[[0-9;]*[a-zA-Z],,g")" - ``` - - 3. Install Azure provider Helm chart - - 1. If your cluster is an AKS cluster: - - ```bash - # Linux Virtual Node - helm install "$RELEASE_NAME" "$CHART_URL" \ - --set providers.azure.targetAKS=true \ - --set providers.azure.masterUri=$MASTER_URI \ - --set nodeName=$NODE_NAME - - # Windows Virtual Node - helm install "$RELEASE_NAME" "$CHART_URL" \ - --set "nodeOsType=Windows" \ - --set providers.azure.targetAKS=true \ - --set providers.azure.masterUri=$MASTER_URI \ - --set nodeName="${NODE_NAME}-win" - ``` - - 2. For any other type of Kubernetes cluster: - - ```bash - # Linux Virtual Node - helm install "$RELEASE_NAME" "$CHART_URL" \ - --set rbac.install=true \ - --set providers.azure.targetAKS=false \ - --set providers.azure.aciResourceGroup=$AZURE_RG \ - --set providers.azure.aciRegion=$ACI_REGION \ - --set providers.azure.tenantId=$AZURE_TENANT_ID \ - --set providers.azure.subscriptionId=$AZURE_SUBSCRIPTION_ID \ - --set providers.azure.clientId=$AZURE_CLIENT_ID \ - --set providers.azure.clientKey=$AZURE_CLIENT_SECRET \ - --set providers.azure.masterUri=$MASTER_URI \ - --set nodeName=$NODE_NAME - - # Windows Virtual Node - helm install "$RELEASE_NAME" "$CHART_URL" \ - --set rbac.install=true \ - --set "nodeOsType=Windows" \ - --set providers.azure.targetAKS=false \ - --set providers.azure.aciResourceGroup=$AZURE_RG \ - --set providers.azure.aciRegion=$ACI_REGION \ - --set providers.azure.tenantId=$AZURE_TENANT_ID \ - --set providers.azure.subscriptionId=$AZURE_SUBSCRIPTION_ID \ - --set providers.azure.clientId=$AZURE_CLIENT_ID \ - --set providers.azure.clientKey=$AZURE_CLIENT_SECRET \ - --set providers.azure.masterUri=$MASTER_URI \ - --set nodeName="${NODE_NAME}-win" - ``` - - >Note: If your cluster has RBAC disabled set ```rbac.install=false``` - -
-Result - -```console -NAME: virtual-kubelet -LAST DEPLOYED: Thu Feb 15 13:17:01 2018 -NAMESPACE: default -STATUS: DEPLOYED - -RESOURCES: -==> v1/Secret -NAME TYPE DATA AGE -virtual-kubelet-virtual-kubelet Opaque 3 1s - -==> v1beta1/Deployment -NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE -virtual-kubelet-virtual-kubelet 1 1 1 0 1s - -==> v1/Pod(related) -NAME READY STATUS RESTARTS AGE -virtual-kubelet-virtual-kubelet-7bcf5dc749-6mvgp 0/1 ContainerCreating 0 1s - - -NOTES: -The virtual kubelet is getting deployed on your cluster. - -To verify that virtual kubelet has started, run: - -```bash - kubectl --namespace=default get pods -l "app=virtual-kubelet-virtual-kubelet" -``` -

-## Create an AKS cluster with VNet - - Run the following commands to create an AKS cluster with a new Azure virtual network. Also, create two subnets. One will be delegated to the cluster and the other will be delegated to Azure Container Instances. - -### Create an Azure virtual network and subnets - - First, set the following variables for your VNet range and two subnet ranges within that VNet. - -```bash - export VNET_RANGE=10.0.0.0/8 - export CLUSTER_SUBNET_RANGE=10.240.0.0/16 - export ACI_SUBNET_RANGE=10.241.0.0/16 - export VNET_NAME=myAKSVNet - export CLUSTER_SUBNET_NAME=myAKSSubnet - export ACI_SUBNET_NAME=myACISubnet - export AKS_CLUSTER_RG=myresourcegroup - export KUBE_DNS_IP=10.0.0.10 -``` - -Run the following command to create a virtual network within Azure, and a subnet within that VNet. The subnet will be dedicated to the nodes in the AKS cluster. - -```bash -az network vnet create \ ---resource-group $AKS_CLUSTER_RG \ ---name $VNET_NAME \ ---address-prefixes $VNET_RANGE \ ---subnet-name $CLUSTER_SUBNET_NAME \ ---subnet-prefix $CLUSTER_SUBNET_RANGE -``` - -Create a subnet that will be delegated to just resources within ACI, note that this needs to be an empty subnet, but within the same VNet that you already created. - -```bash -az network vnet subnet create \ - --resource-group $AKS_CLUSTER_RG \ - --vnet-name $VNET_NAME \ - --name $ACI_SUBNET_NAME \ - --address-prefix $ACI_SUBNET_RANGE -``` - -### Create a service principal (OPTIONAL) - -Create an Azure Active Directory service principal to allow AKS to interact with other Azure resources. You can use a pre-created service principal too. - -```bash -az ad sp create-for-rbac -n "virtual-kubelet-sp" --skip-assignment -``` - -The output should look similar to the following. - -
-Result - -```console -{ - "appId": "bef76eb3-d743-4a97-9534-03e9388811fc", - "displayName": "azure-cli-2018-08-29-22-29-29", - "name": "http://azure-cli-2018-08-29-22-29-29", - "password": "1d257915-8714-4ce7-xxxxxxxxxxxxx", - "tenant": "72f988bf-86f1-41af-91ab-2d7cd011db48" -} -``` -

- -Save the output values from the command output in environment variables. +* Downward APIs (i.e podIP) +* Projected volumes +* Potentially any new features introduced in real Kubelet since 1.24. -```bash -export AZURE_TENANT_ID= -export AZURE_CLIENT_ID= -export AZURE_CLIENT_SECRET= -``` +## Installation -These values can be integrated into the `az aks create` as a field `--service-principal $AZURE_CLIENT_ID`. +### Using Azure Portal for AKS clusters -### Integrating Azure VNet Resource +Please follow this offical [document ](https://learn.microsoft.com/en-us/azure/aks/virtual-nodes-portal) to install virtual node for an AKS cluster using Azure Portal. -If you want to integrate an already created Azure VNet resource with your AKS cluster than follow these steps. -Grab the virtual network resource id with the following command: - -```bash -az network vnet show --resource-group $AKS_CLUSTER_RG --name $VNET_NAME --query id -o tsv -``` +### Using Azure CLI for AKS clusters -Grant access to the AKS cluster to use the virtual network by creating a role and assigning it. +Please follow this offical [document ](https://learn.microsoft.com/en-us/azure/aks/virtual-nodes-cli) to install virtual node for an AKS cluster using Azure CLI. -```bash -az role assignment create --assignee $AZURE_CLIENT_ID --scope --role "Network Contributor" -``` +### Deploy Virtual Kubelet manually using Helm -### Create an AKS cluster with a virtual network +If you want to install a specific version of Virtual Kubelet in a Kubernetes cluster in Azure manually, please follow the documents ([downgrade](./docs/DOWNGRADE-README.md) and [upgrade](./docs/UPGRADE-README.md)). -Grab the id of the cluster subnet you created earlier with the following command. -```bash -az network vnet subnet show --resource-group $AKS_CLUSTER_RG --vnet-name $VNET_NAME --name $CLUSTER_SUBNET_NAME --query id -o tsv -``` +## Demo -Save the entire output starting with "/subscriptions/..." in the following environment variable. - -```bash -export VNET_SUBNET_ID= -``` - -Use the following command to create an AKS cluster with the virtual network you've already created. - -```bash -az aks create \ - --resource-group myResourceGroup \ - --name myAKSCluster \ - --node-count 1 \ - --network-plugin azure \ - --service-cidr 10.0.0.0/16 \ - --dns-service-ip $KUBE_DNS_IP \ - --docker-bridge-address 172.17.0.1/16 \ - --vnet-subnet-id $VNET_SUBNET_ID \ - --client-secret $AZURE_CLIENT_SECRET -``` - -### Deploy Virtual Kubelet - -Manually deploy the Virtual Kubelet, the following env. variables have already been set earlier. You do need to pass through the subnet you created for ACI earlier, otherwise the container instances will not be able to participate with the other pods within the cluster subnet. - -Grab the public master URI for your Kubernetes cluster and save the value. - -```bash -kubectl cluster-info -export MASTER_URI= -``` - -Set the following values for the helm chart. - -```bash -$ export RELEASE_TAG=1.5.1 -$ export RELEASE_NAME=virtual-kubelet-azure-aci -$ export VK_RELEASE=$RELEASE_NAME-$RELEASE_TAG -$ export NODE_NAME=virtual-kubelet -$ export CHART_URL=https://github.com/virtual-kubelet/azure-aci/raw/gh-pages/charts/$VK_RELEASE.tgz -``` - -If your cluster is an AKS cluster: - -```bash -helm install "$RELEASE_NAME" "$CHART_URL" \ - --set providers.azure.targetAKS=true \ - --set providers.azure.vnet.enabled=true \ - --set providers.azure.vnet.subnetName=$ACI_SUBNET_NAME \ - --set providers.azure.vent.subnetCidr=$ACI_SUBNET_RANGE \ - --set providers.azure.vnet.clusterCidr=$CLUSTER_SUBNET_RANGE \ - --set providers.azure.vnet.kubeDnsIp=$KUBE_DNS_IP \ - --set providers.azure.masterUri=$MASTER_URI \ - --set nodeName=$NODE_NAME -``` - -For any other type of cluster: - -```bash -# the resource group where the virtual network is located in -export ACI_VNET_RESOURCE_GROUP= - -# the virtual network name where container will deploy to -export ACI_VNET_NAME= -# subnet name where ACI will deploy to. Virtual Kubelet will automatically create subnet resource if it not exists -export ACI_SUBNET_NAME= -# subnet's IP range, for example 10.1.0.0/16. You don't need specific this system variable if subnet has been exists -export ACI_SUBNET_RANGE= - -helm install "$RELEASE_NAME" "$CHART_URL" \ - --set providers.azure.targetAKS=false \ - --set providers.azure.vnet.enabled=true \ - --set providers.azure.vnet.vnetResourceGroup=$ACI_VNET_RESOURCE_GROUP \ - --set providers.azure.vnet.vnetName=$ACI_VNET_NAME \ - --set providers.azure.vnet.subnetName=$ACI_SUBNET_NAME \ - --set providers.azure.vent.subnetCidr=$ACI_SUBNET_RANGE \ - --set providers.azure.vnet.kubeDnsIp=$KUBE_DNS_IP \ - --set providers.azure.tenantId=$AZURE_TENANT_ID \ - --set providers.azure.subscriptionId=$AZURE_SUBSCRIPTION_ID \ - --set providers.azure.aciResourceGroup=$AZURE_RG \ - --set providers.azure.aciRegion=$ACI_REGION \ - --set providers.azure.masterUri=$MASTER_URI \ - --set providers.azure.clientId=$AZURE_CLIENT_ID \ - --set providers.azure.clientKey=$AZURE_CLIENT_SECRET - - ``` - -## Validate the Virtual Kubelet ACI provider - -To validate that the Virtual Kubelet has been installed, return a list of Kubernetes nodes using the [kubectl get nodes][kubectl-get] command. +To validate that the Virtual Kubelet has been installed, check and find the virtual node in the Kubernetes cluster. ```bash kubectl get nodes -``` - -
-Result - -```console NAME STATUS ROLES AGE VERSION virtual-kubelet-aci-linux Ready agent 2m v1.13.1 -aks-nodepool1-39289454-0 Ready agent 22h v1.12.6 -aks-nodepool1-39289454-1 Ready agent 22h v1.12.6 -aks-nodepool1-39289454-2 Ready agent 22h v1.12.6 +aks-nodepool1-XXXXXXXX-0 Ready agent 22h v1.12.6 +aks-nodepool1-XXXXXXXX-1 Ready agent 22h v1.12.6 +aks-nodepool1-XXXXXXXX-2 Ready agent 22h v1.12.6 ``` -

- -## Schedule a pod in ACI -Create a file named `virtual-kubelet-test.yaml` and copy in the following YAML. +Create a test Pod `virtual-kubelet-test.yaml`. ```yaml apiVersion: v1 @@ -481,7 +97,7 @@ spec: effect: NoSchedule ``` -Notice that Virtual-Kubelet nodes are tainted by default to avoid unexpected pods running on them, i.e. kube-proxy, other virtual-kubelet pods, etc. To schedule a pod to them, you need to add the toleration to the pod spec and a node selector: +Note that virtual nodes are tainted by default to avoid unexpected pods running on them, i.e., `kube-proxy`. To schedule a pod to them, you need to add the toleration to the pod spec and a node selector: ```yaml ... @@ -496,9 +112,7 @@ Notice that Virtual-Kubelet nodes are tainted by default to avoid unexpected pod effect: NoSchedule ``` -### Private registry - -If your image is on a private registry, you need to [add a kubernetes secret to your cluster](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-by-providing-credentials-on-the-command-line) and reference it in the pod spec. +If your image is in a private registry, you need to [add a kubernetes secret to your cluster](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-by-providing-credentials-on-the-command-line) and reference it in the pod spec. ```yaml ... @@ -512,31 +126,25 @@ If your image is on a private registry, you need to [add a kubernetes secret to - name: ``` -Run the application with the [kubectl create][kubectl-create] command. - -```bash -kubectl create -f virtual-kubelet-test.yaml -``` - -Use the [kubectl get pods][kubectl-get] command with the `-o wide` argument to output a list of pods with the scheduled node. +Run the application. ```bash -kubectl get pods -o wide +kubectl apply -f virtual-kubelet-test.yaml ``` -Notice that the `helloworld` pod is running on the `virtual-kubelet` node. +Note that the `helloworld` pod is running on the `virtual-kubelet` node. ```console NAME READY STATUS RESTARTS AGE IP NODE -aci-helloworld-2559879000-8vmjw 1/1 Running 0 39s 52.179.3.180 virtual-kubelet-aci-linux +aci-helloworld-2559879000-XXXXXX 1/1 Running 0 39s 52.179.XXX.XXX virtual-kubelet-aci-linux ``` -If the AKS cluster was configured with a virtual network, then the output will look like the following. The container instance will get a private ip rather than a public one. +If the AKS cluster was configured with a virtual network, then the output will look like the following. The container instance will get a private IP rather than a public one. ```console NAME READY STATUS RESTARTS AGE IP NODE -aci-helloworld-9b55975f-bnmfl 1/1 Running 0 4m 10.241.0.4 virtual-kubelet-aci-linux +aci-helloworld-9b55975f-XXXXX 1/1 Running 0 4m 10.241.XXX.XXX virtual-kubelet-aci-linux ``` To validate that the container is running in an Azure Container Instance, use the [az container list][az-container-list] Azure CLI command. @@ -551,191 +159,21 @@ az container list -o table ```console Name ResourceGroup ProvisioningState Image IP:ports CPU/Memory OsType Location ------------------------------- --------------- ------------------- ------------------------ --------------- --------------- -------- ---------- -helloworld-2559879000-8vmjw myResourceGroup Succeeded microsoft/aci-helloworld 52.179.3.180:80 1.0 core/1.5 gb Linux eastus +helloworld-2559879000-XXXXXX myResourceGroup Succeeded microsoft/aci-helloworld 52.179.XXX.XXX:80 1.0 core/1.5 gb Linux eastus ```
- - -### Create pod with init containers -Multiple init containers can be specified in the podspec similar to how containers are specified - -```yaml -spec: - initContainers: - - image: - name: init-container-01 - command: [ "/bin/sh" ] - args: [ "-c", "echo \"Hi\"" ] - - image: - name: init-container-02 - command: [ "/bin/sh" ] - args: [ "-c", "echo \"Hi\"" ] - containers: - - image: - imagePullPolicy: Always - name: container - command: [ "/bin/sh" ] -``` -More information on init containers can be found in [Kubernetes](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) and [ACI](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-init-container) documentations - -## Work around for the virtual kubelet pod - -If your pod that's scheduled onto the Virtual Kubelet node is in a pending state please add this workaround to your Virtual Kubelet pod spec. - -First, grab the logs from your virtual kubelet pod, with the following command. You can get the pod name from the `kubectl get pods` command +For manual installation, you can remove the virtual node by deleting the Helm deployment. Run the following command: ```bash -kubectl logs virtual-kubelet-virtual-kubelet-7bcf5dc749-6mvgp +helm uninstall $ChartName ``` -### Stream or pod watcher errors - -If you see the following errors in the logs: - -```console -ERROR: logging before flag.Parse: E0914 00:02:01.546132 1 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 181; INTERNAL_ERROR -time="2018-09-14T00:02:01Z" level=error msg="Pod watcher connection is closed unexpectedly" namespace= node=virtual-kubelet-myconnector-linux operatingSystem=Linux provider=azure -``` - -Then copy the master URI with cluster-info. - -```bash -kubectl cluster-info -``` - -Edit virtual kubelet deployment by first getting the deployment name. - -```bash -kubectl get deploy -``` - -
-Result - -```console -NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE -virtual-kubelet-virtual-kubelet 1 1 1 1 5d -aci-helloworld 1 1 1 0 12m -``` -

- -Edit the deployment. - -```bash -kubectl edit deploy virtual-kubelet-virtual-kubelet -``` - -Add the following name and value to the deployment in the environment section. Use your copied AKS master URI. - -```yaml - - ... -- name: MASTER_URI - value: https://aksxxxx-xxxxx-xxxx-xxxxxxx.hcp.uksouth.azmk8s.io:443 - - ``` - -### Taint deprecated errors - -If you see the following errors in the logs: - -```console -Flag --taint has been deprecated, Taint key should now be configured using the VK_TAINT_KEY environment variable -``` - -Then edit your virtual kubelet deployment by first grabbing the deployment name. - -```bash -kubectl get deploy -``` - -
-Result - -```console -NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE -virtual-kubelet-virtual-kubelet 1 1 1 1 5d -aci-helloworld 1 1 1 0 12m -``` -

- -Edit the virtual kubelet deployment. - -```bash -kubectl edit deploy virtual-kubelet-virtual-kubelet -``` - -Add the following as an environment variable within the deployment. - -```yaml -... -- name: VK_TAINT_KEY - value: azure.com/aci - -``` - -Also, delete the following argument in your pod spec: - -```yaml -- --taint - - azure.com/aci -``` - -## Remove the Virtual Kubelet - -You can remove your Virtual Kubelet node by deleting the Helm deployment. Run the following command: - -```bash -helm uninstall virtual-kubelet -``` +If it is an AKS managed virtual node, please follow the steps [here](https://docs.microsoft.com/azure/aks/virtual-nodes-cli#remove-virtual-nodes) to remove the add-on. -If you used Virtual Nodes, can follow the steps [here](https://docs.microsoft.com/azure/aks/virtual-nodes-cli#remove-virtual-nodes) to remove the add-on -[kubectl-create]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#create -[kubectl-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get [az-container-list]: https://docs.microsoft.com/cli/azure/container?view=azure-cli-latest#az_container_list -[az-container-show]: https://docs.microsoft.com/cli/azure/container?view=azure-cli-latest#az_container_show