title | description | author | ms.author | ms.service | ms.topic | ms.date | ms.custom | keywords |
---|---|---|---|---|---|---|---|---|
Quickstart: Connect an existing Kubernetes cluster to Azure Arc |
In this quickstart, learn how to connect an Azure Arc enabled Kubernetes cluster. |
mgoedtel |
magoedte |
azure-arc |
quickstart |
06/30/2021 |
template-quickstart, references_regions, devx-track-azurecli, devx-track-azurepowershell |
Kubernetes, Arc, Azure, cluster |
In this quickstart, you'll learn the benefits of Azure Arc enabled Kubernetes and how to connect an existing Kubernetes cluster to Azure Arc. For a conceptual look at connecting clusters to Azure Arc, see the Azure Arc enabled Kubernetes Agent Architecture article.
[!INCLUDE quickstarts-free-trial-note]
[!INCLUDE azure-cli-prepare-your-environment-no-header.md]
-
An up-and-running Kubernetes cluster. If you don't have one, you can create a cluster using one of these options:
-
Self-managed Kubernetes cluster using Cluster API
-
If you want to connect a OpenShift cluster to Azure Arc, you need to execute the following command just once on your cluster before running
az connectedk8s connect
:oc adm policy add-scc-to-user privileged system:serviceaccount:azure-arc:azure-arc-kube-aad-proxy-sa
[!NOTE] The cluster needs to have at least one node of operating system and architecture type
linux/amd64
. Clusters with onlylinux/arm64
nodes aren't yet supported. -
A
kubeconfig
file and context pointing to your cluster. -
'Read' and 'Write' permissions on the Azure Arc enabled Kubernetes resource type (
Microsoft.Kubernetes/connectedClusters
). -
Install the latest release of Helm 3.
-
Install or upgrade Azure CLI to version >= 2.16.0
-
Install the
connectedk8s
Azure CLI extension of version >= 1.0.0:az extension add --name connectedk8s
Note
For custom locations on your cluster, use East US or West Europe regions. For all other Azure Arc enabled Kubernetes features, select any region from this list.
[!INCLUDE azure-powershell-requirements-no-header.md]
Important
While the Az.ConnectedKubernetes PowerShell module is in preview, you must install it separately using
the Install-Module
cmdlet.
Install-Module -Name Az.ConnectedKubernetes
-
An up-and-running Kubernetes cluster. If you don't have one, you can create a cluster using one of these options:
-
Self-managed Kubernetes cluster using Cluster API
-
If you want to connect a OpenShift cluster to Azure Arc, you need to execute the following command just once on your cluster before running
New-AzConnectedKubernetes
:oc adm policy add-scc-to-user privileged system:serviceaccount:azure-arc:azure-arc-kube-aad-proxy-sa
[!NOTE] The cluster needs to have at least one node of operating system and architecture type
linux/amd64
. Clusters with onlylinux/arm64
nodes aren't yet supported. -
A
kubeconfig
file and context pointing to your cluster. -
'Read' and 'Write' permissions on the Azure Arc enabled Kubernetes resource type (
Microsoft.Kubernetes/connectedClusters
). -
Install the latest release of Helm 3.
Note
For custom locations on your cluster, use East US or West Europe regions. For all other Azure Arc enabled Kubernetes features, select any region from this list.
Important
Azure Arc agents require both of the following protocols/ports/outbound URLs to function:
- TCP on port 443:
https://:443
Endpoint (DNS) | Description |
---|---|
https://management.azure.com (for Azure Cloud), https://management.usgovcloudapi.net (for Azure US Government) |
Required for the agent to connect to Azure and register the cluster. |
https://<region>.dp.kubernetesconfiguration.azure.com (for Azure Cloud), https://<region>.dp.kubernetesconfiguration.azure.us (for Azure US Government) |
Data plane endpoint for the agent to push status and fetch configuration information. |
https://login.microsoftonline.com (for Azure Cloud), https://login.microsoftonline.us (for Azure US Government) |
Required to fetch and update Azure Resource Manager tokens. |
https://mcr.microsoft.com |
Required to pull container images for Azure Arc agents. |
https://gbl.his.arc.azure.com |
Required to get the regional endpoint for pulling system-assigned Managed Service Identity (MSI) certificates. |
https://<region-code>.his.arc.azure.com (for Azure Cloud), https://usgv.his.arc.azure.us (for Azure US Government) |
Required to pull system-assigned Managed Service Identity (MSI) certificates. <region-code> mapping for Azure cloud regions: eus (East US), weu (West Europe), wcus (West Central US), scus (South Central US), sea (South East Asia), uks (UK South), wus2 (West US 2), ae (Australia East), eus2 (East US 2), ne (North Europe), fc (France Central). |
- Enter the following commands:
az provider register --namespace Microsoft.Kubernetes az provider register --namespace Microsoft.KubernetesConfiguration az provider register --namespace Microsoft.ExtendedLocation
- Monitor the registration process. Registration may take up to 10 minutes.
az provider show -n Microsoft.Kubernetes -o table az provider show -n Microsoft.KubernetesConfiguration -o table az provider show -n Microsoft.ExtendedLocation -o table
- Enter the following commands:
Register-AzResourceProvider -ProviderNamespace Microsoft.Kubernetes Register-AzResourceProvider -ProviderNamespace Microsoft.KubernetesConfiguration Register-AzResourceProvider -ProviderNamespace Microsoft.ExtendedLocation
- Monitor the registration process. Registration may take up to 10 minutes.
Get-AzResourceProvider -ProviderNamespace Microsoft.Kubernetes Get-AzResourceProvider -ProviderNamespace Microsoft.KubernetesConfiguration Get-AzResourceProvider -ProviderNamespace Microsoft.ExtendedLocation
Run the following command:
az group create --name AzureArcTest --location EastUS --output table
Output:
Location Name ---------- ------------ eastus AzureArcTest
New-AzResourceGroup -Name AzureArcTest -Location EastUS
Output:
ResourceGroupName : AzureArcTest Location : eastus ProvisioningState : Succeeded Tags : ResourceId : /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/AzureArcTest
Run the following command:
az connectedk8s connect --name AzureArcTest1 --resource-group AzureArcTest
Output:
Helm release deployment succeeded { "aadProfile": { "clientAppId": "", "serverAppId": "", "tenantId": "" }, "agentPublicKeyCertificate": "xxxxxxxxxxxxxxxxxxx", "agentVersion": null, "connectivityStatus": "Connecting", "distribution": "gke", "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/AzureArcTest/providers/Microsoft.Kubernetes/connectedClusters/AzureArcTest1", "identity": { "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", "type": "SystemAssigned" }, "infrastructure": "gcp", "kubernetesVersion": null, "lastConnectivityTime": null, "location": "eastus", "managedIdentityCertificateExpirationTime": null, "name": "AzureArcTest1", "offering": null, "provisioningState": "Succeeded", "resourceGroup": "AzureArcTest", "tags": {}, "totalCoreCount": null, "totalNodeCount": null, "type": "Microsoft.Kubernetes/connectedClusters" }
Tip
The above command without the location parameter specified creates the Azure Arc enabled Kubernetes resource in the same location as the resource group. To create the Azure Arc enabled Kubernetes resource in a different location, specify either --location <region>
or -l <region>
when running the az connectedk8s connect
command.
Note
If you are logged into Azure CLI using a service principal, an additional parameter needs to be set for enabling the custom location feature on the cluster.
New-AzConnectedKubernetes -ClusterName AzureArcTest1 -ResourceGroupName AzureArcTest -Location eastus
Output:
Location Name Type -------- ---- ---- eastus AzureArcTest1 microsoft.kubernetes/connectedclusters
Run the following command:
az connectedk8s list --resource-group AzureArcTest --output table
Output:
Name Location ResourceGroup ------------- ---------- --------------- AzureArcTest1 eastus AzureArcTest
Get-AzConnectedKubernetes -ResourceGroupName AzureArcTest
Output:
Location Name Type -------- ---- ---- eastus AzureArcTest1 microsoft.kubernetes/connectedclusters
Note
After onboarding the cluster, it takes around 5 to 10 minutes for the cluster metadata (cluster version, agent version, number of nodes, etc.) to surface on the overview page of the Azure Arc enabled Kubernetes resource in Azure portal.
If your cluster is behind an outbound proxy server, Azure CLI and the Azure Arc enabled Kubernetes agents need to route their requests via the outbound proxy server.
-
Set the environment variables needed for Azure CLI to use the outbound proxy server:
-
If you are using bash, run the following command with appropriate values:
export HTTP_PROXY=<proxy-server-ip-address>:<port> export HTTPS_PROXY=<proxy-server-ip-address>:<port> export NO_PROXY=<cluster-apiserver-ip-address>:<port>
-
If you are using PowerShell, run the following command with appropriate values:
$Env:HTTP_PROXY = "<proxy-server-ip-address>:<port>" $Env:HTTPS_PROXY = "<proxy-server-ip-address>:<port>" $Env:NO_PROXY = "<cluster-apiserver-ip-address>:<port>"
-
-
Run the connect command with proxy parameters specified:
az connectedk8s connect --name <cluster-name> --resource-group <resource-group> --proxy-https https://<proxy-server-ip-address>:<port> --proxy-http http://<proxy-server-ip-address>:<port> --proxy-skip-range <excludedIP>,<excludedCIDR> --proxy-cert <path-to-cert-file>
Note
- Specify
excludedCIDR
under--proxy-skip-range
to ensure in-cluster communication is not broken for the agents. --proxy-http
,--proxy-https
, and--proxy-skip-range
are expected for most outbound proxy environments.--proxy-cert
is only required if you need to inject trusted certificates expected by proxy into the trusted certificate store of agent pods.
If your cluster is behind an outbound proxy server, Azure PowerShell and the Azure Arc enabled Kubernetes agents need to route their requests via the outbound proxy server.
-
Set the environment variables needed for Azure PowerShell to use the outbound proxy server:
-
Run the following command with appropriate values:
$Env:HTTP_PROXY = "<proxy-server-ip-address>:<port>" $Env:HTTPS_PROXY = "<proxy-server-ip-address>:<port>" $Env:NO_PROXY = "<cluster-apiserver-ip-address>:<port>"
-
-
Run the connect command with the proxy parameter specified:
New-AzConnectedKubernetes -ClusterName <cluster-name> -ResourceGroupName <resource-group> -Location eastus -Proxy 'https://<proxy-server-ip-address>:<port>'
Azure Arc enabled Kubernetes deploys a few operators into the azure-arc
namespace.
-
View these deployments and pods using:
kubectl get deployments,pods -n azure-arc
-
Verify all pods are in a
Running
state.Output:
NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cluster-metadata-operator 1/1 1 1 16h deployment.apps/clusteridentityoperator 1/1 1 1 16h deployment.apps/config-agent 1/1 1 1 16h deployment.apps/controller-manager 1/1 1 1 16h deployment.apps/flux-logs-agent 1/1 1 1 16h deployment.apps/metrics-agent 1/1 1 1 16h deployment.apps/resource-sync-agent 1/1 1 1 16h NAME READY STATUS RESTART AGE pod/cluster-metadata-operator-7fb54d9986-g785b 2/2 Running 0 16h pod/clusteridentityoperator-6d6678ffd4-tx8hr 3/3 Running 0 16h pod/config-agent-544c4669f9-4th92 3/3 Running 0 16h pod/controller-manager-fddf5c766-ftd96 3/3 Running 0 16h pod/flux-logs-agent-7c489f57f4-mwqqv 2/2 Running 0 16h pod/metrics-agent-58b765c8db-n5l7k 2/2 Running 0 16h pod/resource-sync-agent-5cf85976c7-522p5 3/3 Running 0 16h
You can delete the Azure Arc enabled Kubernetes resource, any associated configuration resources, and any agents running on the cluster using Azure CLI using the following command:
az connectedk8s delete --name AzureArcTest1 --resource-group AzureArcTest
Note
Deleting the Azure Arc enabled Kubernetes resource using Azure portal removes any associated configuration resources, but does not remove any agents running on the cluster. Best practice is to delete the Azure Arc enabled Kubernetes resource using az connectedk8s delete
instead of Azure portal.
You can delete the Azure Arc enabled Kubernetes resource, any associated configuration resources, and any agents running on the cluster using Azure PowerShell using the following command:
Remove-AzConnectedKubernetes -ClusterName AzureArcTest1 -ResourceGroupName AzureArcTest
Note
Deleting the Azure Arc enabled Kubernetes resource using Azure portal removes any associated configuration resources, but does not remove any agents running on the cluster. Best practice is to delete the Azure Arc enabled Kubernetes resource using Remove-AzConnectedKubernetes
instead of Azure portal.
Advance to the next article to learn how to deploy configurations to your connected Kubernetes cluster using GitOps.
[!div class="nextstepaction"] Deploy configurations using GitOps