Add documentation for Azure availability zones #57
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,218 @@ | ||
| # Availability Zones | ||
|
|
||
| **Feature Status:** Alpha since v1.12. | ||
|
|
||
| Kubernetes v1.12 adds support for [Azure availability zones (AZ)](https://azure.microsoft.com/en-us/global-infrastructure/availability-zones/). Nodes in availability zone will be added with label `failure-domain.beta.kubernetes.io/zone=<region>-<AZ>` and topology-aware provisioning is added for Azure managed disks storage class. | ||
|
|
||
| **TOC:** | ||
|
|
||
| <!-- TOC --> | ||
|
|
||
| - [Availability Zones](#availability-zones) | ||
| - [Pre-requirements](#pre-requirements) | ||
| - [Node labels](#node-labels) | ||
| - [Load Balancer](#load-balancer) | ||
| - [Managed Disks](#managed-disks) | ||
| - [StorageClass examples](#storageclass-examples) | ||
| - [PV examples](#pv-examples) | ||
| - [Appendix](#appendix) | ||
| - [Reference](#reference) | ||
|
|
||
| <!-- /TOC --> | ||
|
|
||
| ## Pre-requirements | ||
|
|
||
| Because only standard load balancer is supported with AZ, it is a prerequisite to enable AZ for the cluster. It should be configured in Azure cloud provider configure file (e.g. `/etc/kubernetes/azure.json`): | ||
|
|
||
| ```json | ||
| { | ||
| "loadBalancerSku": "standard", | ||
| ... | ||
| } | ||
| ``` | ||
|
|
||
| If topology-aware provisioning feature is used, feature gate `VolumeScheduling` should be enabled on master components (e.g. kube-apiserver, kube-controller-manager and kube-scheduler). | ||
|
|
||
| ## Node labels | ||
|
|
||
| Both zoned and unzoned nodes are supported, but the value of node label `failure-domain.beta.kubernetes.io/zone` are different: | ||
|
|
||
| - For zoned nodes, the value is `<region>-<AZ>`, e.g. `centralus-1`. | ||
| - For unzoned nodes, the value is faultDomain, e.g. `0`. | ||
|
|
||
| e.g. | ||
|
|
||
| ```yaml | ||
| $ kubectl get nodes --show-labels | ||
| NAME STATUS AGE VERSION LABELS | ||
| kubernetes-node12 Ready 6m v1.11 failure-domain.beta.kubernetes.io/region=centralus,failure-domain.beta.kubernetes.io/zone=centralus-1,... | ||
| ``` | ||
|
|
||
| ## Load Balancer | ||
|
|
||
| `loadBalancerSku` has been set to `standard` in cloud provider configure file, so standard load balancer and standard public IPs will be provisioned automatically for services with type `LoadBalancer`. Both load balancer and public IPs are zone redundant. | ||
|
|
||
| ## Managed Disks | ||
|
|
||
| Zone-aware and topology-aware provisioning are supported for Azure managed disks. To support these features, a few options are added in AzureDisk storage class: | ||
|
|
||
| - **zoned**: indicates whether new disks are provisioned with AZ. Default is true. | ||
|
There was a problem hiding this comment. Will this cause backwards compatibility issues for users that have StorageClasses before this change? There was a problem hiding this comment. No, azure supports both zoned and unzoned nodes together. If there's no zoned nodes in the cluster, then the disks will still be provisioned without zone. |
||
| - **allowedTopologies**: indicates which topologies are allowed for topology-aware provisioning. Only can be set if `zoned` is not false. | ||
|
|
||
| ### StorageClass examples | ||
|
|
||
| An example of zone-aware provisioning storage class is: | ||
|
|
||
| ```yaml | ||
| apiVersion: storage.k8s.io/v1 | ||
| kind: StorageClass | ||
| metadata: | ||
| annotations: | ||
| labels: | ||
| kubernetes.io/cluster-service: "true" | ||
| name: managed-premium | ||
| parameters: | ||
| kind: Managed | ||
| storageaccounttype: Premium_LRS | ||
| zoned: "true" | ||
| provisioner: kubernetes.io/azure-disk | ||
| ``` | ||
|
|
||
| Another example of topology-aware provisioning storage class is: | ||
|
|
||
| ```yaml | ||
| apiVersion: storage.k8s.io/v1 | ||
| kind: StorageClass | ||
| metadata: | ||
| annotations: | ||
| labels: | ||
| kubernetes.io/cluster-service: "true" | ||
| name: managed-premium | ||
| parameters: | ||
| kind: Managed | ||
| storageaccounttype: Premium_LRS | ||
| provisioner: kubernetes.io/azure-disk | ||
| allowedTopologies: | ||
| - matchLabelExpressions: | ||
| - key: failure-domain.beta.kubernetes.io/zone | ||
| values: | ||
| - centralus-1 | ||
| - centralus-2 | ||
| ``` | ||
|
|
||
| ### PV examples | ||
|
|
||
| When feature gate `VolumeScheduling` disabled, no `NodeAffinity` set for zoned PV: | ||
|
|
||
| ```sh | ||
| $ kubectl describe pv | ||
| Name: pvc-d30dad05-9ad8-11e8-94f2-000d3a07de8c | ||
| Labels: failure-domain.beta.kubernetes.io/region=southeastasia | ||
| failure-domain.beta.kubernetes.io/zone=southeastasia-2 | ||
| Annotations: pv.kubernetes.io/bound-by-controller=yes | ||
| pv.kubernetes.io/provisioned-by=kubernetes.io/azure-disk | ||
| volumehelper.VolumeDynamicallyCreatedByKey=azure-disk-dynamic-provisioner | ||
| Finalizers: [kubernetes.io/pv-protection] | ||
| StorageClass: default | ||
| Status: Bound | ||
| Claim: default/pvc-azuredisk | ||
| Reclaim Policy: Delete | ||
| Access Modes: RWO | ||
| Capacity: 5Gi | ||
| Node Affinity: | ||
| Required Terms: | ||
| Term 0: failure-domain.beta.kubernetes.io/region in [southeastasia] | ||
| failure-domain.beta.kubernetes.io/zone in [southeastasia-2] | ||
| Message: | ||
| Source: | ||
| Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod) | ||
| DiskName: k8s-5b3d7b8f-dynamic-pvc-d30dad05-9ad8-11e8-94f2-000d3a07de8c | ||
| DiskURI: /subscriptions/<subscription-id>/resourceGroups/<rg-name>/providers/Microsoft.Compute/disks/k8s-5b3d7b8f-dynamic-pvc-d30dad05-9ad8-11e8-94f2-000d3a07de8c | ||
| Kind: Managed | ||
| FSType: | ||
| CachingMode: None | ||
| ReadOnly: false | ||
| Events: <none> | ||
| ``` | ||
|
|
||
| When feature gate `VolumeScheduling` enabled, `NodeAffinity` will be populated for zoned PV: | ||
|
|
||
| ```sh | ||
| $ kubectl describe pv | ||
| Name: pvc-0284337b-9ada-11e8-a7f6-000d3a07de8c | ||
| Labels: failure-domain.beta.kubernetes.io/region=southeastasia | ||
| failure-domain.beta.kubernetes.io/zone=southeastasia-2 | ||
| Annotations: pv.kubernetes.io/bound-by-controller=yes | ||
| pv.kubernetes.io/provisioned-by=kubernetes.io/azure-disk | ||
| volumehelper.VolumeDynamicallyCreatedByKey=azure-disk-dynamic-provisioner | ||
| Finalizers: [kubernetes.io/pv-protection] | ||
| StorageClass: default | ||
| Status: Bound | ||
| Claim: default/pvc-azuredisk | ||
| Reclaim Policy: Delete | ||
| Access Modes: RWO | ||
| Capacity: 5Gi | ||
| Node Affinity: | ||
| Required Terms: | ||
| Term 0: failure-domain.beta.kubernetes.io/region in [southeastasia] | ||
| failure-domain.beta.kubernetes.io/zone in [southeastasia-2] | ||
| Message: | ||
| Source: | ||
| Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod) | ||
| DiskName: k8s-5b3d7b8f-dynamic-pvc-0284337b-9ada-11e8-a7f6-000d3a07de8c | ||
| DiskURI: /subscriptions/<subscription-id>/resourceGroups/<rg-name>/providers/Microsoft.Compute/disks/k8s-5b3d7b8f-dynamic-pvc-0284337b-9ada-11e8-a7f6-000d3a07de8c | ||
| Kind: Managed | ||
| FSType: | ||
| CachingMode: None | ||
| ReadOnly: false | ||
| Events: <none> | ||
| ``` | ||
|
|
||
| While unzoned disks are not able to attach in zoned nodes, `NodeAffinity` will also be set for them so that they will only be scheduled to unzoned nodes: | ||
|
|
||
| ```sh | ||
| $ kubectl describe pv pvc-bdf93a67-9c45-11e8-ba6f-000d3a07de8c | ||
| Name: pvc-bdf93a67-9c45-11e8-ba6f-000d3a07de8c | ||
| Labels: <none> | ||
| Annotations: pv.kubernetes.io/bound-by-controller=yes | ||
| pv.kubernetes.io/provisioned-by=kubernetes.io/azure-disk | ||
| volumehelper.VolumeDynamicallyCreatedByKey=azure-disk-dynamic-provisioner | ||
| Finalizers: [kubernetes.io/pv-protection] | ||
| StorageClass: azuredisk-unzoned | ||
| Status: Bound | ||
| Claim: default/unzoned-pvc | ||
| Reclaim Policy: Delete | ||
| Access Modes: RWO | ||
| Capacity: 5Gi | ||
| Node Affinity: | ||
| Required Terms: | ||
| Term 0: failure-domain.beta.kubernetes.io/region in [southeastasia] | ||
| failure-domain.beta.kubernetes.io/zone in [0] | ||
| Term 1: failure-domain.beta.kubernetes.io/region in [southeastasia] | ||
| failure-domain.beta.kubernetes.io/zone in [1] | ||
| Term 2: failure-domain.beta.kubernetes.io/region in [southeastasia] | ||
| failure-domain.beta.kubernetes.io/zone in [2] | ||
| Message: | ||
| Source: | ||
| Type: AzureDisk (an Azure Data Disk mount on the host and bind mount to the pod) | ||
| DiskName: k8s-5b3d7b8f-dynamic-pvc-bdf93a67-9c45-11e8-ba6f-000d3a07de8c | ||
| DiskURI: /subscriptions/<subscription>/resourceGroups/<rg-name>/providers/Microsoft.Compute/disks/k8s-5b3d7b8f-dynamic-pvc-bdf93a67-9c45-11e8-ba6f-000d3a07de8c | ||
| Kind: Managed | ||
| FSType: | ||
| CachingMode: None | ||
| ReadOnly: false | ||
| Events: <none> | ||
| ``` | ||
|
|
||
| ## Appendix | ||
|
|
||
| Note that unlike most cases, fault domain and availability zones mean different on Azure: | ||
|
|
||
| - A Fault Domain (FD) is essentially a rack of servers. It consumes subsystems like network, power, cooling etc. | ||
| - Availability Zones are unique physical locations within an Azure region. Each zone is made up of one or more data centers equipped with independent power, cooling, and networking. | ||
|
|
||
| An Availability Zone in an Azure region is a combination of a fault domain and an update domain (Same like FD, but for updates. When upgrading a deployment, it is carried out one update domain at a time). For example, if you create three or more VMs across three zones in an Azure region, your VMs are effectively distributed across three fault domains and three update domains. | ||
|
|
||
| ## Reference | ||
|
|
||
| See design docs for AZ in [KEP for Azure availability zones](https://github.com/kubernetes/community/blob/master/keps/sig-azure/0018-20180711-azure-availability-zones.md). | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a feature gate that users have to set to enable this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noop, this is enabled for v1.12, so that new zoned nodes could be joined easily after upgrading an existing cluster.