From 5ed47f65553bbf3a76416f77983d25ec74cf87e5 Mon Sep 17 00:00:00 2001 From: Michael Michael Date: Wed, 20 Mar 2019 20:34:23 -0500 Subject: [PATCH 1/7] Update intro-windows-in-kubernetes.md --- .../windows/intro-windows-in-kubernetes.md | 52 ++++++++++++++++--- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md index ed841d0213c1e..316d7eb7a97f8 100644 --- a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md +++ b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md @@ -44,6 +44,9 @@ Let's start with the operating system version. Refer to the following table for | | *Windows Server 1709* | *Windows Server 1803* | *Windows Server 1809/Windows Server 2019* | | *Kubernetes v1.14* | Not Supported | Not Supported| Supported for Windows Server containers Builds 17763.* with Docker EE-basic 18.09 | +{{< note >}} +We don't expect all Windows customers to update the operating system for their apps frequently. Upgrading your applications is what will dictate and necessitate upgrading or introducing new nodes to the cluster. For the customers that chose to upgrade their operating system for containers running on Kubernetes, we will offer guidance and step-by-step instructions when we add support for a new operating system version. This guidance will include recommended upgrade procedures for upgrading user applications together with cluster nodes. Windows nodes will adhere to Kubernetes [version-skew policy](https://kubernetes.io/docs/setup/version-skew-policy/) (node to control plane versioning) the same way as Linux nodes do today. +{{< /note >}} {{< note >}} The Windows Server Host Operating System is subject to the [Windows Server ](https://www.microsoft.com/en-us/cloud-platform/windows-server-pricing) licensing. The Windows Container images are subject to the [Supplemental License Terms for Windows containers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/images-eula). {{< /note >}} @@ -190,12 +193,12 @@ A best practice to avoid over-provisioning is to configure the kubelet with a sy The behavior of the flags behave differently as described below: -* --kubelet-reserve, --system-reserve , and --eviction-hard flags update Node Allocatable -* Eviction by using --enforce-node-allocable is not implemented -* Eviction by using --eviction-hard and --eviction-soft are not implemented +* `--kubelet-reserve`, `--system-reserve` , and `--eviction-hard` flags update Node Allocatable +* Eviction by using `--enforce-node-allocable` is not implemented +* Eviction by using `--eviction-hard` and `--eviction-soft` are not implemented * MemoryPressure Condition is not implemented * There are no OOM eviction actions taken by the kubelet -* Kubelet running on the windows node does not have memory restrictions. --kubelet-reserve and --system-reserve do not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the node-allocatable and scheduler +* Kubelet running on the windows node does not have memory restrictions. `--kubelet-reserve` and `--system-reserve` do not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the node-allocatable and scheduler #### Storage @@ -218,6 +221,7 @@ As a result, the following storage functionality is not supported on Windows nod * CSI plugins which require privileged containers * File system features like uui/guid, per-user Linux filesystem permissions * NFS based storage/volume support +* Expanding the mounted volume (resizefs) #### Networking @@ -231,6 +235,8 @@ The following networking functionality is not supported on Windows nodes * Local NodePort access from the node itself will fail (works for other nodes or external clients) * Accessing service VIPs from nodes will be available with a future release of Windows Server * Overlay networking support in kube-proxy is an alpha release. In addition, it requires [KB4482887](https://support.microsoft.com/en-us/help/4482887/windows-10-update-kb4482887) to be installed on Windows Server 2019 +* kubectl port-forward +* Local Traffic Policy and DSR mode * Outbound communication using the ICMP protocol via the win-overlay, win-bridge, and Azure-CNI plugin. Specifically, the Windows data plane ([VFP](https://www.microsoft.com/en-us/research/project/azure-virtual-filtering-platform/)) doesn't support ICMP packet transpositions. This means: * ICMP packets directed to destinations within the same network (e.g. pod to pod communication via ping) will work as expected and without any limitations * TCP/UDP packets will work as expected and without any limitations @@ -492,11 +498,13 @@ Your main source of help for troubleshooting your Kubernetes cluster should star Check that your pause image is compatible with your OS version. The [instructions](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/deploying-resources) assume that both the OS and the containers are version 1803. If you have a later version of Windows, such as an Insider build, you will need to adjust the images accordingly. Please refer to the Microsoft's [Docker repository](https://hub.docker.com/u/microsoft/) for images. Regardless, both the pause image Dockerfile and the sample service will expect the image to be tagged as :latest. - With Kubernetes v1.14, Microsoft will release the pause infrastructure container at mcr.microsoft.com/k8s/core/pause:1.0.0. For more information search for "pause" in the "Guide for adding Windows Nodes in Kubernetes". + With Kubernetes v1.14, Microsoft will release the pause infrastructure container at `mcr.microsoft.com/k8s/core/pause:1.0.0`. For more information search for "pause" in the [Guide for adding Windows Nodes in Kubernetes](../user-guide-windows-nodes). -### Further investigation +1. DNS resolution is not properly working -Check the DNS limitations for Windows in this [section](#dns-limitations). + Check the DNS limitations for Windows in this [section](#dns-limitations). + +### Further investigation If these steps don't resolve your problem, you can get help running Windows containers on Windows nodes in Kubernetes through: @@ -526,13 +534,41 @@ We have a lot of features in our roadmap. An abbreviated high level list is incl ContainerD is another OCI-compliant runtime that recently graduated as a CNCF project. It's currently tested on Linux, but 1.3 will bring support for Windows and Hyper-V. [[reference](https://blog.docker.com/2019/02/containerd-graduates-within-the-cncf/)] -The CRI-ContainerD interface will be able to manage sandboxes based on Hyper-V. This provides a foundation where RuntimeClasses could be implemented for new use cases including: +The CRI-ContainerD interface will be able to manage sandboxes based on Hyper-V. This provides a foundation where RuntimeClass could be implemented for new use cases including: * Hypervisor-based isolation between pods for additional security * Backwards compatibility allowing a node to run a newer Windows Server version without requiring containers to be rebuilt * Specific CPU/NUMA settings for a pod * Memory isolation and reservations +### Hyper-V isolation + +The existing Hyper-V isolation support, an experimental feature as of v1.10, will be deprecated in the future in favor of the CRI-ContainerD and RuntimeClass features mentioned above. To use the current features and create a Hyper-V isolated container, the kubelet should be started with feature gates `HyperVContainer=true` and the Pod should include the annotation `experimental.windows.kubernetes.io/isolation-type=hyperv`. In the experiemental release, this feature is limited to 1 container per Pod. + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: iis +spec: + selector: + matchLabels: + app: iis + replicas: 3 + template: + metadata: + labels: + app: iis + annotations: + experimental.windows.kubernetes.io/isolation-type: hyperv + spec: + containers: + - name: iis + image: microsoft/iis + ports: + - containerPort: 80 +``` + ### Deployment with kubeadm and cluster API Kubeadm is becoming the de facto standard for users to deploy a Kubernetes cluster. Windows node support in kubeadm will come in a future release. We are also making investments in cluster API to ensure Windows nodes are properly provisioned. From b2e83c5a535ac7e5119cf11e789533e926394390 Mon Sep 17 00:00:00 2001 From: Michael Michael Date: Wed, 20 Mar 2019 20:37:24 -0500 Subject: [PATCH 2/7] Update intro-windows-in-kubernetes.md --- content/en/docs/setup/windows/intro-windows-in-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md index 316d7eb7a97f8..0e2edb3617b94 100644 --- a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md +++ b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md @@ -45,7 +45,7 @@ Let's start with the operating system version. Refer to the following table for | *Kubernetes v1.14* | Not Supported | Not Supported| Supported for Windows Server containers Builds 17763.* with Docker EE-basic 18.09 | {{< note >}} -We don't expect all Windows customers to update the operating system for their apps frequently. Upgrading your applications is what will dictate and necessitate upgrading or introducing new nodes to the cluster. For the customers that chose to upgrade their operating system for containers running on Kubernetes, we will offer guidance and step-by-step instructions when we add support for a new operating system version. This guidance will include recommended upgrade procedures for upgrading user applications together with cluster nodes. Windows nodes will adhere to Kubernetes [version-skew policy](https://kubernetes.io/docs/setup/version-skew-policy/) (node to control plane versioning) the same way as Linux nodes do today. +We don't expect all Windows customers to update the operating system for their apps frequently. Upgrading your applications is what will dictate and necessitate upgrading or introducing new nodes to the cluster. For the customers that chose to upgrade their operating system for containers running on Kubernetes, we will offer guidance and step-by-step instructions when we add support for a new operating system version. This guidance will include recommended upgrade procedures for upgrading user applications together with cluster nodes. Windows nodes will adhere to Kubernetes [version-skew policy](/docs/setup/version-skew-policy/) (node to control plane versioning) the same way as Linux nodes do today. {{< /note >}} {{< note >}} The Windows Server Host Operating System is subject to the [Windows Server ](https://www.microsoft.com/en-us/cloud-platform/windows-server-pricing) licensing. The Windows Container images are subject to the [Supplemental License Terms for Windows containers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/images-eula). From 08fee159e41f13a8f69cc2b30c62d3e27cb53822 Mon Sep 17 00:00:00 2001 From: Michael Michael Date: Wed, 20 Mar 2019 20:49:17 -0500 Subject: [PATCH 3/7] Update intro-windows-in-kubernetes.md --- .../docs/setup/windows/intro-windows-in-kubernetes.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md index 0e2edb3617b94..cefcef1fbf4f1 100644 --- a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md +++ b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md @@ -235,13 +235,13 @@ The following networking functionality is not supported on Windows nodes * Local NodePort access from the node itself will fail (works for other nodes or external clients) * Accessing service VIPs from nodes will be available with a future release of Windows Server * Overlay networking support in kube-proxy is an alpha release. In addition, it requires [KB4482887](https://support.microsoft.com/en-us/help/4482887/windows-10-update-kb4482887) to be installed on Windows Server 2019 -* kubectl port-forward +* `kubectl port-forward` * Local Traffic Policy and DSR mode * Outbound communication using the ICMP protocol via the win-overlay, win-bridge, and Azure-CNI plugin. Specifically, the Windows data plane ([VFP](https://www.microsoft.com/en-us/research/project/azure-virtual-filtering-platform/)) doesn't support ICMP packet transpositions. This means: * ICMP packets directed to destinations within the same network (e.g. pod to pod communication via ping) will work as expected and without any limitations * TCP/UDP packets will work as expected and without any limitations * ICMP packets directed to pass through a remote network (e.g. pod to external internet communication via ping) cannot be transposed and thus will not be routed back to their source - * Since TCP/UDP packets can still be transposed, one can substitute **ping ** with **curl ** to be able to debug connectivity to the outside world. + * Since TCP/UDP packets can still be transposed, one can substitute `ping ` with `curl ` to be able to debug connectivity to the outside world. ##### CNI Plugins @@ -249,7 +249,7 @@ The following networking functionality is not supported on Windows nodes * The Flannel VXLAN CNI has the following limitations on Windows: 1. Node-pod connectivity isn't possible by design. It's only possible for local pods with Flannel [PR 1096](https://github.com/coreos/flannel/pull/1096) -2. We are restricted to using VNI 4096 and UDP port 4789. The VNI limitation is being worked on and will be overcome (open-source flannel changes). See official [Flannel VXLAN ](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)backend docs for more details on these parameters. +2. We are restricted to using VNI 4096 and UDP port 4789. The VNI limitation is being worked on and will be overcome (open-source flannel changes). See the official [Flannel VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) backend docs for more details on these parameters. ##### DNS {#dns-limitations} @@ -275,7 +275,7 @@ There are no differences in how most of the Kubernetes APIs work for Windows. Th At a high level, these OS concepts are different: -* Identity - Linux uses userID (UID) and groupID (GID) which are represented as integer types. User and group names are not canonical - they are just an alias in /etc/groups or /etc/passwd back to UID+GID. Windows uses a larger binary security identifier (SID) which is stored in the Windows Security Access Manager (SAM) database. This database is not shared between the host and containers, or between containers. +* Identity - Linux uses userID (UID) and groupID (GID) which are represented as integer types. User and group names are not canonical - they are just an alias in `/etc/groups` or `/etc/passwd` back to UID+GID. Windows uses a larger binary security identifier (SID) which is stored in the Windows Security Access Manager (SAM) database. This database is not shared between the host and containers, or between containers. * File permissions - Windows uses an access control list based on SIDs, rather than a bitmask of permissions and UID+GID * File paths - convention on Windows is to use **\** instead of **/**. The Go IO libraries typically accept both and just make it work, but when you're setting a path or command line that's interpreted inside a container, **\** may be needed. * Signals - Windows interactive apps handle termination differently, and can implement one or more of these: @@ -306,7 +306,7 @@ Exit Codes follow the same convention where 0 is success, nonzero is failure. Th * V1.Pod.hostIPC, v1.pod.hostpid - host namespace sharing is not possible on Windows * V1.Pod.hostNetwork - There is no Windows OS support to share the host network * V1.Pod.dnsPolicy - ClusterFirstWithHostNet - is not supported because Host Networking is not supported on Windows. -* V1.Pod.podSecurityContext - see [V1.PodSecurityContext](https://github.com/kubernetes/enhancements/blob/master/keps/sig-windows/20190103-windows-node-support.md#v1podsecuritycontext) +* V1.Pod.podSecurityContext - see V1.PodSecurityContext below * V1.Pod.shareProcessNamespace - this is a beta feature, and depends on Linux namespaces which are not implemented on Windows. Windows cannot share process namespaces or the container's root filesystem. Only the network can be shared. * V1.Pod.terminationGracePeriodSeconds - this is not fully implemented in Docker on Windows, see: [reference](https://github.com/moby/moby/issues/25982). The behavior today is that the ENTRYPOINT process is sent CTRL_SHUTDOWN_EVENT, then Windows waits 5 seconds by default, and finally shuts down all processes using the normal Windows shutdown behavior. The 5 second default is actually in the Windows registry [inside the container](https://github.com/moby/moby/issues/25982#issuecomment-426441183), so it can be overridden when the container is built. * V1.Pod.volumeDevices - this is a beta feature, and is not implemented on Windows. Windows cannot attach raw block devices to pods. From dc5347fdae2842b0ea1078fe2f3462a0a03f7921 Mon Sep 17 00:00:00 2001 From: Michael Michael Date: Wed, 20 Mar 2019 23:38:28 -0500 Subject: [PATCH 4/7] Update intro-windows-in-kubernetes.md --- content/en/docs/setup/windows/intro-windows-in-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md index cefcef1fbf4f1..7efd7b32e2ea3 100644 --- a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md +++ b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md @@ -498,7 +498,7 @@ Your main source of help for troubleshooting your Kubernetes cluster should star Check that your pause image is compatible with your OS version. The [instructions](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/deploying-resources) assume that both the OS and the containers are version 1803. If you have a later version of Windows, such as an Insider build, you will need to adjust the images accordingly. Please refer to the Microsoft's [Docker repository](https://hub.docker.com/u/microsoft/) for images. Regardless, both the pause image Dockerfile and the sample service will expect the image to be tagged as :latest. - With Kubernetes v1.14, Microsoft will release the pause infrastructure container at `mcr.microsoft.com/k8s/core/pause:1.0.0`. For more information search for "pause" in the [Guide for adding Windows Nodes in Kubernetes](../user-guide-windows-nodes). + Starting with Kubernetes v1.14, Microsoft releases the pause infrastructure container at `mcr.microsoft.com/k8s/core/pause:1.0.0`. For more information search for "pause" in the [Guide for adding Windows Nodes in Kubernetes](../user-guide-windows-nodes). 1. DNS resolution is not properly working From 9671c63af9bff1ab8d94f4f878bb75f141ff28ac Mon Sep 17 00:00:00 2001 From: Michael Michael Date: Fri, 22 Mar 2019 09:46:01 -0500 Subject: [PATCH 5/7] Update intro-windows-in-kubernetes.md --- .../windows/intro-windows-in-kubernetes.md | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md index 7efd7b32e2ea3..4c255149bc76f 100644 --- a/content/en/docs/setup/windows/intro-windows-in-kubernetes.md +++ b/content/en/docs/setup/windows/intro-windows-in-kubernetes.md @@ -22,7 +22,7 @@ To enable the orchestration of Windows containers in Kubernetes, simply include In order to run Windows containers, your Kubernetes cluster must include multiple operating systems, with control plane nodes running Linux and workers running either Windows or Linux depending on your workload needs. Windows Server 2019 is the only Windows operating system supported, enabling [Kubernetes Node](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/architecture.md#the-kubernetes-node) on Windows (including kubelet, [container runtime](https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/containerd), and kube-proxy). For a detailed explanation of Windows distribution channels see the [Microsoft documentation](https://docs.microsoft.com/en-us/windows-server/get-started-19/servicing-channels-19). {{< note >}} -The Kubernetes control plane, including the [master components](/docs/concepts/overview/components/), will continue to run on Linux. There are no plans to have a Windows-only Kubernetes cluster. +The Kubernetes control plane, including the [master components](/docs/concepts/overview/components/), continues to run on Linux. There are no plans to have a Windows-only Kubernetes cluster. {{< /note >}} {{< note >}} @@ -45,7 +45,7 @@ Let's start with the operating system version. Refer to the following table for | *Kubernetes v1.14* | Not Supported | Not Supported| Supported for Windows Server containers Builds 17763.* with Docker EE-basic 18.09 | {{< note >}} -We don't expect all Windows customers to update the operating system for their apps frequently. Upgrading your applications is what will dictate and necessitate upgrading or introducing new nodes to the cluster. For the customers that chose to upgrade their operating system for containers running on Kubernetes, we will offer guidance and step-by-step instructions when we add support for a new operating system version. This guidance will include recommended upgrade procedures for upgrading user applications together with cluster nodes. Windows nodes will adhere to Kubernetes [version-skew policy](/docs/setup/version-skew-policy/) (node to control plane versioning) the same way as Linux nodes do today. +We don't expect all Windows customers to update the operating system for their apps frequently. Upgrading your applications is what dictates and necessitates upgrading or introducing new nodes to the cluster. For the customers that chose to upgrade their operating system for containers running on Kubernetes, we will offer guidance and step-by-step instructions when we add support for a new operating system version. This guidance will include recommended upgrade procedures for upgrading user applications together with cluster nodes. Windows nodes adhere to Kubernetes [version-skew policy](/docs/setup/version-skew-policy/) (node to control plane versioning) the same way as Linux nodes do today. {{< /note >}} {{< note >}} The Windows Server Host Operating System is subject to the [Windows Server ](https://www.microsoft.com/en-us/cloud-platform/windows-server-pricing) licensing. The Windows Container images are subject to the [Supplemental License Terms for Windows containers](https://docs.microsoft.com/en-us/virtualization/windowscontainers/images-eula). @@ -54,7 +54,7 @@ The Windows Server Host Operating System is subject to the [Windows Server ](htt Windows containers with process isolation have strict compatibility rules, [where the host OS version must match the container base image OS version](https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility). Once we support Windows containers with Hyper-V isolation in Kubernetes, the limitation and compatibility rules will change. {{< /note >}} -Key Kubernetes elements work the same way in Windows as they do in Linux. In this section, we will talk about some of the key workload enablers and how they map to Windows. +Key Kubernetes elements work the same way in Windows as they do in Linux. In this section, we talk about some of the key workload enablers and how they map to Windows. * [Pods](/docs/concepts/workloads/pods/pod-overview/) @@ -134,7 +134,7 @@ Windows supports five different networking drivers/modes: L2bridge, L2tunnel, Ov | L2bridge | Containers are attached to an external vSwitch. Containers are attached to the underlay network, although the physical network doesn't need to learn the container MACs because they are rewritten on ingress/egress. Inter-container traffic is bridged inside the container host. | MAC is rewritten to host MAC, IP remains the same. | [win-bridge](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-bridge), [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md), Flannel host-gateway uses win-bridge | win-bridge uses L2bridge network mode, connects containers to the underlay of hosts, offering best performance. Requires L2 adjacency between container hosts | | L2Tunnel | This is a special case of l2bridge, but only used on Azure. All packets are sent to the virtualization host where SDN policy is applied. | MAC rewritten, IP visible on the underlay network | [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md) | Azure-CNI allows integration of containers with Azure vNET, and allows them to leverage the set of capabilities that [Azure Virtual Network provides](https://azure.microsoft.com/en-us/services/virtual-network/). For example, securely connect to Azure services or use Azure NSGs. See [azure-cni for some examples](https://docs.microsoft.com/en-us/azure/aks/concepts-network#azure-cni-advanced-networking) | | Overlay (Overlay networking for Windows in Kubernetes is in *alpha* stage) | Containers are given a vNIC connected to an external vSwitch. Each overlay network gets its own IP subnet, defined by a custom IP prefix.The overlay network driver uses VXLAN encapsulation. | Encapsulated with an outer header, inner packet remains the same. | [Win-overlay](https://github.com/containernetworking/plugins/tree/master/plugins/main/windows/win-overlay), Flannel VXLAN (uses win-overlay) | win-overlay should be used when virtual container networks are desired to be isolated from underlay of hosts (e.g. for security reasons). Allows for IPs to be re-used for different overlay networks (which have different VNID tags) if you are restricted on IPs in your datacenter. This option may be used when the container hosts are not L2 adjacent but have L3 connectivity | -| Transparent (special use case for [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes)) | Requires an external vSwitch. Containers are attached to an external vSwitch which will enable intra-pod communication via logical networks (logical switches and routers). | Packet is encapsulated either via [GENEVE](https://datatracker.ietf.org/doc/draft-gross-geneve/) or [STT](https://datatracker.ietf.org/doc/draft-davie-stt/) tunneling to reach pods which are not on the same host.
Packets are forwarded or dropped via the tunnel metadata information supplied by the ovn network controller.
NAT is done for north-south communication. | [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes) | [Deploy via ansible](https://github.com/openvswitch/ovn-kubernetes/tree/master/contrib). Distributed ACLs can be applied via Kubernetes policies. IPAM support. Load-balancing can be achieved without kube-proxy. NATing is done without using iptables/netsh. | +| Transparent (special use case for [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes)) | Requires an external vSwitch. Containers are attached to an external vSwitch which enables intra-pod communication via logical networks (logical switches and routers). | Packet is encapsulated either via [GENEVE](https://datatracker.ietf.org/doc/draft-gross-geneve/) or [STT](https://datatracker.ietf.org/doc/draft-davie-stt/) tunneling to reach pods which are not on the same host.
Packets are forwarded or dropped via the tunnel metadata information supplied by the ovn network controller.
NAT is done for north-south communication. | [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes) | [Deploy via ansible](https://github.com/openvswitch/ovn-kubernetes/tree/master/contrib). Distributed ACLs can be applied via Kubernetes policies. IPAM support. Load-balancing can be achieved without kube-proxy. NATing is done without using iptables/netsh. | | NAT (*not used in Kubernetes*) | Containers are given a vNIC connected to an internal vSwitch. DNS/DHCP is provided using an internal component called [WinNAT](https://blogs.technet.microsoft.com/virtualization/2016/05/25/windows-nat-winnat-capabilities-and-limitations/) | MAC and IP is rewritten to host MAC/IP. | [nat](https://github.com/Microsoft/windows-container-networking/tree/master/plugins/nat) | Included here for completeness | As outlined above, the [Flannel](https://github.com/coreos/flannel) CNI [meta plugin](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel) is also supported on [Windows](https://github.com/containernetworking/plugins/tree/master/plugins/meta/flannel#windows-support-experimental) via the [VXLAN network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) (**alpha support** ; delegates to win-overlay) and [host-gateway network backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) (stable support; delegates to win-bridge). This plugin supports delegating to one of the reference CNI plugins (win-overlay, win-bridge), to work in conjunction with Flannel daemon on Windows (Flanneld) for automatic node subnet lease assignment and HNS network creation. This plugin reads in its own configuration file (net-conf.json), and aggregates it with the environment variables from the FlannelD generated subnet.env file. It then delegates to one of the reference CNI plugins for network plumbing, and sends the correct configuration containing the node-assigned subnet to the IPAM plugin (e.g. host-local). @@ -185,9 +185,9 @@ Windows has strict compatibility rules, where the host OS version must match the ##### Memory Reservations and Handling -Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. The net effect is that Windows won't reach out of memory conditions the same way Linux does, and processes will page to disk instead of being subject to out of memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance. +Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. The net effect is that Windows won't reach out of memory conditions the same way Linux does, and processes page to disk instead of being subject to out of memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance. -Keeping memory usage within reasonable bounds is possible with a two-step process. First, use the kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to account for memory usage on the node (outside of containers). This will reduce [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable)). As you deploy workloads, use resource limits (must set only limits or limits must equal requests) on containers. This will also subtract from NodeAllocatable and prevent the scheduler from adding more pods once a node is full. +Keeping memory usage within reasonable bounds is possible with a two-step process. First, use the kubelet parameters `--kubelet-reserve` and/or `--system-reserve` to account for memory usage on the node (outside of containers). This reduces [NodeAllocatable](/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable)). As you deploy workloads, use resource limits (must set only limits or limits must equal requests) on containers. This also subtracts from NodeAllocatable and prevents the scheduler from adding more pods once a node is full. A best practice to avoid over-provisioning is to configure the kubelet with a system reserved memory of at least 2GB to account for Windows, Docker, and Kubernetes processes. @@ -232,14 +232,14 @@ The Windows host networking networking service and virtual switch implement name The following networking functionality is not supported on Windows nodes * Host networking mode is not available for Windows pods -* Local NodePort access from the node itself will fail (works for other nodes or external clients) +* Local NodePort access from the node itself fails (works for other nodes or external clients) * Accessing service VIPs from nodes will be available with a future release of Windows Server * Overlay networking support in kube-proxy is an alpha release. In addition, it requires [KB4482887](https://support.microsoft.com/en-us/help/4482887/windows-10-update-kb4482887) to be installed on Windows Server 2019 * `kubectl port-forward` * Local Traffic Policy and DSR mode * Outbound communication using the ICMP protocol via the win-overlay, win-bridge, and Azure-CNI plugin. Specifically, the Windows data plane ([VFP](https://www.microsoft.com/en-us/research/project/azure-virtual-filtering-platform/)) doesn't support ICMP packet transpositions. This means: - * ICMP packets directed to destinations within the same network (e.g. pod to pod communication via ping) will work as expected and without any limitations - * TCP/UDP packets will work as expected and without any limitations + * ICMP packets directed to destinations within the same network (e.g. pod to pod communication via ping) work as expected and without any limitations + * TCP/UDP packets work as expected and without any limitations * ICMP packets directed to pass through a remote network (e.g. pod to external internet communication via ping) cannot be transposed and thus will not be routed back to their source * Since TCP/UDP packets can still be transposed, one can substitute `ping ` with `curl ` to be able to debug connectivity to the outside world. @@ -249,12 +249,12 @@ The following networking functionality is not supported on Windows nodes * The Flannel VXLAN CNI has the following limitations on Windows: 1. Node-pod connectivity isn't possible by design. It's only possible for local pods with Flannel [PR 1096](https://github.com/coreos/flannel/pull/1096) -2. We are restricted to using VNI 4096 and UDP port 4789. The VNI limitation is being worked on and will be overcome (open-source flannel changes). See the official [Flannel VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) backend docs for more details on these parameters. +2. We are restricted to using VNI 4096 and UDP port 4789. The VNI limitation is being worked on and will be overcome in a future release (open-source flannel changes). See the official [Flannel VXLAN](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan) backend docs for more details on these parameters. ##### DNS {#dns-limitations} * ClusterFirstWithHostNet is not supported for DNS. Windows treats all names with a '.' as a FQDN and skips PQDN resolution -* On Linux, you have a DNS suffix list, which is used when trying to resolve PQDNs. On Windows, we only have 1 DNS suffix, which is the DNS suffix associated with that pod's namespace (mydns.svc.cluster.local for example). Windows can resolve FQDNs and services or names resolvable with just that suffix. For example, a pod spawned in the default namespace, will have the DNS suffix **default.svc.cluster.local**. On a Windows pod, we will be able to resolve both **kubernetes.default.svc.cluster.local** and **kubernetes**, but not the in-betweens, like **kubernetes.default** or **kubernetes.default.svc**. +* On Linux, you have a DNS suffix list, which is used when trying to resolve PQDNs. On Windows, we only have 1 DNS suffix, which is the DNS suffix associated with that pod's namespace (mydns.svc.cluster.local for example). Windows can resolve FQDNs and services or names resolvable with just that suffix. For example, a pod spawned in the default namespace, will have the DNS suffix **default.svc.cluster.local**. On a Windows pod, you can resolve both **kubernetes.default.svc.cluster.local** and **kubernetes**, but not the in-betweens, like **kubernetes.default** or **kubernetes.default.svc**. ##### Security @@ -279,11 +279,11 @@ At a high level, these OS concepts are different: * File permissions - Windows uses an access control list based on SIDs, rather than a bitmask of permissions and UID+GID * File paths - convention on Windows is to use **\** instead of **/**. The Go IO libraries typically accept both and just make it work, but when you're setting a path or command line that's interpreted inside a container, **\** may be needed. * Signals - Windows interactive apps handle termination differently, and can implement one or more of these: - * A UI thread will handle well-defined messages including WM_CLOSE - * Console apps will handle ctrl-c or ctrl-break using a Control Handler - * Services will register a Service Control Handler function that can accept SERVICE_CONTROL_STOP control codes + * A UI thread handles well-defined messages including WM_CLOSE + * Console apps handle ctrl-c or ctrl-break using a Control Handler + * Services register a Service Control Handler function that can accept SERVICE_CONTROL_STOP control codes -Exit Codes follow the same convention where 0 is success, nonzero is failure. The specific error codes may differ across Windows and Linux. However, exit codes passed from the Kubernetes components (kubelet, kube-proxy) will be unchanged. +Exit Codes follow the same convention where 0 is success, nonzero is failure. The specific error codes may differ across Windows and Linux. However, exit codes passed from the Kubernetes components (kubelet, kube-proxy) are unchanged. ##### V1.Container @@ -426,7 +426,7 @@ Your main source of help for troubleshooting your Kubernetes cluster should star Windows Pods do not have outbound rules programmed for the ICMP protocol today. However, TCP/UDP is supported. When trying to demonstrate connectivity to resources outside of the cluster, please substitute `ping ` with corresponding `curl ` commands. - If you are still facing problems, most likely your network configuration in [cni.conf](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf) deserves some extra attention. You can always edit this static file, the configuration will be applied to any newly created Kubernetes resources. + If you are still facing problems, most likely your network configuration in [cni.conf](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf) deserves some extra attention. You can always edit this static file. The configuration update will apply to any newly created Kubernetes resources. One of the Kubernetes networking requirements (see [Kubernetes model](/docs/concepts/cluster-administration/networking/)) is for cluster communication to occur without NAT internally. To honor this requirement, there is an [ExceptionList](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/l2bridge/cni/config/cni.conf#L20) for all the communication where we do not want outbound NAT to occur. However, this also means that you need to exclude the external IP you are trying to query from the ExceptionList. Only then will the traffic originating from your Windows pods be SNAT'ed correctly to receive a response from the outside world. In this regard, your ExceptionList in `cni.conf` should look as follows: @@ -440,7 +440,7 @@ Your main source of help for troubleshooting your Kubernetes cluster should star 1. My Windows node cannot access NodePort service - Local NodePort access from the node itself will fail. This is a known limitation. NodePort access will work from other nodes or external clients. + Local NodePort access from the node itself fails. This is a known limitation. NodePort access works from other nodes or external clients. 1. vNICs and HNS endpoints of containers are being deleted @@ -452,7 +452,7 @@ Your main source of help for troubleshooting your Kubernetes cluster should star 1. With flannel my nodes are having issues after rejoining a cluster - Whenever a previously deleted node is being re-joined to the cluster, flannelD will try to assign a new pod subnet to the node. Users should remove the old pod subnet configuration files in the following paths: + Whenever a previously deleted node is being re-joined to the cluster, flannelD tries to assign a new pod subnet to the node. Users should remove the old pod subnet configuration files in the following paths: ```powershell Remove-Item C:\k\SourceVip.json @@ -496,7 +496,7 @@ Your main source of help for troubleshooting your Kubernetes cluster should star 1. My Pods are stuck at "Container Creating" or restarting over and over - Check that your pause image is compatible with your OS version. The [instructions](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/deploying-resources) assume that both the OS and the containers are version 1803. If you have a later version of Windows, such as an Insider build, you will need to adjust the images accordingly. Please refer to the Microsoft's [Docker repository](https://hub.docker.com/u/microsoft/) for images. Regardless, both the pause image Dockerfile and the sample service will expect the image to be tagged as :latest. + Check that your pause image is compatible with your OS version. The [instructions](https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/deploying-resources) assume that both the OS and the containers are version 1803. If you have a later version of Windows, such as an Insider build, you need to adjust the images accordingly. Please refer to the Microsoft's [Docker repository](https://hub.docker.com/u/microsoft/) for images. Regardless, both the pause image Dockerfile and the sample service expect the image to be tagged as :latest. Starting with Kubernetes v1.14, Microsoft releases the pause infrastructure container at `mcr.microsoft.com/k8s/core/pause:1.0.0`. For more information search for "pause" in the [Guide for adding Windows Nodes in Kubernetes](../user-guide-windows-nodes). From 3444832fc2d6f240bfc9754ed03b57d955d110eb Mon Sep 17 00:00:00 2001 From: Michael Michael Date: Fri, 22 Mar 2019 09:47:28 -0500 Subject: [PATCH 6/7] Update user-guide-windows-containers.md --- .../en/docs/setup/windows/user-guide-windows-containers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/setup/windows/user-guide-windows-containers.md b/content/en/docs/setup/windows/user-guide-windows-containers.md index 1d2e8afae6b69..0928f394f8d92 100644 --- a/content/en/docs/setup/windows/user-guide-windows-containers.md +++ b/content/en/docs/setup/windows/user-guide-windows-containers.md @@ -87,7 +87,7 @@ Port mapping is also supported, but for simplicity in this example the container kubectl get pods -o wide -w ``` - When the service is deployed correctly both Pods will be marked as Ready. To exit the watch command, press Ctrl+C. + When the service is deployed correctly both Pods are marked as Ready. To exit the watch command, press Ctrl+C. 1. Check that the deployment succeeded. To verify: @@ -110,7 +110,7 @@ Starting with Kubernetes v1.14, Windows container workloads can be configured to ## Taints and Tolerations -Users today will need to use some combination of taints and node selectors in order to keep Linux and Windows workloads on their respective OS-specific nodes. This will likely impose a burden only on Windows users. The recommended approach is outlined below, with one of its main goals being that this approach should not break compatibility for existing Linux workloads. +Users today need to use some combination of taints and node selectors in order to keep Linux and Windows workloads on their respective OS-specific nodes. This likely imposes a burden only on Windows users. The recommended approach is outlined below, with one of its main goals being that this approach should not break compatibility for existing Linux workloads. ### Ensuring OS-specific workloads land on the appropriate container host From ddcde5193008a2f43be9d2d8946ddcdd693b59f5 Mon Sep 17 00:00:00 2001 From: Michael Michael Date: Fri, 22 Mar 2019 09:50:04 -0500 Subject: [PATCH 7/7] Update user-guide-windows-nodes.md --- .../en/docs/setup/windows/user-guide-windows-nodes.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/en/docs/setup/windows/user-guide-windows-nodes.md b/content/en/docs/setup/windows/user-guide-windows-nodes.md index 4290832b03a0d..454e67084b33e 100644 --- a/content/en/docs/setup/windows/user-guide-windows-nodes.md +++ b/content/en/docs/setup/windows/user-guide-windows-nodes.md @@ -20,7 +20,7 @@ The Kubernetes platform can now be used to run both Linux and Windows containers ## Before you begin -* Obtain a [Windows Server license](https://www.microsoft.com/en-us/cloud-platform/windows-server-pricing) in order to run the Windows node that will execute the Windows container. You can use your organization's licenses for the cluster, or acquire one from Microsoft, a reseller, or via the major cloud providers such as GCP, AWS, and Azure by provisioning a virtual machine running Windows Server through their marketplaces. A [time-limited trial](https://www.microsoft.com/en-us/cloud-platform/windows-server-trial) is also available. +* Obtain a [Windows Server license](https://www.microsoft.com/en-us/cloud-platform/windows-server-pricing) in order to configure the Windows node that hosts Windows containers. You can use your organization's licenses for the cluster, or acquire one from Microsoft, a reseller, or via the major cloud providers such as GCP, AWS, and Azure by provisioning a virtual machine running Windows Server through their marketplaces. A [time-limited trial](https://www.microsoft.com/en-us/cloud-platform/windows-server-trial) is also available. * Build a Linux-based Kubernetes cluster in which you have access to the control plane (some examples include [Getting Started from Scratch](/docs/setup/scratch/), [kubeadm](/docs/setup/independent/create-cluster-kubeadm/), [AKS Engine](/docs/setup/turnkey/azure/), [GCE](/docs/setup/turnkey/gce/), [AWS](/docs/setup/turnkey/aws/)). ## Getting Started: Adding a Windows Node to Your Cluster @@ -29,19 +29,19 @@ The Kubernetes platform can now be used to run both Linux and Windows containers Kubernetes cluster management requires careful planning of your IP addresses so that you do not inadvertently cause network collision. This guide assumes that you are familiar with the [Kubernetes networking concepts](/docs/concepts/cluster-administration/networking/). -In order to deploy your cluster you will need the following address spaces: +In order to deploy your cluster you need the following address spaces: | Subnet / address range | Description | Default value | | --- | --- | --- | | Service Subnet | A non-routable, purely virtual subnet that is used by pods to uniformly access services without caring about the network topology. It is translated to/from routable address space by `kube-proxy` running on the nodes. | 10.96.0.0/12 | | Cluster Subnet | This is a global subnet that is used by all pods in the cluster. Each node is assigned a smaller /24 subnet from this for their pods to use. It must be large enough to accommodate all pods used in your cluster. To calculate *minimumsubnet* size: `(number of nodes) + (number of nodes * maximum pods per node that you configure)`. Example: for a 5 node cluster for 100 pods per node: `(5) + (5 * 100) = 505.` | 10.244.0.0/16 | -| Kubernetes DNS Service IP | IP address of `kube-dns` service that will be used for DNS resolution & cluster service discovery. | 10.96.0.10 | +| Kubernetes DNS Service IP | IP address of `kube-dns` service that is used for DNS resolution & cluster service discovery. | 10.96.0.10 | Review the networking options supported in 'Intro to Windows containers in Kubernetes: Supported Functionality: Networking' to determine how you need to allocate IP addresses for your cluster. ### Components that run on Windows -While the Kubernetes control plane runs on your Linux node(s), the following components will be configured and run on your Windows node(s). +While the Kubernetes control plane runs on your Linux node(s), the following components are configured and run on your Windows node(s). 1. kubelet 2. kube-proxy @@ -225,7 +225,7 @@ wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/s ``` {{< note >}} -[start.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/start.ps1) references [install.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/install.ps1), which will download additional files such as the `flanneld` executable and the [Dockerfile for infrastructure pod](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/Dockerfile) and install those for you. For overlay networking mode, the [firewall](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/helper.psm1#L111) will be opened for local UDP port 4789. There may be multiple powershell windows being opened/closed as well as a few seconds of network outage while the new external vSwitch for the pod network is being created the first time. Run the script using the arguments as specified below: +[start.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/flannel/start.ps1) references [install.ps1](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/install.ps1), which downloads additional files such as the `flanneld` executable and the [Dockerfile for infrastructure pod](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/Dockerfile) and install those for you. For overlay networking mode, the [firewall](https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/helper.psm1#L111) is opened for local UDP port 4789. There may be multiple powershell windows being opened/closed as well as a few seconds of network outage while the new external vSwitch for the pod network is being created the first time. Run the script using the arguments as specified below: {{< /note >}} ```PowerShell