Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recommend runtimeclasses for scheduling #29062

Closed
wants to merge 4 commits into from
Closed

Recommend runtimeclasses for scheduling #29062

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bcec2c5
Recommend runtimeclasses for scheduling
ravisantoshgudimetla Jul 21, 2021
0daed0a
Address reviewers comments
ravisantoshgudimetla Jul 22, 2021
02e7dfb
Address reviewers comments
ravisantoshgudimetla Jul 22, 2021
ecafc37
Address reviewers comments
ravisantoshgudimetla Jul 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
255 changes: 132 additions & 123 deletions ...t/en/docs/setup/production-environment/windows/user-guide-windows-containers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,51 +36,7 @@ The example in the section below is provided to jumpstart your experience with W

To deploy a Windows container on Kubernetes, you must first create an example application.
The example YAML file below creates a simple webserver application.
Create a service spec named `win-webserver.yaml` with the contents below:

```yaml
apiVersion: v1
kind: Service
metadata:
name: win-webserver
labels:
app: win-webserver
spec:
ports:
# the port that this service should serve on
- port: 80
targetPort: 80
selector:
app: win-webserver
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: win-webserver
name: win-webserver
spec:
replicas: 2
selector:
matchLabels:
app: win-webserver
template:
metadata:
labels:
app: win-webserver
name: win-webserver
spec:
containers:
- name: windowswebserver
image: mcr.microsoft.com/windows/servercore:ltsc2019
command:
- powershell.exe
- -command
- "<#code used from https://gist.github.com/19WAS85/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ; ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus) } ; "
nodeSelector:
kubernetes.io/os: windows
```
Create a file named `win-webserver.yaml` with the contents from [using runtimeclasses](#use-runtimeclass-recommended-approach)

{{< note >}}
Port mapping is also supported, but for simplicity in this example
Expand Down Expand Up @@ -154,14 +110,137 @@ simplified service principal name (SPN) management, and the ability to delegate
Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA.
Learn more about configuring and using GMSA for Windows containers [here](/docs/tasks/configure-pod-container/configure-gmsa/).

## Taints and Tolerations
## Ensuring OS-specific workloads land on the appropriate container host

Users today need to use some combination of taints and node selectors in order to
Users today need to use some combination of taints on the node and runtimeclasses on the pod in order to
keep Linux and Windows workloads on their respective OS-specific nodes.
This likely imposes a burden only on Windows users. The recommended approach is outlined below,
with one of its main goals being that this approach should not break compatibility for existing Linux workloads.
We understand that in many cases users have a pre-existing large number of deployments for Linux containers,
as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators.
In those situations, you may be hesitant to make the configuration change to add runtimeclasses.
The alternative is to use Taints. Because the kubelet can set Taints during registration,
it could easily be modified to automatically add a taint when running on Windows only.

For example: adding the flag `--register-with-taints='os=windows:NoSchedule'` to kubelet parameters causes kubelet to register the taint.

By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods).

In order for the pod to land on the Windows host, we need to use either runtimeclasses or a combination of
nodeSelector and tolerations on the pod spec. Using the runtimeclasses is recommended approach as explained below.

### Use RuntimeClass (Recommended Approach)

### Ensuring OS-specific workloads land on the appropriate container host
[RuntimeClass] can be used to simplify the process of using taints and tolerations. In this approach,
the cluster administrator creates a `RuntimeClass` object which is used to encapsulate these tolerations
and nodeSelector.

Please note that this is the recommended approach to schedule pods onto Windows nodes
instead of using nodeSelector and tolerations as they can be added to pod spec by any user making them
insecure to be used.


1. Save this file to `runtimeClasses.yml`. It includes the appropriate `nodeSelector`
for the Windows OS, architecture. Please choose the runtimeclass based on the
container runtime you configured, which can be either docker or containerd


{{< tabs name="tab-runtimeclasses-for-windows" >}}
{{% tab name="docker" %}}

```yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: windows-2019
handler: 'docker'
scheduling:
nodeSelector:
kubernetes.io/os: 'windows'
kubernetes.io/arch: 'amd64'
tolerations:
- effect: NoSchedule
key: os
operator: Equal
value: "windows"
```

{{% /tab %}}

{{% tab name="containerd" %}}

```yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: windows-2019
handler: 'runhcs-wcow-process'
scheduling:
nodeSelector:
kubernetes.io/os: 'windows'
kubernetes.io/arch: 'amd64'
tolerations:
- effect: NoSchedule
key: os
operator: Equal
value: "windows"
```

{{% /tab %}}
{{< /tabs >}}

1. Run `kubectl create -f runtimeClasses.yml` using as a cluster administrator
1. Add `runtimeClassName: windows-2019` as appropriate to Pod specs

For example:

```yaml
apiVersion: v1
kind: Service
metadata:
name: win-webserver
labels:
app: win-webserver
spec:
ports:
# the port that this service should serve on
- port: 80
targetPort: 80
selector:
app: win-webserver
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: win-webserver
name: win-webserver
spec:
replicas: 2
selector:
matchLabels:
app: win-webserver
template:
metadata:
labels:
app: win-webserver
name: win-webserver
spec:
runtimeClassName: windows-2019
containers:
- name: windowswebserver
image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
command:
- powershell.exe
- -command
- "<#code used from https://gist.github.com/19WAS85/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ; ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus) } ; "
```

[RuntimeClass]: https://kubernetes.io/docs/concepts/containers/runtime-class/
[containerd default runtimeclass handler](https://kubernetes.io/docs/concepts/containers/runtime-class/#cri-configuration)

### Use NodeSelector and tolerations explicitly(This approach is not recommended)

Users can ensure Windows containers can be scheduled on the appropriate host using Taints and Tolerations.
All Kubernetes nodes today have the following default labels:
Expand All @@ -172,17 +251,8 @@ All Kubernetes nodes today have the following default labels:
If a Pod specification does not specify a nodeSelector like `"kubernetes.io/os": windows`,
it is possible the Pod can be scheduled on any host, Windows or Linux.
This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux.
The best practice is to use a nodeSelector.

However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers,
as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators.
In those situations, you may be hesitant to make the configuration change to add nodeSelectors.
The alternative is to use Taints. Because the kubelet can set Taints during registration,
it could easily be modified to automatically add a taint when running on Windows only.

For example: `--register-with-taints='os=windows:NoSchedule'`

By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods).
In order for a Windows Pod to be scheduled on a Windows node,
it would need both the nodeSelector to choose Windows, and the appropriate matching toleration.

Expand All @@ -197,6 +267,8 @@ tolerations:
effect: "NoSchedule"
```

This is not recommended, please use runtimeclasses instead.

### Handling multiple Windows versions in the same cluster

The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows
Expand All @@ -214,22 +286,14 @@ Here are values used today for each Windows Server version.
| Windows Server version 1809 | 10.0.17763 |
| Windows Server version 1903 | 10.0.18362 |


### Simplifying with RuntimeClass

[RuntimeClass] can be used to simplify the process of using taints and tolerations.
A cluster administrator can create a `RuntimeClass` object which is used to encapsulate these taints and tolerations.


1. Save this file to `runtimeClasses.yml`. It includes the appropriate `nodeSelector`
for the Windows OS, architecture, and version.
A sample runtimeclass with nodeSelectors to include build numbers is shown below

```yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: windows-2019
handler: 'docker'
handler: 'runhcs-wcow-process'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use tabs to show handles for docker and the containerd default handler similar to OS tabs at https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o?

A lot of windows users are still running with docker (unfortunately) and this can be a pretty big change for them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need some help from @sftim or someone from familiar with that <tab> tag. I am not sure if I am having correct tags.

scheduling:
nodeSelector:
kubernetes.io/os: 'windows'
Expand All @@ -240,59 +304,4 @@ scheduling:
key: os
operator: Equal
value: "windows"
```

1. Run `kubectl create -f runtimeClasses.yml` using as a cluster administrator
1. Add `runtimeClassName: windows-2019` as appropriate to Pod specs

For example:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: iis-2019
labels:
app: iis-2019
spec:
replicas: 1
template:
metadata:
name: iis-2019
labels:
app: iis-2019
spec:
runtimeClassName: windows-2019
containers:
- name: iis
image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
resources:
limits:
cpu: 1
memory: 800Mi
requests:
cpu: .1
memory: 300Mi
ports:
- containerPort: 80
selector:
matchLabels:
app: iis-2019
---
apiVersion: v1
kind: Service
metadata:
name: iis
spec:
type: LoadBalancer
ports:
- protocol: TCP
port: 80
selector:
app: iis-2019
```




[RuntimeClass]: https://kubernetes.io/docs/concepts/containers/runtime-class/
```