Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows HostProcess Container Documentation #28413

Merged
merged 12 commits into from
Jul 27, 2021
Merged

Windows HostProcess Container Documentation #28413

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
85be93a
Rebasing HostProcess security changes.
brasmith-ms Jul 22, 2021
bc9786a
Incorporated initial round of feedback
brasmith-ms Jul 22, 2021
b0db71e
Minor wording updates
brasmith-ms Jul 22, 2021
e5c8b6a
Finished up remaining todo items
brasmith-ms Jul 23, 2021
21af0d5
Apply suggestions from code review
brasmith-ms Jul 23, 2021
b098cef
Merge branch 'dev-1.22' of https://github.com/kubernetes/website into…
brasmith-ms Jul 23, 2021
fe963d1
Moved HostProcess security documentation into PSS and create-host-pro…
brasmith-ms Jul 23, 2021
977f11d
Updated with for James' review
brasmith-ms Jul 23, 2021
9360d1a
Apply suggestions from code review
brasmith-ms Jul 24, 2021
48a3bf0
Minor edits
brasmith-ms Jul 24, 2021
acfabeb
Merge branch 'dev-1.22' of https://github.com/kubernetes/website into…
brasmith-ms Jul 26, 2021
4a0c02e
Modifications for additional feedback
brasmith-ms Jul 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 42 additions & 2 deletions content/en/docs/concepts/security/pod-security-standards.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,24 @@ fail validation.
<td><strong>Control</strong></td>
<td><strong>Policy</strong></td>
</tr>
<tr>
<td style="white-space: nowrap">HostProcess</td>
<td>
<p>Windows pods offer the ability to run <a href="/docs/tasks/configure-pod-container/create-hostprocess-pod">HostProcess containers</a> which enables privileged access to the Windows node. Privileged access to the host is disallowed in the baseline policy. </p>
<p><strong>Restricted Fields</strong></p>
<ul>
<li><code>spec.securityContext.windowsOptions.hostProcess</code></li>
<li><code>spec.containers[*].securityContext.windowsOptions.hostProcess</code></li>
<li><code>spec.initContainers[*].securityContext.windowsOptions.hostProcess</code></li>
<li><code>spec.ephemeralContainers[*].securityContext.windowsOptions.hostProcess</code></li>
</ul>
<p><strong>Allowed Values</strong></p>
<ul>
<li>Undefined/nil</li>
<li><code>false</code></li>
</ul>
</td>
</tr>
<tr>
<td style="white-space: nowrap">Host Namespaces</td>
<td>
Expand Down Expand Up @@ -257,6 +275,22 @@ fail validation.
</ul>
</td>
</tr>
<tr>
<td style="white-space: nowrap">Windows HostProcess</td>
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
<td>
<p>Running Windows pods as <a href="/docs/tasks/configure-pod-container/create-hostprocess-pod">HostProcess containers</a> enables privileged access to the Windows node and should be disallowed.</p>
<p><strong>Restricted Fields</strong></p>
<ul>
<li><code>spec.securityContext.windowsOptions.hostProcess</code></li>
<li><code>spec.containers[*].securityContext.windowsOptions.hostProcess</code></li>
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
</ul>
<p><strong>Allowed Values</strong></p>
<ul>
<li>Undefined/null</li>
<li><code>false</code></li>
</ul>
</td>
</tr>
</tbody>
</table>

Expand Down Expand Up @@ -483,10 +517,17 @@ ecosystem, such as [OPA Gatekeeper](https://github.com/open-profile-agent/gateke
### What profiles should I apply to my Windows Pods?
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

Windows in Kubernetes has some limitations and differentiators from standard Linux-based
workloads. Specifically, the Pod SecurityContext fields [have no effect on
workloads. Specifically, many of the Pod SecurityContext fields [have no effect on
Windows](/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#v1-podsecuritycontext). As
such, no standardized Pod Security profiles currently exist.

Windows HostProcess containers can be enabled for the privileged profile, but are explicitly blocked for baseline.
Windows pods _may_ be broken by the restricted profile, which requires setting linux-specific
settings (such as seccomp profile and disallow privilege escalation). If the Kubelet and/or
container runtime choose to ignore these linux-specific values at runtime, then Windows
pods should still be allowed under the restricted profile, although the profile will not
add additional enforcement over baseline (for Windows).
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

### What about sandboxed Pods?

There is not currently an API standard that controls whether a Pod is considered sandboxed or
Expand All @@ -500,4 +541,3 @@ kernel. This allows for workloads requiring heightened permissions to still be i
Additionally, the protection of sandboxed workloads is highly dependent on the method of
sandboxing. As such, no single recommended profile is recommended for all sandboxed workloads.


5 changes: 3 additions & 2 deletions content/en/docs/concepts/workloads/pods/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,8 +257,9 @@ section.

## Privileged mode for containers

Any container in a Pod can enable privileged mode, using the `privileged` flag on the [security context](/docs/tasks/configure-pod-container/security-context/) of the container spec. This is useful for containers that want to use operating system administrative capabilities such as manipulating the network stack or accessing hardware devices.
Processes within a privileged container get almost the same privileges that are available to processes outside a container.
In Linux, any container in a Pod can enable privileged mode using the `privileged` (Linux) flag on the [security context](/docs/tasks/configure-pod-container/security-context/) of the container spec. This is useful for containers that want to use operating system administrative capabilities such as manipulating the network stack or accessing hardware devices.

For Windows clusters with the `WindowsHostProcessContainers` feature enabled, administrators can create a [Windows HostProcess pod](/docs/tasks/configure-pod-container/create-hostprocess-pod) by setting the `windowsOptions.hostProcess` flag on the security context of the pod spec. All containers in these pods must run as Windows HostProcess containers. HostProcess pods run directly on the host and can also be used to perform administrative tasks as is done with Linux privileged containers.
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

{{< note >}}
Your {{< glossary_tooltip text="container runtime" term_id="container-runtime" >}} must support the concept of a privileged container for this setting to be relevant.
Expand Down
2 changes: 2 additions & 0 deletions content/en/docs/reference/command-line-tools-reference/feature-gates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,6 +449,7 @@ different Kubernetes components.
| `WindowsGMSA` | `false` | Alpha | 1.14 | 1.15 |
| `WindowsGMSA` | `true` | Beta | 1.16 | 1.17 |
| `WindowsGMSA` | `true` | GA | 1.18 | - |
| `WindowsHostProcessContainers` | `false` | Alpha | 1.22 |
| `WindowsRunAsUserName` | `false` | Alpha | 1.16 | 1.16 |
| `WindowsRunAsUserName` | `true` | Beta | 1.17 | 1.17 |
| `WindowsRunAsUserName` | `true` | GA | 1.18 | - |
Expand Down Expand Up @@ -956,6 +957,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
- `WinDSR`: Allows kube-proxy to create DSR loadbalancers for Windows.
- `WinOverlay`: Allows kube-proxy to run in overlay mode for Windows.
- `WindowsGMSA`: Enables passing of GMSA credential specs from pods to container runtimes.
- `WindowsHostProcessContainers`: Enables support for Windows HostProcess containers.
- `WindowsRunAsUserName` : Enable support for running applications in Windows containers
with as a non-default user. See
[Configuring RunAsUserName](/docs/tasks/configure-pod-container/configure-runasusername)
Expand Down
205 changes: 205 additions & 0 deletions content/en/docs/tasks/configure-pod-container/create-hostprocess-pod.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
---
title: Create a Windows HostProcess Pod
content_type: task
weight: 20
---
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

<!-- overview -->

{{< feature-state for_k8s_version="v1.22" state="alpha" >}}

Windows HostProcess containers enable you to run containerized
workloads on a Windows host. These containers operate as
normal processes but have access to the host network namespace,
storage, and devices when given the appropriate user privileges.
HostProcess containers can be used to deploy CNIs,
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
storage configurations, device plugins, kube-proxy, and other
components to Windows nodes without the need for dedicated proxies or
the direct installation of host services.

Administrative tasks such as installation of security patches, event
log collection, and more can be performed without requiring cluster operators to
log onto each Window node. HostProcess containers can run as any user that is
available on the host or is in the domain of the host machine, allowing administrators
to restrict resource access through user permissions. While neither filesystem or process
isolation are supported, a new volume is created on the host upon starting the container
to give it a clean and consolidated workspace. HostProcess containers can also be built on
top of existing Windows base images and do not inherit the same
[compatibility requirements](https://docs.microsoft.com/virtualization/windowscontainers/deploy-containers/version-compatibility)
as Windows server containers, meaning that the version of the base images does not need
to match that of the host.

### When should I use a Windows HostProcess container?

- When you need to perform tasks which require the networking namespace of the host.
HostProcess containers have access to the host's network interfaces and IP.
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
- You need access to resources on the host such as the filesystem, event logs, etc.
- Installation of specific devices drivers or services
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
- Consolidation of administrative tasks and security policies. This reduces the degree of
privileges needed by Windows nodes.

### Important Notes
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

- HostProcess containers will **only run when using version 1.5.4 (or higher) of the containerd [container runtime](https://kubernetes.io/docs/setup/production-environment/container-runtimes/)**.
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
- As of v1.22 HostProcess pods can only contain HostProcess containers. This is a current limitation
of the Windows OS; non-privileged Windows containers cannot share a vNIC with the host IP namespace.
- HostProcess containers run as a process on the host and do not have any degree of
isolation other than resource constraints imposed on the HostProcess user account. Neither
filesystem or Hyper-V isolation are supported for HostProcess containers.
- Volume mounts are supported and are mounted under the container volume.
See [Volume Mounts](./create-hostprocess-pod#volume-mounts)
- A limited set of host user accounts are available for HostProcess containers by default.
See [Choosing a User Account](./create-hostprocess-pod#choosing-a-user-account).
- Resource limits (disk, memory, cpu count) are supported in the same fashion as processes
on the host.
- Both Named pipe mounts and Unix domain sockets are **not** currently supported and should instead
be accessed via their path on the host (e.g. \\\\.\\pipe\\\*)

## {{% heading "prerequisites" %}}
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
To enable HostProcess containers while in Alpha you need to pass the following feature gate flag to
**kubelet** and **kube-apiserver**.
See [Features Gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#overview)
documentation for more details.

```
--feature-gates=WindowsHostProcessContainers=true
```

You can use the latest version of Containerd (v1.5.4+) with the following settings using the containerd
v2 configuration. Add these annotations to any runtime configurations were you wish to enable the
HostProcess Container feature.
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

```
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
container_annotations = ["microsoft.com/hostprocess-container"]
pod_annotations = ["microsoft.com/hostprocess-container"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process]
container_annotations = ["microsoft.com/hostprocess-container"]
pod_annotations = ["microsoft.com/hostprocess-container"]
```

The current versions of containerd ship with a version of hcsshim that does not have support.
You will need to build a version of hcsshim from the main branch following the
[instructions in hcsshim](https://github.com/Microsoft/hcsshim/#containerd-shim).
Once the containerd shim is built you can replace the file in your contianerd installation.
For example if you followed the instructions to
[install containerd](https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd)
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
replace the `containerd-shim-runhcs-v1.exe` is installed at `$Env:ProgramFiles\containerd` with the newly built shim.

We are working to improve this process by enabling HostProcess support directly in
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
[containerd via the CRI api](https://github.com/containerd/containerd/pull/5131)
as well as shipping a version of the containerd shim for Windows with support direclty via containerd releases.

## Creating a HostProcess Pod

### Security Policy Config
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
Enabling a Windows HostProcess pod simply requires setting the right configurations in the pod security configuration.
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
Many of the policies in the [Pod Security Standards](/docs/concepts/security/pod-security-standards) do not apply to
HostProcess containers (and Windows in general) due to architectural differences between Windows and Linux. As such,
here are the set of policies and configuration values required to enable HostProcess pods:

<table>
<caption style="display:none">Privileged policy specification</caption>
<tbody>
<tr>
<td><strong>Control</strong></td>
<td><strong>Policy</strong></td>
</tr>
<tr>
<td style="white-space: nowrap"><a href="/docs/concepts/security/pod-security-standards">Windows HostProcess</a></td>
<td>
<p>Windows pods offer the ability to run <a href="/docs/tasks/configure-pod-container/create-hostprocess-pod">
HostProcess containers</a> which enables privileged access to the Windows node. </p>
<p><strong>Allowed Values</strong></p>
<ul>
<li><code>true</code></li>
</ul>
</td>
</tr>
<tr>
<td style="white-space: nowrap"><a href="/docs/concepts/security/pod-security-standards">Host Networking</a></td>
<td>
<p>Will be in host network by default initially. Support
to set network to a different compartment may be desirable in
the future.</p>
<p><strong>Allowed Values</strong></p>
<ul>
<li><code>true</code></li>
</ul>
</td>
</tr>
<tr>
<td style="white-space: nowrap"><a href="/docs/tasks/configure-pod-container/configure-runasusername/">runAsUsername</a></td>
<td>
<p>Specification of which user the HostProcess container should run as is required for the pod spec.</p>
<p><strong>Allowed Values</strong></p>
<ul>
<li><code>"NT AUTHORITY\\SYSTEM"</code></li>
<li><code>"NT AUTHORITY\\Local service"</code></li>
<li><code>"NT AUTHORITY\\NetworkService"</code></li>
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
</ul>
</td>
</tr>
<tr>
<td style="white-space: nowrap"><a href="/docs/concepts/security/pod-security-standards">runAsNonRoot</a></td>
<td>
<p>HostProcess containers have privileged access to the host so the runAsNonRoot field cannot be set to true.</p>
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved
<p><strong>Allowed Values</strong></p>
<ul>
<li>Undefined/Nil</li>
<li><code>false</code></li>
</ul>
</td>
</tr>
</tbody>
</table>

### Example Manifest
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

```yaml
spec:
securityContext:
windowsOptions:
hostProcess: true
runAsUserName: "NT AUTHORITY\\Local service"
hostNetwork: true
containers:
- name: test
image: image1:latest
command:
- ping
- -t
- 127.0.0.1
nodeSelector:
"kubernetes.io/os": windows
```

## Volume Mounts

HostProcess containers support the ability to mount volumes within the container volume space.
Applications running inside the container can access volume mounts directly via relative or
absolute paths. An environment variable `$CONTAINER_SANDBOX_MOUNT_POINT` is set upon container
creation and provides the absolute host path to the container volume. Relative paths are based
upon the `Pod.containers.volumeMounts.mountPath` configuration.

Example: To access service account tokens the following path structures are supported within the container:
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

`.\var\run\secrets\kubernetes.io\serviceaccount\`
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved

`$CONTAINER_SANDBOX_MOUNT_POINT\var\run\secrets\kubernetes.io\serviceaccount\`

## Choosing a User Account

Currently HostProcess containers only support the ability to run as one of three supported Windows service accounts:

- **[LocalSystem](https://docs.microsoft.com/en-us/windows/win32/services/localsystem-account)**
- **[LocalService](https://docs.microsoft.com/en-us/windows/win32/services/localservice-account)**
- **[NetworkService](https://docs.microsoft.com/en-us/windows/win32/services/networkservice-account)**

Operators should use these accounts to limit the degree of privileges made available to the HostProcess container to prevent accidental damage to the host. The LocalSystem account has the highest level of privilege of the three and should be used only if absolutely necessary. It is recommended that operators follow the principle of least privilege and use the LocalService account when possible.
brasmith-ms marked this conversation as resolved.
Show resolved Hide resolved