Support for pod user namespaces

[Piccirello]

This issue is to track Talos's support for user namespaces¹ in Kubernetes pods. User namespaces allow for strict separation between the root user in pods and the root user on the host. From the docs: "A process running as root in a container can run as a different (non-root) user in the host; in other words, the process has full privileges for operations inside the user namespace, but is unprivileged for operations outside the namespace."

User namespaces requires at least Linux 6.3, which it appears Talos v1.7.0 will support. The Kubernetes docs also state that "containerd v1.7 is not compatible with the userns support in Kubernetes v1.27 to v1.29." That may mean waiting for containerd 2.0², though this is unclear to me.

When user namespaces are eventually supported, it would be worth mentioning as a feature in the Talos release's changelog.

Footnotes

1. https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/ ↩

2. https://github.com/containerd/containerd/releases/tag/v2.0.0-rc.0 ↩

[sanmai-NL]

How will this affect Talos Linux nodes running inside containers? And potentially, in user-namespaced/rootless containers?

[Piccirello]

The kubernetes docs page¹ linked above has been updated with more information. It now seems more explicitly clear that containerd v2 is needed.

containerd v1.7 is not compatible with the userns support in Kubernetes v1.27 to v1.30. Kubernetes v1.25 and v1.26 used an earlier implementation that is compatible with containerd v1.7, in terms of userns support.

Footnotes

1. https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/ ↩

[Piccirello]

Based on #8766, #8777, and #8484, it appears Talos 1.8 will use containerd 2.0. That may mean that pod user namespaces will be supported in Talos 1.8.

[smira]

Yes, it should be ready for that, I believe there's nothing to be done on Talos OS side itself to support that. If you have a good testcase for user namespaces (e.g. something you can kubectl apply), we'd be happy to get it into the integration tests. Thanks!

[Piccirello]

It appears that there are two feature gates that need to be enabled: UserNamespacesSupport and UserNamespacesPodSecurityStandards. Both of these currently default to false, with the latter's effect described here.

I love the idea of including a testcase. The KEP states that if the runtime doesn't support user namespaces, a deployment with hostUsers: false will fail to be created. A sample Pod definition is provided for testing. In my testing against Kubernetes v1.30.1 on Talos 1.7.3, that Pod definition is deployed without issue, despite the Pod running in the host user namespace (confirmed with cat /proc/self/uid_map). I don't know why this works, though I suspect it's because the feature gate is disabled. If the feature gate were enabled, I would expect the deployment to fail due to the use of containerd v1.7. I'm not sure that this presents a clear testcase though, as it sounds like there would only be an error produced when the feature gate is enabled AND containerd <1.8 is used. Ideally the test case would fail whenever the host user namespace is used (i.e. including when the feature gate is disabled).

[frezbo]

I think there some bug with kubelet, it never fails and the mappings inside the pod are completely wrong, I would have expected kubelet to fail to create or throw an error and that is not the expected behavior, tested by adding the feature gate

[frezbo]

More updates on this, this feature seems to give a false sense of security, if the feature gate is not enabled a pod with hostUsers: false set would be happily scheduled and running on a node that does not meet any requirement for user namespaces, this seems weird and seems like a security issue giving a false sense of security.

With the feature gate enabled and hostUsers: false set the pod fails to be scheduled with this error:

failed to mount rootfs component: no space left on device

which in indeed a red herring and might be some other issue masked by this error

[frezbo]

Created an issue in k8s kubernetes/kubernetes#126484

[Piccirello]

@smira I believe Talos should enable the above feature gates by default in v1.8.

With the feature gate enabled and hostUsers: false set the pod fails to be scheduled with this error:

failed to mount rootfs component: no space left on device

My guess is that this error is specific to Talos's read-only environment.

[frezbo]

kubelet doesn't provide more info on how to debug this, so we're at a loss

[frezbo]

more updates:

10.5.0.3: {"ts":1730915127677.5244,"caller":"internal/log.go:32","msg":"RunPodSandbox from runtime service failed","err":"rpc error: code = Unknown desc = failed to create network namespace for sandbox \"0752eaba05387bb7782ff1ad87ebd37ec3e1129e4e6e8c0695fc4337eac088d5\": failed to start noop process for unshare: fork/exec /proc/self/exe: no space left on device"}
10.5.0.3: {"ts":1730915127677.6318,"caller":"kuberuntime/kuberuntime_sandbox.go:72","msg":"Failed to create sandbox for pod","pod":{"name":"userns","namespace":"default"},"err":"rpc error: code = Unknown desc = failed to create network namespace for sandbox \"0752eaba05387bb7782ff1ad87ebd37ec3e1129e4e6e8c0695fc4337eac088d5\": failed to start noop process for unshare: fork/exec /proc/self/exe: no space left on device"}
10.5.0.3: {"ts":1730915127677.6892,"caller":"kuberuntime/kuberuntime_manager.go:1154","msg":"CreatePodSandbox for pod failed","pod":{"name":"userns","namespace":"default"},"err":"rpc error: code = Unknown desc = failed to create network namespace for sandbox \"0752eaba05387bb7782ff1ad87ebd37ec3e1129e4e6e8c0695fc4337eac088d5\": failed to start noop process for unshare: fork/exec /proc/self/exe: no space left on device"}
10.5.0.3: {"ts":1730915127677.7856,"caller":"kubelet/pod_workers.go:1301","msg":"Error syncing pod, skipping","pod":{"name":"userns","namespace":"default"},"podUID":"7012e990-fcd5-4f1a-9c41-1fa10780c91f","err":"failed to \"CreatePodSandbox\" for \"userns_default(7012e990-fcd5-4f1a-9c41-1fa10780c91f)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"userns_default(7012e990-fcd5-4f1a-9c41-1fa10780c91f)\\\": rpc error: code = Unknown desc = failed to create network namespace for sandbox \\\"0752eaba05387bb7782ff1ad87ebd37ec3e1129e4e6e8c0695fc4337eac088d5\\\": failed to start noop process for unshare: fork/exec /proc/self/exe: no space left on device\"","errCauses":[{"error":"failed to \"CreatePodSandbox\" for \"userns_default(7012e990-fcd5-4f1a-9c41-1fa10780c91f)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"userns_default(7012e990-fcd5-4f1a-9c41-1fa10780c91f)\\\": rpc error: code = Unknown desc = failed to create network namespace for sandbox \\\"0752eaba05387bb7782ff1ad87ebd37ec3e1129e4e6e8c0695fc4337eac088d5\\\": failed to start noop process for unshare: fork/exec /proc/self/exe: no space left on device\""}]}

[Piccirello]

Confirmed that user namespaces is working as expected on Talos v1.8.2 after following the docs added in #9662. talosctl read /proc/$PID/status -n $WORKER_IP | grep "Uid:" displays non-zero values for all fields when $PID's corresponding deployment contains hostUsers: false.