Support Rootless Docker / Kubernetes

[BenTheElder]

What would you like to be added:

Support for "rootless" docker hosts, docker 20.X should support this.

Why is this needed:

To continue working with the ecosystem as it moves forward.

• KIND PR blocked on the issues below Support Rootless Docker #1727

Related issues / PRS:

• kubelet, cgroupv2: make hugetlb optional kubernetes/kubernetes#93012
• Remove runc default devices that overlap with spec devices. opencontainers/runc#2522
• cgroup2: /sys/fs/cgroup/sys/fs/cgroup is created kubernetes/kubernetes#94104
  • Fix pending in cgroup2: /sys/fs/cgroup/sys/fs/cgroup is created kubernetes/kubernetes#94104
• kubelet & kube-proxy: ignore sysctl errors and rlimit errors when running in UserNS (for rootless) kubernetes/kubernetes#92863
  • Blocked on This KEP being accepted sig-node: Kubelet-in-UserNS, aka Rootless mode kubernetes/enhancements#1371

[BenTheElder]

I would also assign @AkihiroSuda but they're not an org member so github won't let me (kubernetes does not do repo level collaboration by policy, only org membership 🤷)

Consider them assigned anyhow 🙃 (and thank you!)

[spiffxp]

(FWIW I think GitHub has changed permissions such that if they comment on this issue then they can be assigned)

[AkihiroSuda]

👍

[spiffxp]

/assign @AkihiroSuda
let's find out

[AkihiroSuda]

/remove-lifecycle stale

[AkihiroSuda]

/remove-lifecycle stale

[BenTheElder]

The KEP is still blocked, but I expect to ship @AkihiroSuda's kind workaround PR O(soon) / in this next release.

[BenTheElder]

#1935 is shipped. this will be in v0.11.0

[AkihiroSuda]

Thanks for merging #1935 and updating the base image in #2116 !

Can we also have a new node image, so that we can have CI for rootless?

[BenTheElder]

Yes, I'd sort of been holding off for kubernetes/kubernetes#99336 (fix a large regression in startup time for tiny efficient clusters in kubernetes v1.20.4+), but we can just push a new node image for 1.20.2

[BenTheElder]

that is done, and the default at HEAD #2119

[AkihiroSuda]

Thanks @BenTheElder !

[AkihiroSuda]

I'll open a PR to add rootless CI after getting #2127 merged.

[marcofranssen]

experiencing rootless Docker issues as well on selfhosted Github runner.

helm/kind-action#34 (comment)

essentially I followed the setup as described here. https://kind.sigs.k8s.io/docs/user/rootless/#host-requirements

# https://kind.sigs.k8s.io/docs/user/rootless/#host-requirements
echo GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1" >> /etc/default/grub
update-grub

cat >/etc/systemd/system/user@.service.d/delegate.conf <<-EOF
[Service]
Delegate=yes

EOF

systemctl daemon-reload

That still fails.

Run helm/kind-action@v1.1.0
  with:
    node_image: kindest/node:v1.19.7
  env:
    pythonLocation: /home/runners/actions-runner/_work/_tool/Python/3.7.10/x64
    LD_LIBRARY_PATH: /home/runners/actions-runner/_work/_tool/Python/3.7.10/x64/lib
    CT_CONFIG_DIR: /home/runners/actions-runner/_work/_tool/ct/v3.3.1/x86_64/etc
    VIRTUAL_ENV: /home/runners/actions-runner/_work/_tool/ct/v3.3.1/x86_64/venv
Adding kind directory to PATH...
Adding kubectl directory to PATH...
kind v0.9.0 go1.15.2 linux/amd64
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:26:42Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Creating kind cluster...
Creating cluster "chart-testing" ...
 • Ensuring node image (kindest/node:v1.19.7) 🖼  ...
 ✓ Ensuring node image (kindest/node:v1.19.7) 🖼
 • Preparing nodes 📦   ...
 ✓ Preparing nodes 📦 
 • Writing configuration 📜  ...
 ✗ Writing configuration 📜
ERROR: failed to create cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "docker exec --privileged chart-testing-control-plane cat /kind/version" failed with error: exit status 1

Command Output: Error response from daemon: Container 2cfd8f32bd974c3dc7ea5ae24a7e9fe959648eebacb0428d46383ff84f490540 is not running

• Is the documentation running ahead of the feature?
• Is there a bug?
• Is there some missing info in the documentation?

[AkihiroSuda]

You need kind v0.11. Unreleased yet, but feature is available on main branch. (You need to build the node image in addition to kind binary by yourself)

[BenTheElder]

Starting with kind 0.11.0, Rootless Docker and Rootless Podman can be used as the node provider of kind.

(First sentence on the page, note that install guide and release pages list v0.10.0)

It will be released soon. In the meantime you can try it early from installing at HEAD. Clone, make build, use bin/kind. We have one default relatively up to date node image for HEAD so you don't have to build that.

[AkihiroSuda]

We have one default relatively up to date node image for HEAD so you don't have to build that.

kindest/node:v1.20.2@sha256:15d3b5c4f521a84896ed1ead1b14e4774d02202d5c65ab68f30eeaf310a3b1a7 (Mar 11) lacks #2131 , so it doesn't work without kernel 5.11, Ubuntu kernel, or Debian kernel.
Any chance to get the node image synced with the latest base image?

[djannot]

I've built it from master to try it and it looks like it's using docker info instead of podman info to determine the cgroupVersion

[AkihiroSuda]

I've built it from master to try it and it looks like it's using docker info instead of podman info to determine the cgroupVersion

Please set export KIND_EXPERIMENTAL_PROVIDER=podman if you want to use Podman

[djannot]

@AkihiroSuda I did that, but it still seems to use docker info

using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires cgroup v2, see https://kind.sigs.k8s.io/docs/user/rootless/

$ docker info --format '{{json .}}' | jq . | grep -i cgroup
  "CgroupDriver": "cgroupfs",
  "CgroupVersion": "1",
$ podman info --format '{{json .}}' | jq . | grep -i cgroup
    "cgroupManager": "systemd",
    "cgroupVersion": "v2",

[AkihiroSuda]

I don't think it's reading cgroup version from Docker (you can verify with sudo execsnoop.bt).

Does /sys/fs/cgroup/cgroup.controllers exist with non-empty content? If not, podman is returning wrong cgroup version. (Please report to Podman repo)

[djannot]

@AkihiroSuda I have podman configured to point to a Linux VM (from my Mac). So it should use podman info to get the cgroupVersion of the Linux VM instead of trying to get it locally.

[AkihiroSuda]

Remote podman is unsupported

Reason:

kind/pkg/cluster/internal/providers/podman/provider.go

Lines 374 to 375 in b6bc112

	// Unlike `docker info`, `podman info` does not print available cgroup controllers.
	// So we parse "cgroup.subtree_control" file by ourselves.

[djannot]

Thanks @AkihiroSuda

[AkihiroSuda]

FYI: I added rootless (and kind) to Apr 20 SIG-node agenda https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit#

[AkihiroSuda]

Support for remote podman is being added in #2256

[BenTheElder]

from the original checklist we are now just waiting on kubernetes/kubernetes#92863 which is almost to the finish line 🤞

[AkihiroSuda]

KubeletInUserNamespace feature gate (kubernetes/kubernetes#92863) was merged in Kubernetes v1.22.

I'll open a PR to use KubeletInUserNamespace feature gate when kind (v0.12.X in my assumption) is updated to use Kubernetes v1.22.

The current kind v0.11.X already supports Kubernetes v1.21 with rootless providers, but support for Kubernetes v1.21 with rootless providers will be probably dropped in kind v0.12.0 for simplifying entrypoint script.

[aojea]

can we say that this is done @BenTheElder ?