Allow configuration of Rootlesskit's CopyUpDirs through an environment variable

[ludost]

Is your feature request related to a problem? Please describe.

In our use-case, (see also issue: #10295) we are running a K3S Agent in a Rootless environment at the end-user's own workstation. This works well (enough), but we also want to provide access to the local folders (through e.g. HostPath volume mounts). But the Agent runs inside the rootless namespace, disallowing read/write access to the actual file-system.

Currently the rootless configuration is hardcoded where the CopyUpDirs are concerned. With this issue, I've created a PR that allows the Agent to setup extra "CopyUpDirs" entries, through an environment variable called "K3S_ROOTLESS_COPYUPDIRS". This environment variable can receive a comma-separated list absolute paths, which will be added to the existing list.

Describe the solution you'd like

See the related PR.

Describe alternatives you've considered

Within the context of running rootless I see no alternative to this change. The namespace container is meant specifically to block this kind of access, with the CopyUpDirs parameter the "normal" way of explicitly exposing external folders. We can also consider a more advanced configuration model, but the rest of the rootless package is using environment variables as well for this purpose.

Additional context

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[brandond]

Unstale, we can evaluate for 2024-10

[brandond]

@ludost can you address PR feedback and rebase your PR? If so we can look at this for the October cycle.

[mdrahman-suse]

@ludost I am not able to validate the issue by following the steps mentioned: #10386 (comment) as I see the same behavior on previous versions and master commit, regardless of using the env var K3S_ROOTLESS_COPYUPDIRS. Using single node Ubuntu server

$ systemctl --user status k3s-rootless
● k3s-rootless.service - k3s (Rootless)
     Loaded: loaded (/home/ubuntu/.config/systemd/user/k3s-rootless.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-10-10 03:23:52 UTC; 10min ago
   Main PID: 903 (k3s-server)
      Tasks: 187
     Memory: 1.9G
        CPU: 1min 53.010s
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/k3s-rootless.service

I deployed a simple nginx pod

apiVersion: "v1"
kind: "Pod"
metadata:
  name: "basic"
  labels:
    name: "basic"
spec:
  nodeSelector:
    kubernetes.io/os: linux
  containers:
    - name: "basic"
      image: nginx:1.14.2
      ports:
        - containerPort: 80
          name: "basic"
      volumeMounts:
        - mountPath: "/my_share/"
          name: "container-share"
  volumes:
    - hostPath:
        path: "/home/ubuntu/containerShare/"
        type: ""
      name: "container-share"

and also created a local folder containerShare. When I created a file inside the container, I was able to see the file on the local folder as well, even on the older version (v1.31.1+k3s1). So I am not sure if I am missing any steps/configs.
I tried with setting env var with export K3S_ROOTLESS_COPYUPDIRS=/home/ubuntu/containerShare and without it too, but see the same behavior.

I am following these steps to enable k3s-rootless.service on an ubuntu instance: #3636 (comment)

Please advice.
CC @brandond @ShylajaDevadiga

• On previous version v1.31.1+k3s1, (Downloaded k3s-rootless.service for that version)

$ k3s -v
k3s version v1.31.1+k3s1 (452dbbc1)
go version go1.22.6

$ kubectl exec -it pod/basic -- touch my_share/hello.txt
$ kubectl exec -it pod/basic -- ls my_share
hello.txt
$ ls containerShare/
hello.txt

• On master commit (Downloaded k3s-rootless.service from master)

$ k3s -v
k3s version v1.31.1+k3s-e475ba70 (e475ba70)
go version go1.22.6

$ kubectl exec -it pod/basic -- touch my_share/hello.txt
$ kubectl exec -it pod/basic -- ls my_share
hello.txt
$ ls containerShare/
hello.txt

[brandond]

@mdrahman-suse where are you setting K3S_ROOTLESS_COPYUPDIRS=/home/ubuntu/containerShare? This needs to be in the environment of k3s-rootless.service, not in your shell. It functions just like the other existing K3S_ROOTLESS_ env vars documented at https://docs.k3s.io/advanced#advanced-rootless-configuration

[mdrahman-suse]

I was running the command on the server itself export K3S_ROOTLESS_COPYUPDIRS=/home/ubuntu/containerShare, let me re-try by adding on the service file, ty!

[mdrahman-suse]

I added the env var on k3s-rootless.service unit file but now I am not seeing the expected behavior.
Here is where I added the env var...

sudo vi ~/.config/systemd/user/k3s-rootless.service
...
[Service]
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Environment=K3S_ROOTLESS_CIDR="10.41.0.0/16"
Environment=K3S_ROOTLESS_PORT_DRIVER=slirp4netns
Environment=K3S_ROOTLESS_DISABLE_HOST_LOOPBACK=true
Environment=K3S_ROOTLESS_MTU=1500
Environment=K3S_ROOTLESS_COPYUPDIRS="/home/ubuntu/containerShare"

And when I check the logs, I see the var is there

$ journalctl --user -eu k3s-rootless | grep "K3S"
...
"k3s.io/node-env":"{\"K3S_ROOTLESS_CIDR\":\"10.41.0.0/16\",\"K3S_ROOTLESS_COPYUPDIRS\":\"/home/ubuntu/containerShare\",\"K3S_ROOTLESS_DISABLE_HOST_LOOPBACK\":\"true\",\"K3S_ROOTLESS_MTU\":\"1500\",\"K3S_ROOTLESS_PORT_DRIVER\":\"slirp4netns\"}",
...

But when I add a file on the container, I am not seeing the file on local shared folder

$ kubectl exec -it pod/basic -- touch my_share/hello.txt
$ kubectl exec -it pod/basic -- ls my_share
hello.txt
$ ls containerShare
$

Any suggestions? @brandond

[mdrahman-suse]

But on previous version, I am able to see it working fine, even without setting the new env var 🤷‍♂️

[mdrahman-suse]

I am not able to verify this issue, can I get some guidance on this? CC @ludost @brandond
So with the latest commit, the behavior is when I add the env var on k3s-rootless.service file, I am not able to see the file on the shared folder which was created on the pod as mentioned here: #10385 (comment) but when I dont add that env var, I am able to see the file on local shared folder

[mdrahman-suse]

@ludost As I am not able to replicate or validate this change, can you please verify this issue on your end and provide necessary steps, if possible?
CC @ShylajaDevadiga

[ludost]

I am not able to verify this issue, can I get some guidance on this? CC @ludost @brandond So with the latest commit, the behavior is when I add the env var on k3s-rootless.service file, I am not able to see the file on the shared folder which was created on the pod as mentioned here: #10385 (comment) but when I dont add that env var, I am able to see the file on local shared folder

I believe the behaviour you describe is exactly the intended design.

Normally with a rootless setup the filesystem is made available to a process namespace. Although the user sees these files as if the user has root rights, any files created will still be owned by the user (and visible outside the namespace). This is a problem for files that normally live in a place where the user has no access, for example in /etc/. So, copy-up creates a copy of those folders that only live in the namespace, where the user can actually create files, etc. But these can't be seen from outside the namespace.

So the use-case is: Whenever we want to create a file in a folder that is only available to root, we add it to the environment variable, so the file can then be created. However, after K3S stops, those files are lost. We use it for a very specific case where we are running a modified Flannel setup, which needs write access to "/etc/cni/net.d/" in a rootless environment.

[mdrahman-suse]

I am not able to verify this issue, can I get some guidance on this? CC @ludost @brandond So with the latest commit, the behavior is when I add the env var on k3s-rootless.service file, I am not able to see the file on the shared folder which was created on the pod as mentioned here: #10385 (comment) but when I dont add that env var, I am able to see the file on local shared folder

I believe the behaviour you describe is exactly the intended design.

Normally with a rootless setup the filesystem is made available to a process namespace. Although the user sees these files as if the user has root rights, any files created will still be owned by the user (and visible outside the namespace). This is a problem for files that normally live in a place where the user has no access, for example in /etc/. So, copy-up creates a copy of those folders that only live in the namespace, where the user can actually create files, etc. But these can't be seen from outside the namespace.

So the use-case is: Whenever we want to create a file in a folder that is only available to root, we add it to the environment variable, so the file can then be created. However, after K3S stops, those files are lost. We use it for a very specific case where we are running a modified Flannel setup, which needs write access to "/etc/cni/net.d/" in a rootless environment.

hi @ludost Thanks for explaining it. So to confirm, the steps to validate this:

• Create a folder outside the namespace, add a file there and make sure the file is visible within the pod (where the folder is shared)
• Delete the file from the shared folder and make sure it is not removed from the pod pointing to the shared folder
• Create a file in the pod directory that points to the shared folder and make sure that it not visible/accessible from outside the namespace

Please let me know if I am missing anything or any additional steps is needed
Also it would help if you can validate on your end with this commit 38d13e0

[ludost]

@mdrahman-suse I'm not entirely sure what would be needed to be tested further. This is a very simple fix to add additional folders to an existing feature that works as advertised. I know that the folder is correctly added when using the environment variable, as I see the expected behavior.
Your description "So with the latest commit, .... shared folder" describes the expected behavior and as such verifies that this commit works as intended.

[brandond]

@mdrahman-suse it sounds like what you saw above is specifically what is wanted, so you can close this out.

[mdrahman-suse]

Thanks y'll! I will close out this issue once I verify the other version

[mdrahman-suse]

Validated on other branches as well as this branch. Closing this issue