Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue: Excessive Creation of Temporary Directories in Root with Sysbox #829

Closed
xabru opened this issue Aug 25, 2024 · 7 comments · Fixed by nestybox/sysbox-fs#100
Closed
Assignees

Comments

@xabru
Copy link

xabru commented Aug 25, 2024

Description:

We have observed that many temporary directories are being created in the root directory with names following the pattern .sysbox-sysfs-<number>. This is causing clutter and potential confusion.

System Environment:

  • Sysbox version: v0.6.4
  • Dockerfile Template: Systemd in Docker

Additional Information:

This issue might have arisen because I masked the following services using systemctl:

systemctl mask \
	systemd-udevd.service \
	systemd-udevd-kernel.socket \
	systemd-udevd-control.socket \
	systemd-modules-load.service \
	sys-kernel-config.mount \
	sys-kernel-debug.mount \
	sys-kernel-tracing.mount \
	e2scrub_all.timer \
	e2scrub_reap \
	accounts-daemon.service \
	rtkit-daemon.service \
	systemd-hostnamed.service

$_ ls -hasl /

> ll /
total 284
drwxr-xr-x   1 root   root    4096 Aug 24 17:51 ./
drwxr-xr-x   1 root   root    4096 Aug 24 17:51 ../
lrwxrwxrwx   1 root   root       7 Apr 22 08:08 bin -> usr/bin/
drwxr-xr-x   2 root   root    4096 Mar 31 04:00 bin.usr-is-merged/
drwxr-xr-x   2 root   root    4096 Apr 22 08:08 boot/
drwxr-xr-x   6 root   root     440 Aug 24 17:50 dev/
-rwxr-xr-x   1 root   root       0 Aug 24 17:50 .dockerenv*
drwxr-xr-x   2 root   root    4096 Aug 23 07:48 dockerstartup/
drwxr-xr-x   1 root   root    4096 Aug 24 17:50 etc/
drwxr-xr-x   1 root   root    4096 Aug 23 07:40 home/
lrwxrwxrwx   1 root   root       7 Apr 22 08:08 lib -> usr/lib/
lrwxrwxrwx   1 root   root       9 Apr 22 08:08 lib64 -> usr/lib64/
drwxr-xr-x   2 root   root    4096 Apr  8 09:37 lib.usr-is-merged/
drwxr-xr-x   2 root   root    4096 Aug  1 06:59 media/
drwxr-xr-x   2 root   root    4096 Aug  1 06:59 mnt/
drwxr-xr-x   1 root   root    4096 Aug 24 17:50 opt/
dr-xr-xr-x 283 root   root       0 Aug 24 17:50 proc/
drwx------   1 root   root    4096 Aug 24 18:59 root/
drwxr-xr-x  20 root   root     580 Aug 24 17:51 run/
lrwxrwxrwx   1 root   root       8 Apr 22 08:08 sbin -> usr/sbin/
drwxr-xr-x   2 root   root    4096 Mar 31 04:00 sbin.usr-is-merged/
drwxr-xr-x   2 root   root    4096 Aug  1 06:59 srv/
dr-xr-xr-x  13 nobody nogroup    0 Aug 24 18:10 sys/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1086082501/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1098625909/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1220785409/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1227003664/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1270347815/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1402730274/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1427450857/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1431106688/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-148529927/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-149844544/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1557262215/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1591870079/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1692417296/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-1863566724/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-1988865741/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2052994440/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2188883916/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2412479115/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-269341428/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-270781546/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2722782661/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-273937039/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2892516956/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-2924257033/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3224083108/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3229380096/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-324496473/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3591021414/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-3622867146/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3633530951/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-3707117874/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3742760121/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-3787667621/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-4095605731/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-4136466978/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-4175937595/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-586173192/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-628597558/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-714564999/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-735075657/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-740134491/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-863786270/
drwx------   2 root   root    4096 Aug 24 17:51 .sysbox-sysfs-867797088/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-927480688/
drwx------   2 root   root    4096 Aug 24 17:50 .sysbox-sysfs-981093558/
drwxrwxrwt   1 root   root    4096 Aug 24 18:59 tmp/
drwxr-xr-x   1 root   root    4096 Aug  1 06:59 usr/
drwxr-xr-x   1 root   root    4096 Aug 24 17:50 var/
@AidanAbd
Copy link

We have experiences this as well, and are currently working around it by manually cleaning up these empty directories.

@ctalledo
Copy link
Member

Hi @xabru, @AidanAbd , thanks for reporting this.

Yes, I can confirm this is a bug. I have a fix in mind, will work on this ASAP. Sorry for the inconvenience.

@EddieX64
Copy link

EddieX64 commented Nov 11, 2024

Hello @ctalledo @rodnymolina

I have tested sysbox v0.6.5 on GKE 1.29 and still see .sysbox-sysfs directories, but now there are also .sysbox-procfs directories in /

When i execute docker run -it alpine:latest sh inside a sysbox pod, additionally i can see .sysbox-procfs directories in the / of the inner container. In some scenarios this directory causing permission denied error as it's owned by root:root. Previously I used image ghcr.io/nestybox/sysbox-deploy-k8s:v0.6.5-dev-0 and it was working fine, but for some reason got deleted after official release of v0.6.5 :(

Now my projects are broken, because i can't rollback to v0.6.4 since i need support of sysbox-fs emulation to allow writes to '/proc/sys/kernel/shm*' paths, and this feature was added in v0.6.5-dev-0, but now in official v0.6.5 these .sysbox-procfs and .sysbox-sysfs are causing .sysbox-procfs permission denied issues.
Could you please have a look?

ctalledo added a commit to nestybox/sysbox-fs that referenced this issue Dec 5, 2024
Fixes left-over .sysbox-procfs-* and .sysbox-sysfs-* dirs inside
containers. They were being incorrectly left over by sysbox-fs nsenter
processes.

Fixes nestybox/sysbox#829.

Signed-off-by: Cesar Talledo <cesar.talledo@docker.com>
@ctalledo
Copy link
Member

ctalledo commented Dec 5, 2024

Hi @EddieX64, apologies for the late reply.

Thanks for letting me know, it was my mistake (thought I had fixed this in v0.6.5 but I made a mistake in the fix and did not catch it somehow).

This PR will fix it once and for all: nestybox/sysbox-fs#100

We will generate a v0.6.6 release soon afterwards.

@ctalledo ctalledo self-assigned this Dec 5, 2024
ctalledo added a commit to nestybox/sysbox-fs that referenced this issue Dec 5, 2024
Fixes left-over .sysbox-procfs-* and .sysbox-sysfs-* dirs inside
containers. They were being incorrectly left over by sysbox-fs nsenter
processes.

Fixes nestybox/sysbox#829.

Signed-off-by: Cesar Talledo <cesar.talledo@docker.com>
@aldcons
Copy link

aldcons commented Dec 10, 2024

Hi @EddieX64, apologies for the late reply.

Thanks for letting me know, it was my mistake (thought I had fixed this in v0.6.5 but I made a mistake in the fix and did not catch it somehow).

This PR will fix it once and for all: nestybox/sysbox-fs#100

We will generate a v0.6.6 release soon afterwards.

We are also experiencing the permissions problem with '/proc/sys/kernel/shm*' on certain builds.

Is there a rough estimate when release 0.6.6 will be out?

Or is there a Dev registry we can use in the meantime?

Thanks.

@aldcons
Copy link

aldcons commented Dec 10, 2024

Hi @EddieX64, apologies for the late reply.
Thanks for letting me know, it was my mistake (thought I had fixed this in v0.6.5 but I made a mistake in the fix and did not catch it somehow).
This PR will fix it once and for all: nestybox/sysbox-fs#100
We will generate a v0.6.6 release soon afterwards.

We are also experiencing the permissions problem with '/proc/sys/kernel/shm*' on certain builds.

Is there a rough estimate when release 0.6.6 will be out?

Or is there a Dev registry we can use in the meantime?

Thanks.

After all that, we seem to now be getting caught with this one - nestybox/sysbox-fs#101

Is that also in the next release @ctalledo ?

@ctalledo
Copy link
Member

ctalledo commented Dec 10, 2024

Hi @aldcons,

Is there a rough estimate when release 0.6.6 will be out?

We are targeting end of this week.

It will fix both nestybox/sysbox-fs#100 and nestybox/sysbox-fs#101. Plus some other fixes.

Or is there a Dev registry we can use in the meantime?

There is no dev registry, but you can always build the sysbox package from source by cloning the sysbox repo.

$ git clone --recurse-submodules git@github.com:nestybox/sysbox.git
$ cd sysbox
$ Edit the VERSION file (e.g., `0.6.6-dev`)
$ cd sysbox-pkgr
$ make sysbox-ce-repo <path-to-where-you-cloned-sysbox>
$ make sysbox-ce-deb

This will generate the Sysbox Debian package under directory sysbox-pkgr/deb/build/amd64/ubuntu-jammy/.

You can then install that on your host with sudo apt-get install .... But first uninstall any current sysbox installation with sudo apt-get purge sysbox-ce.

If all is rignt, the systemctl status sysbox should show Sysbox has the version you put in the VERSION file in the steps above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants